1*** Settings *** 2 3Documentation Utilities for Robot keywords that do not use REST. 4 5Resource ../lib/resource.robot 6Resource ../lib/connection_client.robot 7Resource ../lib/boot_utils.robot 8Library String 9Library DateTime 10Library Process 11Library OperatingSystem 12Library gen_print.py 13Library gen_robot_print.py 14Library gen_cmd.py 15Library gen_robot_valid.py 16Library gen_robot_keyword.py 17Library bmc_ssh_utils.py 18Library utils.py 19Library var_funcs.py 20Library SCPLibrary WITH NAME scp 21 22*** Variables *** 23 24${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION 25 26${dbuscmdBase} 27... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host 28${dbuscmdGet} 29... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get 30${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string: 31 32# Assign default value to QUIET for programs which may not define it. 33${QUIET} ${0} 34 35${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4 36${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2 37${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9 38${HOST_SETTING} ${SETTINGS_URI}host0 39 40# /run/initramfs/ro associate filesystem should be 100% full always 41${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l 42${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l 43${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l 44 45${BOOT_TIME} ${0} 46${BOOT_COUNT} ${0} 47${count} ${0} 48${devicetree_base} /sys/firmware/devicetree/base/model 49 50# Initialize default debug value to 0. 51${DEBUG} ${0} 52 53${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh 54${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh 55${target_file_path} /root/ 56 57${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz 58 59# These variables are used to straddle between new and old methods of setting 60# values. 61${bmc_power_policy_method} ${EMPTY} 62@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON 63... ALWAYS_POWER_OFF 64 65 66*** Keywords *** 67 68Check BMC Performance 69 [Documentation] Check BMC basic CPU Mem File system performance. 70 71 Check BMC CPU Performance 72 Check BMC Mem Performance 73 Check BMC File System Performance 74 75 76Verify PNOR Update 77 [Documentation] Verify that the PNOR is not corrupted. 78 # Example: 79 # FFS: Flash header not found. Code: 100 80 # Error 100 opening ffs ! 81 82 ${stdout} ${stderr} ${rc}= 83 ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip 84 ... ignore_err=${1} 85 ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096 86 ... ${pflash_cmd} 87 ${pnor_info}= BMC Execute Command ${pflash_cmd} 88 Should Not Contain Any ${pnor_info} Flash header not found Error 89 90 91Get BMC System Model 92 [Documentation] Get the BMC model from the device tree and return it. 93 94 ${bmc_model} ${stderr} ${rc}= BMC Execute Command 95 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True 96 ... test_mode=0 97 Should Be Empty ${stderr} 98 Should Not Be Empty ${bmc_model} msg=BMC model is empty. 99 [Return] ${bmc_model} 100 101 102Verify BMC System Model 103 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}. 104 [Arguments] ${bmc_model} 105 106 # Description of argument(s): 107 # bmc_model System model (e.g. "witherspoon"). 108 109 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} / 110 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} . 111 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model} 112 ... ${tmp_bmc_model} ignore_case=True 113 [Return] ${ret} 114 115 116Wait For Host To Ping 117 [Documentation] Wait for the given host to ping. 118 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min 119 ... ${interval}=5 sec 120 121 # Description of argument(s): 122 # host The host name or IP of the host to ping. 123 # timeout The amount of time after which ping attempts cease. 124 # This should be expressed in Robot Framework's time format 125 # (e.g. "10 seconds"). 126 # interval The amount of time in between attempts to ping. 127 # This should be expressed in Robot Framework's time format 128 # (e.g. "5 seconds"). 129 130 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host} 131 132 133Ping Host 134 [Documentation] Ping the given host. 135 [Arguments] ${host} 136 137 # Description of argument(s): 138 # host The host name or IP of the host to ping. 139 140 Should Not Be Empty ${host} msg=No host provided 141 ${RC} ${output}= Run and return RC and Output ping -c 4 ${host} 142 Log RC: ${RC}\nOutput:\n${output} 143 Should be equal ${RC} ${0} 144 145 146Check OS 147 [Documentation] Attempts to ping the host OS and then checks that the host 148 ... OS is up by running an SSH command. 149 150 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME} 151 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET} 152 ... ${print_string}=${EMPTY} 153 [Teardown] SSHLibrary.Close Connection 154 155 # Description of argument(s): 156 # os_host The DNS name/IP of the OS host associated with our BMC. 157 # os_username The username to be used to sign on to the OS host. 158 # os_password The password to be used to sign on to the OS host. 159 # quiet Indicates whether this keyword should write to console. 160 # print_string A string to be printed before checking the OS. 161 162 Log To Console ${print_string} no_newline=True 163 164 # Attempt to ping the OS. Store the return code to check later. 165 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host} 166 167 SSHLibrary.Open connection ${os_host} 168 169 ${status} ${msg}= Run Keyword And Ignore Error SSHLibrary.Login ${os_username} 170 ... ${os_password} 171 ${err_msg1}= Sprint Error ${msg} 172 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} 173 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg} 174 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True 175 ... return_rc=True 176 177 ${temp_msg}= Catenate Could not execute a command on the operating 178 ... system.\n 179 ${err_msg1}= Sprint Error ${temp_msg} 180 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} 181 182 # If the return code returned by "Execute Command" is non-zero, this 183 # keyword will fail. 184 Should Be Equal ${rc} ${0} msg=${err_msg} 185 # We will likewise fail if there is any stderr data. 186 Should Be Empty ${stderr} 187 188 ${temp_msg}= Set Variable Could not ping the operating system.\n 189 ${err_msg1}= Sprint Error ${temp_msg} 190 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} 191 # We will likewise fail if the OS did not ping, as we could SSH but not 192 # ping 193 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg} 194 195 196Wait for OS 197 [Documentation] Waits for the host OS to come up via calls to "Check OS". 198 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME} 199 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT} 200 ... ${quiet}=${0} 201 [Teardown] Printn 202 203 # Description of argument(s): 204 # os_host The DNS name or IP of the OS host associated with our 205 # BMC. 206 # os_username The username to be used to sign on to the OS host. 207 # os_password The password to be used to sign on to the OS host. 208 # timeout The timeout in seconds indicating how long you're 209 # willing to wait for the OS to respond. 210 # quiet Indicates whether this keyword should write to console. 211 212 # The interval to be used between calls to "Check OS". 213 ${interval}= Set Variable 5 214 215 ${message}= Catenate Checking every ${interval} seconds for up to 216 ... ${timeout} seconds for the operating system to communicate. 217 Qprint Timen ${message} 218 219 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS 220 ... ${os_host} ${os_username} ${os_password} 221 ... print_string=\# 222 223 Qprintn 224 225 Qprint Timen The operating system is now communicating. 226 227 228Copy PNOR to BMC 229 [Documentation] Copy the PNOR image to the BMC. 230 Import Library SCPLibrary WITH NAME scp 231 Open Connection for SCP 232 Log Copying ${PNOR_IMAGE_PATH} to /tmp 233 scp.Put File ${PNOR_IMAGE_PATH} /tmp 234 235 236Is OS Starting 237 [Documentation] Check if boot progress is OS starting. 238 ${boot_progress}= Get Boot Progress 239 Should Be Equal ${boot_progress} OSStart 240 241 242Is OS Off 243 [Documentation] Check if boot progress is "Off". 244 ${boot_progress}= Get Boot Progress 245 Should Be Equal ${boot_progress} Off 246 247 248Get Boot Progress To OS Starting State 249 [Documentation] Get the system to a boot progress state of 'FW Progress, 250 ... Starting OS'. 251 252 ${boot_progress}= Get Boot Progress 253 Run Keyword If '${boot_progress}' == 'OSStart' 254 ... Log Host is already in OS starting state 255 ... ELSE 256 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot 257 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting 258 259 260Check If warmReset is Initiated 261 [Documentation] Ping would be still alive, so try SSH to connect 262 ... if fails the ports are down indicating reboot 263 ... is in progress 264 265 # Warm reset adds 3 seconds delay before forcing reboot 266 # To minimize race conditions, we wait for 7 seconds 267 Sleep 7s 268 ${alive}= Run Keyword and Return Status 269 ... Open Connection And Log In 270 Return From Keyword If '${alive}' == '${False}' ${False} 271 [Return] ${True} 272 273 274Initialize DBUS cmd 275 [Documentation] Initialize dbus string with property string to extract 276 [Arguments] ${boot_property} 277 278 # Description of argument(s): 279 # boot_property Property string. 280 281 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString} 282 ${cmd}= Catenate ${cmd}${boot_property} 283 Set Global Variable ${dbuscmd} ${cmd} 284 285 286Create OS Console Command String 287 [Documentation] Return a command string to start OS console logging. 288 289 # First make sure that the ssh_pw program is available. 290 ${cmd}= Catenate which ssh_pw 2>/dev/null || find 291 ... ${EXECDIR} -name 'ssh_pw' 292 293 Dprint Issuing ${cmd} 294 ${rc} ${output}= Run And Return Rc And Output ${cmd} 295 Rdpvars rc output 296 297 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw. 298 299 ${ssh_pw_file_path}= Set Variable ${output} 300 301 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p 2200 302 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST} 303 304 [Return] ${cmd} 305 306 307Get SOL Console Pid 308 [Documentation] Get the pid of the active SOL console job. 309 [Arguments] ${expect_running}=${0} ${log_file_path}=${EMPTY} 310 311 # Description of argument(s): 312 # expect_running If set and if no SOL console job is found, print debug info and fail. 313 # log_file_path Needed to print debug info if expect_running is set and no pid is found. 314 315 # Find the pid of the active system console logging session (if any). 316 ${search_string}= Create OS Console Command String 317 # At least in some cases, ps output does not show double quotes so we must 318 # replace them in our search string with the regexes to indicate that they 319 # are optional. 320 ${search_string}= Replace String ${search_string} " ["]? 321 ${ps_cmd}= Catenate ps axwwo user,pid,cmd 322 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' | 323 ... egrep -v grep | cut -c10-14) 324 Dprint Issuing ${cmd_buf} 325 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf} 326 Rdpvars os_con_pid 327 # If rc is not zero it just means that there is no OS Console process 328 # running. 329 330 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid} 331 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid} 332 333 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0} 334 ... print_output=${1} show_err=${1} 335 Valid Value os_con_pid 336 337 338Stop SOL Console Logging 339 [Documentation] Stop system console logging and return log output. 340 [Arguments] ${log_file_path}=${EMPTY} 341 ... ${targ_file_path}=${EXECDIR}${/}logs${/} 342 ... ${return_data}=${1} 343 344 # If there are muliple system console processes, they will all be stopped. 345 # If there is no existing log file this keyword will return an error 346 # message to that effect (and write that message to targ_file_path, if 347 # specified). 348 # NOTE: This keyword will not fail if there is no running system console 349 # process. 350 351 # Description of arguments: 352 # log_file_path The file path that was used to call "Start SOL 353 # Console Logging". See that keyword (above) for details. 354 # targ_file_path If specified, the file path to which the source 355 # file path (i.e. "log_file_path") should be copied. 356 # return_data If this is set to ${1}, this keyword will return the SOL 357 # data to the caller as a unicode string. 358 359 ${log_file_path}= Create OS Console File Path ${log_file_path} 360 361 ${os_con_pid}= Get SOL Console Pid 362 363 ${cmd_buf}= Catenate kill -9 ${os_con_pid} 364 Run Keyword If '${os_con_pid}' != '${EMPTY}' Dprint Issuing ${cmd_buf} 365 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}' 366 ... Run And Return Rc And Output ${cmd_buf} 367 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output 368 369 Run Keyword If '${targ_file_path}' != '${EMPTY}' 370 ... Run Keyword And Ignore Error 371 ... Copy File ${log_file_path} ${targ_file_path} 372 373 ${output}= Set Variable ${EMPTY} 374 ${loc_quiet}= Evaluate ${debug}^1 375 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}' 376 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet} 377 ... print_output=${0} show_err=${0} 378 379 [Return] ${output} 380 381 382Start SOL Console Logging 383 [Documentation] Start system console log to file. 384 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1} 385 386 # This keyword will first call "Stop SOL Console Logging". Only then will 387 # it start SOL console logging. The data returned by "Stop SOL Console 388 # Logging" will in turn be returned by this keyword. 389 390 # Description of arguments: 391 # log_file_path The file path to which system console log data should be 392 # written. Note that this path is taken to be a location 393 # on the machine where this program is running rather than 394 # on the Open BMC system. 395 # return_data If this is set to ${1}, this keyword will return any SOL 396 # data to the caller as a unicode string. 397 398 ${log_file_path}= Create OS Console File Path ${log_file_path} 399 400 ${log_output}= Stop SOL Console Logging ${log_file_path} 401 ... return_data=${return_data} 402 403 # Validate by making sure we can create the file. Problems creating the 404 # file would not be noticed by the subsequent ssh command because we fork 405 # the command. 406 Create File ${log_file_path} 407 ${sub_cmd_buf}= Create OS Console Command String 408 # Routing stderr to stdout so that any startup error text will go to the 409 # output file. 410 # TODO: Doesn't work with tox so reverting temporarily. 411 # nohup detaches the process completely from our pty. 412 #${cmd_buf}= Catenate nohup ${sub_cmd_buf} &> ${log_file_path} & 413 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 & 414 Dprint Issuing ${cmd_buf} 415 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf} 416 # Because we are forking this command, we essentially will never get a 417 # non-zero return code or any output. 418 Should Be Equal ${rc} ${0} 419 420 Wait Until Keyword Succeeds 10 seconds 0 seconds 421 ... Get SOL Console Pid ${1} ${log_file_path} 422 423 [Return] ${log_output} 424 425 426Get Time Stamp 427 [Documentation] Get the current time stamp data 428 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f 429 [Return] ${cur_time} 430 431 432Start Journal Log 433 [Documentation] Start capturing journal log to a file in /tmp using 434 ... journalctl command. By default journal log is collected 435 ... at /tmp/journal_log else user input location. 436 ... The File is appended with datetime. 437 [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY} 438 439 # Description of arguments: 440 # file_path The file path of the journal file. 441 442 ${cur_time}= Get Time Stamp 443 Set Global Variable ${LOG_TIME} ${cur_time} 444 Open Connection And Log In 445 Start Command 446 ... journalctl -f ${filter} > ${file_path}-${LOG_TIME} 447 Log Journal Log Started: ${file_path}-${LOG_TIME} 448 449 450Stop Journal Log 451 [Documentation] Stop journalctl process if its running. 452 ... By default return log from /tmp/journal_log else 453 ... user input location. 454 [Arguments] ${file_path}=/tmp/journal_log 455 456 # Description of arguments: 457 # file_path The file path of the journal file. 458 459 Open Connection And Log In 460 461 ${rc}= 462 ... Execute Command 463 ... ps | grep journalctl | grep -v grep 464 ... return_stdout=False return_rc=True 465 466 Return From Keyword If '${rc}' == '${1}' 467 ... No journal log process running 468 469 ${output} ${stderr}= 470 ... Execute Command killall journalctl 471 ... return_stderr=True 472 Should Be Empty ${stderr} 473 474 ${journal_log} ${stderr}= 475 ... Execute Command 476 ... cat ${file_path}-${LOG_TIME} 477 ... return_stderr=True 478 Should Be Empty ${stderr} 479 480 Log ${journal_log} 481 482 Execute Command rm ${file_path}-${LOG_TIME} 483 484 [Return] ${journal_log} 485 486 487Mac Address To Hex String 488 [Documentation] Converts MAC address into hex format. 489 ... Example 490 ... Given the following MAC: 00:01:6C:80:02:78 491 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78 492 ... Description of arguments: 493 ... i_macaddress MAC address in the following format 494 ... 00:01:6C:80:02:78 495 [Arguments] ${i_macaddress} 496 497 # Description of arguments: 498 # i_macaddress The MAC address. 499 500 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')} 501 [Return] ${mac_hex} 502 503 504IP Address To Hex String 505 [Documentation] Converts IP address into hex format. 506 ... Example: 507 ... Given the following IP: 10.3.164.100 508 ... This keyword will return: 0xa 0x3 0xa4 0xa0 509 [Arguments] ${i_ipaddress} 510 511 # Description of arguments: 512 # i_macaddress The IP address in the format 10.10.10.10. 513 514 @{ip}= Split String ${i_ipaddress} . 515 ${index}= Set Variable ${0} 516 517 :FOR ${item} IN @{ip} 518 \ ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes 519 \ Set List Value ${ip} ${index} ${hex} 520 \ ${index}= Set Variable ${index + 1} 521 ${ip_hex}= Catenate @{ip} 522 523 [Return] ${ip_hex} 524 525 526BMC CPU Performance Check 527 [Documentation] Minimal 10% of proc should be free in this instance 528 529 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command 530 ... ${bmc_cpu_usage_cmd} 531 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command 532 ... ${bmc_cpu_usage_cmd} 533 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} % 534 Rprint Vars bmc_cpu_percentage 535 Should be true ${bmc_cpu_percentage} < 90 536 537 538BMC Mem Performance Check 539 [Documentation] Minimal 10% of memory should be free in this instance 540 541 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command 542 ... ${bmc_mem_free_cmd} 543 544 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command 545 ... ${bmc_mem_total_cmd} 546 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command 547 ... ${bmc_mem_free_cmd} 548 549 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command 550 ... ${bmc_mem_total_cmd} 551 552 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100 553 ${bmc_mem_percentage}= Evaluate 554 ... ${bmc_mem_percentage}/${bmc_mem_total_output} 555 Rprint Vars bmc_mem_percentage 556 Should be true ${bmc_mem_percentage} > 10 557 558 559BMC File System Usage Check 560 [Documentation] Check the file system space. 4 file system should be 561 ... 100% full which is expected 562 # Filesystem Size Used Available Use% Mounted on 563 # /dev/root 14.4M 14.4M 0 100% / 564 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e 565 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3 566 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c 567 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4 568 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command 569 ... ${bmc_file_system_usage_cmd} 570 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command 571 ... ${total_pnor_ro_file_system_cmd} 572 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command 573 ... ${total_bmc_ro_file_system_cmd} 574 ${total_bmc_pnor_image}= Evaluate 575 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output} 576 # Considering /dev/root also in total 100% used file system 577 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1 578 Rprint Vars bmc_fs_usage_output bmc_pnor_fs_usage_output bmc_bmc_fs_usage_output 579 ... total_full_fs 580 Should Be True ${bmc_fs_usage_output}==${total_full_fs} 581 582 583Check BMC CPU Performance 584 [Documentation] Minimal 10% of proc should be free in 3 sample 585 :FOR ${var} IN RANGE 1 4 586 \ BMC CPU Performance check 587 588 589Check BMC Mem Performance 590 [Documentation] Minimal 10% of memory should be free 591 592 :FOR ${var} IN RANGE 1 4 593 \ BMC Mem Performance check 594 595 596Check BMC File System Performance 597 [Documentation] Check for file system usage for 4 times 598 599 :FOR ${var} IN RANGE 1 4 600 \ BMC File System Usage check 601 602 603Get URL List 604 [Documentation] Return list of URLs under given URL. 605 [Arguments] ${openbmc_url} 606 607 # Description of argument(s): 608 # openbmc_url URL for list operation (e.g. 609 # /xyz/openbmc_project/inventory). 610 611 ${url_list}= Read Properties ${openbmc_url}list quiet=${1} 612 Sort List ${url_list} 613 614 [Return] ${url_list} 615 616 617Check Zombie Process 618 [Documentation] Check if any defunct process exist or not on BMC 619 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l 620 ... return_stderr=True return_rc=True 621 Should Be True ${count}==0 622 Should Be Empty ${stderr} 623 624 625Prune Journal Log 626 [Documentation] Prune archived journal logs. 627 [Arguments] ${vacuum_size}=1M 628 629 # This keyword can be used to prevent the journal 630 # log from filling up the /run filesystem. 631 # This command will retain only the latest logs 632 # of the user specified size. 633 634 # Description of argument(s): 635 # vacuum_size Size of journal. 636 637 Open Connection And Log In 638 ${output} ${stderr} ${rc}= 639 ... Execute Command 640 ... journalctl --vacuum-size=${vacuum_size} 641 ... return_stderr=True return_rc=True 642 643 Should Be Equal ${rc} ${0} msg=${stderr} 644 645 646Get System Power Policy 647 [Documentation] Returns the BMC power policy. 648 649 # Set the bmc_power_policy_method to either 'Old' or 'New'. 650 Set Power Policy Method 651 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy 652 # Run the appropriate keyword. 653 ${currentPolicy}= Run Keyword @{cmd_buf} 654 655 [Return] ${currentPolicy} 656 657 658Set BMC Reset Reference Time 659 [Documentation] Set current boot time as a reference and increment 660 ... boot count. 661 662 ${cur_btime}= Get BMC Boot Time 663 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0} 664 ... Set Global Variable ${BOOT_TIME} ${cur_btime} 665 ... ELSE IF ${cur_btime} > ${BOOT_TIME} 666 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime} 667 ... AND 668 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1} 669 670 671Get BMC Boot Time 672 [Documentation] Returns boot time from /proc/stat. 673 674 Open Connection And Log In 675 ${output} ${stderr}= 676 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' ' 677 ... return_stderr=True 678 Should Be Empty ${stderr} 679 ${btime}= Convert To Integer ${output} 680 [Return] ${btime} 681 682 683Enable Core Dump On BMC 684 [Documentation] Enable core dump collection. 685 ${core_pattern} ${stderr} ${rc}= BMC Execute Command 686 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern 687 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p 688 689 690Get Number Of BMC Core Dump Files 691 [Documentation] Returns number of core dump files on BMC. 692 Open Connection And Log In 693 ${num_of_core_dump}= Execute Command 694 ... ls /tmp/core* 2>/dev/null | wc -l 695 [Return] ${num_of_core_dump} 696 697 698Set Core Dump File Size Unlimited 699 [Documentation] Set core dump file size to unlimited. 700 BMC Execute Command ulimit -c unlimited 701 702 703Check For Core Dumps 704 [Documentation] Check for any core dumps exist. 705 ${output}= Get Number Of BMC Core Dump Files 706 Run Keyword If ${output} > 0 707 ... Log **Warning** BMC core dump files exist level=WARN 708 709 710Configure Initial Settings 711 [Documentation] Restore old IP and route. 712 ... This keyword requires initial settings viz IP address, 713 ... Network Mask, default gatway and serial console IP and port 714 ... information which should be provided in command line. 715 716 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK} 717 ... ${gw_ip}=${GW_IP} 718 719 # Description of arguments: 720 # host IP address of the OS Host. 721 # mask Network mask. 722 # gu_ip Gateway IP address or hostname. 723 724 # Open telnet connection and ignore the error, in case telnet session is 725 # already opened by the program calling this keyword. 726 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console 727 Telnet.write ifconfig eth0 ${host} netmask ${mask} 728 Telnet.write route add default gw ${gw_ip} 729 730 731Install Debug Tarball On BMC 732 [Documentation] Copy the debug tar file to BMC and install. 733 [Arguments] ${tarball_file_path}=${default_tarball} 734 ... ${targ_tarball_dir_path}=/tmp/tarball/ 735 736 # Description of arguments: 737 # tarball_file_path Path of the debug tarball file. 738 # The tar file is downloaded from the build page 739 # https://openpower.xyz/job/openbmc-build/ 740 # obmc-phosphor-debug-tarball-witherspoon.tar.xz 741 # 742 # targ_tarball_dir_path The directory path where the tarball is to be 743 # installed. 744 745 OperatingSystem.File Should Exist ${tarball_file_path} 746 ... msg=${tarball_file_path} doesn't exist. 747 748 # Upload the file to BMC. 749 Import Library SCPLibrary WITH NAME scp 750 Open Connection for SCP 751 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz 752 753 # Create tarball directory and install. 754 BMC Execute Command mkdir -p ${targ_tarball_dir_path} 755 BMC Execute Command 756 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path} 757 758 # Remove the tarball file from BMC. 759 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz 760 761 762Get BMC Boot Count 763 [Documentation] Returns BMC boot count based on boot time. 764 ${cur_btime}= Get BMC Boot Time 765 766 # Set global variable BOOT_TIME to current boot time if current boot time 767 # is changed. Also increase value of global variable BOOT_COUNT by 1. 768 Run Keyword If ${cur_btime} > ${BOOT_TIME} 769 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime} 770 ... AND 771 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1} 772 773 [Return] ${BOOT_COUNT} 774 775 776Set BMC Boot Count 777 [Documentation] Set BMC boot count to given value. 778 [Arguments] ${count} 779 780 # Description of arguments: 781 # count boot count value. 782 ${cur_btime}= Get BMC Boot Time 783 784 # Set global variable BOOT_COUNT to given value. 785 Set Global Variable ${BOOT_COUNT} ${count} 786 787 # Set BOOT_TIME variable to current boot time. 788 Set Global Variable ${BOOT_COUNT} ${count} 789 790 791Delete Error Log Entry 792 [Documentation] Delete error log entry. 793 [Arguments] ${entry_path} 794 795 # Description of argument(s): 796 # entry_path Delete an error log entry. 797 # Ex. /xyz/openbmc_project/logging/entry/1 798 799 # Skip delete if entry URI is a callout. 800 # Examples: 801 # /xyz/openbmc_project/logging/entry/1/callout 802 # /xyz/openbmc_project/logging/entry/1/callouts/0 803 ${callout_entry}= Run Keyword And Return Status 804 ... Should Match Regexp ${entry_path} /callout[s]?(/|$) 805 Return From Keyword If ${callout_entry} 806 807 ${data}= Create Dictionary data=@{EMPTY} 808 ${resp}= Openbmc Delete Request ${entry_path} data=${data} 809 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} 810 811 812Get BMC Version 813 [Documentation] Returns BMC version from /etc/os-release. 814 ... e.g. "v1.99.6-141-ge662190" 815 816 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '=' 817 ${output} ${stderr} ${rc}= BMC Execute Command ${cmd} 818 [Return] ${output} 819 820 821Get PNOR Version 822 [Documentation] Returns the PNOR version from the BMC. 823 824 ${pnor_attrs}= Get PNOR Attributes 825 [Return] ${pnor_attrs['version']} 826 827 828Get PNOR Attributes 829 [Documentation] Return PNOR software attributes as a dictionary. 830 831 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc 832 # into key/value pairs. 833 834 ${outbuf} ${stderr} ${rc}= BMC Execute Command 835 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc 836 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim== 837 838 [Return] ${pnor_attrs} 839 840 841GET BMC PNOR Version 842 [Documentation] Return BMC & PNOR version from openbmc shell. 843 844 ${bmc_version}= GET BMC Version 845 ${pnor_version}= GET PNOR Version 846 Log ${bmc_version} 847 Rprint Vars bmc_version 848 Log ${pnor_version} 849 Rprint Vars pnor_version 850 851 [Return] ${bmc_version} ${pnor_version} 852 853 854Redfish Get BMC Version 855 [Documentation] Get BMC version via Redfish. 856 857 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Managers/bmc FirmwareVersion 858 [Return] ${output} 859 860Redfish Get Host Version 861 [Documentation] Get host version via Redfish. 862 863 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Systems/system BiosVersion 864 [Return] ${output} 865 866 867Copy Address Translation Utils To HOST OS 868 [Documentation] Copy address translation utils to host OS. 869 870 OperatingSystem.File Should Exist ${probe_cpu_tool_path} 871 ... msg=${probe_cpu_tool_path} doesn't exist. 872 OperatingSystem.File Should Exist ${probe_cpu_tool_path} 873 ... msg=${probe_cpu_tool_path} doesn't exist. 874 875 scp.Open connection ${OS_HOST} username=${OS_USERNAME} 876 ... password=${OS_PASSWORD} 877 scp.Put File ${probe_cpu_tool_path} ${target_file_path} 878 scp.Put File ${scom_addrs_tool_path} ${target_file_path} 879 880 881Verify BMC RTC And UTC Time Drift 882 [Documentation] Verify that the RTC and UTC time difference is less than 883 ... the given time_drift_max. 884 [Arguments] ${time_diff_max}=${10} 885 886 # Description of argument(s): 887 # time_diff_max The max allowable RTC and UTC time difference in seconds. 888 889 # Example: 890 # time_dict: 891 # [local_time]: Fri 2017-11-03 152756 UTC 892 # [local_time_seconds]: 1509740876 893 # [universal_time]: Fri 2017-11-03 152756 UTC 894 # [universal_time_seconds]: 1509740876 895 # [rtc_time]: Fri 2016-05-20 163403 896 # [rtc_time_seconds]: 1463780043 897 # [time_zone]: n/a (UTC, +0000) 898 # [network_time_on]: yes 899 # [ntp_synchronized]: no 900 # [rtc_in_local_tz]: no 901 902 ${time}= Get BMC Date Time 903 ${time_diff}= Evaluate 904 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']} 905 Should Be True ${time_diff} < ${time_diff_max} 906 907 908Validate IP On BMC 909 [Documentation] Validate IP address is present in set of IP addresses. 910 [Arguments] ${ip_address} ${ip_data} 911 912 # Description of argument(s): 913 # ip_address IP address to check (e.g. xx.xx.xx.xx). 914 # ip_data Set of the IP addresses present. 915 916 Should Contain Match ${ip_data} ${ip_address}/* 917 ... msg=${ip_address} not found in the list provided. 918 919 920Remove Journald Logs 921 [Documentation] Remove all journald logs and restart service. 922 923 ${cmd}= Catenate systemctl stop systemd-journald.service && 924 ... rm -rf /var/log/journal && systemctl start systemd-journald.service 925 926 BMC Execute Command ${cmd} 927 928 929Check For Regex In Journald 930 [Documentation] Parse the journal log and check for regex string. 931 [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY} 932 933 # Description of argument(s): 934 # regex Strings to be filter. 935 # error_check Check for errors. 936 # boot Argument to check current or persistent full boot log 937 # (e.g. "-b"). 938 939 ${journal_log} ${stderr} ${rc}= BMC Execute Command 940 ... journalctl --no-pager ${boot} | egrep '${regex}' ignore_err=1 941 942 Run Keyword If ${error_check} == ${0} 943 ... Should Be Empty ${journal_log} 944 ... ELSE 945 ... Should Not Be Empty ${journal_log} 946 947 948Get Service Attribute 949 [Documentation] Get service attribute policy output. 950 [Arguments] ${option} ${servicename} 951 952 # Description of argument(s): 953 # option systemctl supported options 954 # servicename Qualified service name 955 ${cmd}= Set Variable 956 ... systemctl -p ${option} show ${servicename} | cut -d = -f2 957 ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd} 958 [Return] ${attr} 959 960 961Verify Watchdog Enabled 962 [Documentation] Check that watchdog settings are enabled. 963 964 ${properties}= Read Properties /xyz/openbmc_project/watchdog/host0 965 Should Be Equal As Strings ${properties["Enabled"]} ${True} 966 Should Not Be Equal As Strings ${properties["TimeRemaining"]} 0 967