1a28061a0SGeorge Keishing*** Settings *** 2a28061a0SGeorge KeishingDocumentation Keywords for system test. 3a28061a0SGeorge Keishing 4a28061a0SGeorge KeishingLibrary ../lib/gen_robot_keyword.py 5a28061a0SGeorge KeishingLibrary ../lib/gen_print.py 6a28061a0SGeorge KeishingLibrary ../lib/gen_robot_print.py 7a28061a0SGeorge KeishingResource ../lib/boot_utils.robot 8a28061a0SGeorge KeishingResource ../extended/obmc_boot_test_resource.robot 9a28061a0SGeorge KeishingResource ../lib/utils.robot 10a28061a0SGeorge KeishingResource ../lib/state_manager.robot 11a28061a0SGeorge KeishingResource ../lib/rest_client.robot 12a28061a0SGeorge KeishingResource ../lib/htx_resource.robot 13a28061a0SGeorge KeishingLibrary OperatingSystem 14a28061a0SGeorge KeishingLibrary DateTime 15a28061a0SGeorge Keishing 16a28061a0SGeorge Keishing*** Variables *** 17a28061a0SGeorge Keishing 18a28061a0SGeorge Keishing${htx_log_dir_path} ${EXECDIR}${/}logs${/} 19a28061a0SGeorge Keishing 20a28061a0SGeorge Keishing# Error strings to check from dmesg. 21a28061a0SGeorge Keishing${ERROR_REGEX} error|GPU|NVRM|nvidia 22a28061a0SGeorge Keishing 23a28061a0SGeorge Keishing# GPU specific error message from dmesg. 24a28061a0SGeorge Keishing${ERROR_DBE_MSG} (DBE) has been detected on GPU 25a28061a0SGeorge Keishing 26a28061a0SGeorge Keishing# Inventory - List of I/O devices to collect for Inventory 27a28061a0SGeorge Keishing@{I/O} communication disk display generic input multimedia 28a28061a0SGeorge Keishing... network printer tape 29a28061a0SGeorge Keishing 30a28061a0SGeorge Keishing# Inventory Paths of the JSON and YAML files 31a28061a0SGeorge Keishing${json_tmp_file_path} ${EXECDIR}/inventory_temp_file.json 32a28061a0SGeorge Keishing${yaml_file_path} ${EXECDIR}/inventory_temp_file.yaml 33a28061a0SGeorge Keishing 34a28061a0SGeorge Keishing 35a28061a0SGeorge Keishing 36a28061a0SGeorge Keishing*** Keywords *** 37a28061a0SGeorge Keishing 38a28061a0SGeorge KeishingLogin To OS 39a28061a0SGeorge Keishing [Documentation] Login to OS Host. 40a28061a0SGeorge Keishing [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME} 41a28061a0SGeorge Keishing ... ${os_password}=${OS_PASSWORD} 42a28061a0SGeorge Keishing ... ${alias_name}=os_connection 43a28061a0SGeorge Keishing # Description of argument(s): 44a28061a0SGeorge Keishing # os_host IP address of the OS Host. 45a28061a0SGeorge Keishing # os_username OS Host Login user name. 46a28061a0SGeorge Keishing # os_password OS Host Login passwrd. 47a28061a0SGeorge Keishing # alias_name Default OS SSH session connection alias name. 48a28061a0SGeorge Keishing 49a28061a0SGeorge Keishing Ping Host ${os_host} 50a28061a0SGeorge Keishing SSHLibrary.Open Connection ${os_host} alias=${alias_name} 51a28061a0SGeorge Keishing SSHLibrary.Login ${os_username} ${os_password} 52a28061a0SGeorge Keishing 53a28061a0SGeorge Keishing 54a28061a0SGeorge KeishingTool Exist 55a28061a0SGeorge Keishing [Documentation] Check whether given tool is installed on OS. 56a28061a0SGeorge Keishing [Arguments] ${tool_name} 57a28061a0SGeorge Keishing # Description of argument(s): 58a28061a0SGeorge Keishing # tool_name Tool name whose existence is to be checked. 59a28061a0SGeorge Keishing 60a28061a0SGeorge Keishing ${output} ${stderr} ${rc}= OS Execute Command which ${tool_name} 61a28061a0SGeorge Keishing Should Contain ${output} ${tool_name} 62a28061a0SGeorge Keishing ... msg=Please install ${tool_name} tool. 63a28061a0SGeorge Keishing 64a28061a0SGeorge Keishing 65a28061a0SGeorge KeishingBoot To OS 66a28061a0SGeorge Keishing [Documentation] Boot host OS. 67a28061a0SGeorge Keishing Run Key OBMC Boot Test \ REST Power On 68a28061a0SGeorge Keishing 69a28061a0SGeorge Keishing 70a28061a0SGeorge KeishingPower Off Host 71a28061a0SGeorge Keishing [Documentation] Power off host. 72a28061a0SGeorge Keishing Run Key OBMC Boot Test \ REST Power Off 73a28061a0SGeorge Keishing 74a28061a0SGeorge Keishing 75a28061a0SGeorge KeishingFile Exist On OS 76a28061a0SGeorge Keishing [Documentation] Check if the given file path exist on OS. 77a28061a0SGeorge Keishing [Arguments] ${file_path} 78a28061a0SGeorge Keishing # Description of argument(s): 79a28061a0SGeorge Keishing # file_path Absolute file path. 80a28061a0SGeorge Keishing 81a28061a0SGeorge Keishing Login To OS 82a28061a0SGeorge Keishing ${out} ${stderr} ${rc}= OS Execute Command ls ${file_path} 83a28061a0SGeorge Keishing Log To Console \n File Exist: ${out} 84a28061a0SGeorge Keishing 85a28061a0SGeorge Keishing 86a28061a0SGeorge KeishingIs HTX Running 87a28061a0SGeorge Keishing [Documentation] Return "True" if the HTX is running, "False" 88a28061a0SGeorge Keishing ... otherwise. 89a28061a0SGeorge Keishing 90a28061a0SGeorge Keishing # Example usage: 91a28061a0SGeorge Keishing # ${status}= Is HTX Running 92a28061a0SGeorge Keishing # Run Keyword If '${status}' == 'True' Shutdown HTX Exerciser 93a28061a0SGeorge Keishing 94a28061a0SGeorge Keishing ${status} ${stderr} ${rc}= OS Execute Command 95a28061a0SGeorge Keishing ... htxcmdline -getstats ignore_err=1 96a28061a0SGeorge Keishing # Get HTX state 97a28061a0SGeorge Keishing # (idle, currently running, selected_mdt but not running). 98a28061a0SGeorge Keishing ${running}= Set Variable If 99a28061a0SGeorge Keishing ... "Currently running" in """${status}""" ${True} ${False} 100a28061a0SGeorge Keishing 101a28061a0SGeorge Keishing [Return] ${running} 102a28061a0SGeorge Keishing 103a28061a0SGeorge Keishing 104a28061a0SGeorge KeishingWrite Log Data To File 105a28061a0SGeorge Keishing [Documentation] Write log data to the logs directory. 106a28061a0SGeorge Keishing [Arguments] ${data}= ${log_file_path}= 107a28061a0SGeorge Keishing # Description of argument(s): 108a28061a0SGeorge Keishing # data String buffer. 109a28061a0SGeorge Keishing # log_file_path The log file path. 110a28061a0SGeorge Keishing 111a28061a0SGeorge Keishing Create File ${log_file_path} ${data} 112a28061a0SGeorge Keishing 113a28061a0SGeorge Keishing 114a28061a0SGeorge KeishingCollect HTX Log Files 115a28061a0SGeorge Keishing [Documentation] Collect status and error log files. 116a28061a0SGeorge Keishing # Collects the following files: 117a28061a0SGeorge Keishing # HTX error log file /tmp/htxerr 118a28061a0SGeorge Keishing # HTX status log file /tmp/htxstats 119a28061a0SGeorge Keishing 120a28061a0SGeorge Keishing # Create logs directory and get current datetime. 121a28061a0SGeorge Keishing Create Directory ${htx_log_dir_path} 122a28061a0SGeorge Keishing ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f 123a28061a0SGeorge Keishing 124a28061a0SGeorge Keishing File Exist On OS /tmp/htxerr 125a28061a0SGeorge Keishing ${htx_err} ${std_err} ${rc}= OS Execute Command cat /tmp/htxerr 126a28061a0SGeorge Keishing Write Log Data To File 127a28061a0SGeorge Keishing ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr 128a28061a0SGeorge Keishing 129a28061a0SGeorge Keishing File Exist On OS /tmp/htxstats 130a28061a0SGeorge Keishing ${htx_stats} ${std_err} ${rc}= OS Execute Command 131a28061a0SGeorge Keishing ... cat /tmp/htxstats 132a28061a0SGeorge Keishing Write Log Data To File 133a28061a0SGeorge Keishing ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats 134a28061a0SGeorge Keishing 135a28061a0SGeorge Keishing 136a28061a0SGeorge KeishingREST Upload File To BMC 137a28061a0SGeorge Keishing [Documentation] Upload a file via REST to BMC. 138a28061a0SGeorge Keishing 139a28061a0SGeorge Keishing # Generate 32 MB file size 140a28061a0SGeorge Keishing Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB 141a28061a0SGeorge Keishing OperatingSystem.File Should Exist dummyfile 142a28061a0SGeorge Keishing 143a28061a0SGeorge Keishing # Get the content of the file and upload to BMC 144a28061a0SGeorge Keishing ${image_data}= OperatingSystem.Get Binary File dummyfile 145a28061a0SGeorge Keishing 146a28061a0SGeorge Keishing # Get REST session to BMC 147a28061a0SGeorge Keishing Initialize OpenBMC 148a28061a0SGeorge Keishing 149a28061a0SGeorge Keishing # Create the REST payload headers and data 150a28061a0SGeorge Keishing ${data}= Create Dictionary data ${image_data} 151a28061a0SGeorge Keishing ${headers}= Create Dictionary Content-Type=application/octet-stream 152a28061a0SGeorge Keishing ... Accept=application/octet-stream 153a28061a0SGeorge Keishing Set To Dictionary ${data} headers ${headers} 154a28061a0SGeorge Keishing 155a28061a0SGeorge Keishing ${resp}= Post Request openbmc /upload/image &{data} 156a28061a0SGeorge Keishing Should Be Equal As Strings ${resp.status_code} ${HTTP_BAD_REQUEST} 157a28061a0SGeorge Keishing ... msg=Openbmc /upload/image failed. 158a28061a0SGeorge Keishing 159a28061a0SGeorge Keishing # Take SSH connection to BMC and switch to BMC connection to perform 160a28061a0SGeorge Keishing # the task. 161a28061a0SGeorge Keishing &{bmc_connection_args}= Create Dictionary alias=bmc_connection 162a28061a0SGeorge Keishing Open Connection And Log In &{bmc_connection_args} 163a28061a0SGeorge Keishing 164a28061a0SGeorge Keishing # Currently OS SSH session is active, switch to BMC connection. 165a28061a0SGeorge Keishing Switch Connection bmc_connection 166a28061a0SGeorge Keishing 167a28061a0SGeorge Keishing # Switch back to OS SSH connection. 168a28061a0SGeorge Keishing Switch Connection os_connection 169a28061a0SGeorge Keishing 170a28061a0SGeorge Keishing 171a28061a0SGeorge KeishingGet CPU Min Frequency Limit 172a28061a0SGeorge Keishing [Documentation] Get CPU minimum assignable frequency. 173a28061a0SGeorge Keishing 174a28061a0SGeorge Keishing # lscpu | grep min returns 175a28061a0SGeorge Keishing # CPU min MHz: 1983.0000 176a28061a0SGeorge Keishing 177a28061a0SGeorge Keishing ${cmd}= Catenate lscpu | grep min | tr -dc '0-9.\n' 178a28061a0SGeorge Keishing ${cpu_freq} ${stderr} ${rc}= OS Execute Command ${cmd} 179a28061a0SGeorge Keishing [Return] ${cpu_freq} 180a28061a0SGeorge Keishing 181a28061a0SGeorge Keishing 182a28061a0SGeorge KeishingGet CPU Min Frequency 183a28061a0SGeorge Keishing [Documentation] Get CPU assigned minimum frequency. 184a28061a0SGeorge Keishing 185a28061a0SGeorge Keishing # ppc64_cpu --frequency -t 10 returns 186a28061a0SGeorge Keishing # min: 3.295 GHz (cpu 143) 187a28061a0SGeorge Keishing # max: 3.295 GHz (cpu 0) 188a28061a0SGeorge Keishing # avg: 3.295 GHz 189a28061a0SGeorge Keishing 190a28061a0SGeorge Keishing ${cmd}= Catenate ppc64_cpu --frequency -t 10 | grep min 191a28061a0SGeorge Keishing ... | cut -f 2 | cut -d ' ' -f 1 | tr -dc '0-9\n' 192a28061a0SGeorge Keishing ${cpu_freq} ${stderr} ${rc}= OS Execute Command ${cmd} 193a28061a0SGeorge Keishing [Return] ${cpu_freq} 194a28061a0SGeorge Keishing 195a28061a0SGeorge Keishing 196a28061a0SGeorge KeishingGet CPU Max Frequency Limit 197a28061a0SGeorge Keishing [Documentation] Get CPU maximum assignable frequency. 198a28061a0SGeorge Keishing 199a28061a0SGeorge Keishing # lscpu | grep max returns 200a28061a0SGeorge Keishing # CPU max MHz: 3300.0000 201a28061a0SGeorge Keishing 202a28061a0SGeorge Keishing ${cmd}= Catenate lscpu | grep max | tr -dc '0-9.\n' 203a28061a0SGeorge Keishing ${cpu_freq} ${stderr} ${rc}= OS Execute Command ${cmd} 204a28061a0SGeorge Keishing [Return] ${cpu_freq} 205a28061a0SGeorge Keishing 206a28061a0SGeorge Keishing 207a28061a0SGeorge KeishingGet CPU Max Frequency 208a28061a0SGeorge Keishing [Documentation] Get CPU assigned maximum frequency. 209a28061a0SGeorge Keishing 210a28061a0SGeorge Keishing # ppc64_cpu --frequency -t 10 returns 211a28061a0SGeorge Keishing # min: 3.295 GHz (cpu 143) 212a28061a0SGeorge Keishing # max: 3.295 GHz (cpu 0) 213a28061a0SGeorge Keishing # avg: 3.295 GHz 214a28061a0SGeorge Keishing 215a28061a0SGeorge Keishing ${cmd}= Catenate ppc64_cpu --frequency -t 10 | grep max 216a28061a0SGeorge Keishing ... | cut -f 2 | cut -d ' ' -f 1 | tr -dc '0-9\n' 217a28061a0SGeorge Keishing ${cpu_freq} ${stderr} ${rc}= OS Execute Command ${cmd} 218a28061a0SGeorge Keishing [Return] ${cpu_freq} 219a28061a0SGeorge Keishing 220a28061a0SGeorge Keishing 221a28061a0SGeorge KeishingGet CPU Max Temperature 222a28061a0SGeorge Keishing [Documentation] Get the highest CPU Temperature. 223a28061a0SGeorge Keishing 224a28061a0SGeorge Keishing ${temperature_objs}= Read Properties 225a28061a0SGeorge Keishing ... ${SENSORS_URI}temperature/enumerate 226a28061a0SGeorge Keishing # Filter the dictionary to get just the CPU temperature info. 227a28061a0SGeorge Keishing ${cmd}= Catenate {k:v for k,v in $temperature_objs.items() 228a28061a0SGeorge Keishing ... if re.match('${SENSORS_URI}temperature/p.*core.*temp', k)} 229a28061a0SGeorge Keishing ${cpu_temperatuture_objs} Evaluate ${cmd} modules=re 230a28061a0SGeorge Keishing # Create a list of the CPU temperature values (current). 231a28061a0SGeorge Keishing ${cpu_temperatures}= Evaluate 232a28061a0SGeorge Keishing ... [ x['Value'] for x in $cpu_temperatuture_objs.values() ] 233a28061a0SGeorge Keishing 234a28061a0SGeorge Keishing ${cpu_max_temp} Evaluate int(max(map(int, $cpu_temperatures))/1000) 235a28061a0SGeorge Keishing [Return] ${cpu_max_temp} 236a28061a0SGeorge Keishing 237a28061a0SGeorge Keishing 238a28061a0SGeorge KeishingGet CPU Min Temperature 239a28061a0SGeorge Keishing [Documentation] Get the CPU Temperature. 240a28061a0SGeorge Keishing 241a28061a0SGeorge Keishing ${temperature_objs}= Read Properties 242a28061a0SGeorge Keishing ... ${SENSORS_URI}temperature/enumerate 243a28061a0SGeorge Keishing # Filter the dictionary to get just the CPU temperature info. 244a28061a0SGeorge Keishing ${cmd}= Catenate {k:v for k,v in $temperature_objs.items() 245a28061a0SGeorge Keishing ... if re.match('${SENSORS_URI}temperature/p.*core.*temp', k)} 246a28061a0SGeorge Keishing ${cpu_temperatuture_objs}= Evaluate ${cmd} modules=re 247a28061a0SGeorge Keishing # Create a list of the CPU temperature values (current). 248a28061a0SGeorge Keishing ${cpu_temperatures}= Evaluate 249a28061a0SGeorge Keishing ... [ x['Value'] for x in $cpu_temperatuture_objs.values() ] 250a28061a0SGeorge Keishing 251a28061a0SGeorge Keishing ${cpu_min_temp} Evaluate int(min(map(int, $cpu_temperatures))/1000) 252a28061a0SGeorge Keishing [Return] ${cpu_min_temp} 253a28061a0SGeorge Keishing 254a28061a0SGeorge Keishing 255a28061a0SGeorge KeishingCheck For Errors On OS Dmesg Log 256a28061a0SGeorge Keishing [Documentation] Check if dmesg has nvidia errors logged. 257a28061a0SGeorge Keishing 258a28061a0SGeorge Keishing ${dmesg_log} ${stderr} ${rc}= OS Execute Command 259a28061a0SGeorge Keishing ... dmesg | egrep '${ERROR_REGEX}' 260a28061a0SGeorge Keishing # To enable multiple string check. 261a28061a0SGeorge Keishing Should Not Contain ${dmesg_log} ${ERROR_DBE_MSG} 262a28061a0SGeorge Keishing ... msg=OS dmesg shows ${ERROR_DBE_MSG}. 263a28061a0SGeorge Keishing 264a28061a0SGeorge Keishing 265a28061a0SGeorge KeishingCollect NVIDIA Log File 266a28061a0SGeorge Keishing [Documentation] Collect ndivia-smi command output. 267a28061a0SGeorge Keishing [Arguments] ${suffix} 268a28061a0SGeorge Keishing # Description of argument(s): 269a28061a0SGeorge Keishing # suffix String name to append. 270a28061a0SGeorge Keishing 271a28061a0SGeorge Keishing # Collects the output of ndivia-smi cmd output. 272a28061a0SGeorge Keishing # +--------------------------------------------------------------------+ 273a28061a0SGeorge Keishing # | NVIDIA-SMI 361.89 Driver Version: 361.89 | 274a28061a0SGeorge Keishing # |-------------------------------+----------------------+-------------+ 275a28061a0SGeorge Keishing # | GPU Name Persistence-M| Bus-Id Disp.A | GPU ECC | 276a28061a0SGeorge Keishing # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | Utiliz err | 277a28061a0SGeorge Keishing # |===============================+======================+=============| 278a28061a0SGeorge Keishing # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 | 279a28061a0SGeorge Keishing # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default | 280a28061a0SGeorge Keishing # +-------------------------------+----------------------+-------------+ 281a28061a0SGeorge Keishing # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 | 282a28061a0SGeorge Keishing # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default | 283a28061a0SGeorge Keishing # +-------------------------------+----------------------+-------------+ 284a28061a0SGeorge Keishing # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 | 285a28061a0SGeorge Keishing # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default | 286a28061a0SGeorge Keishing # +-------------------------------+----------------------+-------------+ 287a28061a0SGeorge Keishing # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 | 288a28061a0SGeorge Keishing # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 0% Default | 289a28061a0SGeorge Keishing # +-------------------------------+----------------------+-------------+ 290a28061a0SGeorge Keishing # +--------------------------------------------------------------------+ 291a28061a0SGeorge Keishing # | Processes: GPU Memory | 292a28061a0SGeorge Keishing # | GPU PID Type Process name Usage | 293a28061a0SGeorge Keishing # |====================================================================| 294a28061a0SGeorge Keishing # | 0 28459 C hxenvidia 929MiB | 295a28061a0SGeorge Keishing # | 1 28460 C hxenvidia 1475MiB | 296a28061a0SGeorge Keishing # | 2 28461 C hxenvidia 929MiB | 297a28061a0SGeorge Keishing # | 3 28462 C hxenvidia 963MiB | 298a28061a0SGeorge Keishing # +--------------------------------------------------------------------+ 299a28061a0SGeorge Keishing 300a28061a0SGeorge Keishing # Create logs directory and get current datetime. 301a28061a0SGeorge Keishing Create Directory ${htx_log_dir_path} 302a28061a0SGeorge Keishing ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f 303a28061a0SGeorge Keishing 304a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command nvidia-smi 305a28061a0SGeorge Keishing Write Log Data To File 306a28061a0SGeorge Keishing ... ${nvidia_out} 307a28061a0SGeorge Keishing ... ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix} 308a28061a0SGeorge Keishing 309a28061a0SGeorge Keishing 310a28061a0SGeorge KeishingGet GPU Power Limit 311a28061a0SGeorge Keishing [Documentation] Get NVIDIA GPU maximum permitted power draw. 312a28061a0SGeorge Keishing 313a28061a0SGeorge Keishing # nvidia-smi --query-gpu=power.limit --format=csv returns 314a28061a0SGeorge Keishing # power.limit [W] 315a28061a0SGeorge Keishing # 300.00 W 316a28061a0SGeorge Keishing # 300.00 W 317a28061a0SGeorge Keishing # 300.00 W 318a28061a0SGeorge Keishing # 300.00 W 319a28061a0SGeorge Keishing 320a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=power.limit 321a28061a0SGeorge Keishing ... --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 322a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 323a28061a0SGeorge Keishing # Allow for sensor overshoot. That is, max power reported for 324a28061a0SGeorge Keishing # a GPU could be a few watts above the limit. 325a28061a0SGeorge Keishing ${power_max}= Evaluate ${nvidia_out}+${7.00} 326a28061a0SGeorge Keishing [Return] ${power_max} 327a28061a0SGeorge Keishing 328a28061a0SGeorge Keishing 329a28061a0SGeorge KeishingGet GPU Max Power 330a28061a0SGeorge Keishing [Documentation] Get the maximum GPU power dissipation. 331a28061a0SGeorge Keishing 332a28061a0SGeorge Keishing # nvidia-smi --query-gpu=power.draw --format=csv returns 333a28061a0SGeorge Keishing # power.draw [W] 334a28061a0SGeorge Keishing # 34.12 W 335a28061a0SGeorge Keishing # 34.40 W 336a28061a0SGeorge Keishing # 36.55 W 337a28061a0SGeorge Keishing # 36.05 W 338a28061a0SGeorge Keishing 339a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=power.draw 340a28061a0SGeorge Keishing ... --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 341a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 342a28061a0SGeorge Keishing [Return] ${nvidia_out} 343a28061a0SGeorge Keishing 344a28061a0SGeorge Keishing 345a28061a0SGeorge KeishingGet GPU Min Power 346a28061a0SGeorge Keishing [Documentation] Return the minimum GPU power value as record by 347a28061a0SGeorge Keishing ... nvidia-smi. 348a28061a0SGeorge Keishing 349a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=power.draw --format=csv | 350a28061a0SGeorge Keishing ... grep -v 'power.draw' | cut -f 1 -d ' ' | sort -n -u | head -1 351a28061a0SGeorge Keishing ${gpu_min_power} ${stderr} ${rc}= OS Execute Command ${cmd} 352a28061a0SGeorge Keishing [Return] ${gpu_min_power} 353a28061a0SGeorge Keishing 354a28061a0SGeorge Keishing 355a28061a0SGeorge KeishingGet GPU Temperature Limit 356a28061a0SGeorge Keishing [Documentation] Get NVIDIA GPU maximum permitted temperature. 357a28061a0SGeorge Keishing 358a28061a0SGeorge Keishing # nvidia-smi -q -d TEMPERATURE | grep "GPU Max" returns 359a28061a0SGeorge Keishing # GPU Max Operating Temp : 83 C 360a28061a0SGeorge Keishing # GPU Max Operating Temp : 83 C 361a28061a0SGeorge Keishing # GPU Max Operating Temp : 83 C 362a28061a0SGeorge Keishing # GPU Max Operating Temp : 83 C 363a28061a0SGeorge Keishing 364a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi -q -d TEMPERATURE | grep "GPU Max" 365a28061a0SGeorge Keishing ... | cut -f 2 -d ":" | tr -dc '0-9\n' | sort -n -u | tail -n 1 366a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 367a28061a0SGeorge Keishing [Return] ${nvidia_out} 368a28061a0SGeorge Keishing 369a28061a0SGeorge Keishing 370a28061a0SGeorge KeishingGet GPU Min Temperature 371a28061a0SGeorge Keishing [Documentation] Get the minimum GPU temperature. 372a28061a0SGeorge Keishing 373a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=temperature.gpu 374a28061a0SGeorge Keishing ... --format=csv | grep -v 'temperature.gpu' | sort -n -u | head -1 375a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 376a28061a0SGeorge Keishing [Return] ${nvidia_out} 377a28061a0SGeorge Keishing 378a28061a0SGeorge Keishing 379a28061a0SGeorge KeishingGet GPU Max Temperature 380a28061a0SGeorge Keishing [Documentation] Get the maximum GPU temperature. 381a28061a0SGeorge Keishing 382a28061a0SGeorge Keishing # nvidia-smi --query-gpu=temperature.gpu --format=csv returns 383a28061a0SGeorge Keishing # 38 384a28061a0SGeorge Keishing # 41 385a28061a0SGeorge Keishing # 38 386a28061a0SGeorge Keishing # 40 387a28061a0SGeorge Keishing 388a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=temperature.gpu 389a28061a0SGeorge Keishing ... --format=csv | sort -n -u | tail -n 1 390a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 391a28061a0SGeorge Keishing [Return] ${nvidia_out} 392a28061a0SGeorge Keishing 393a28061a0SGeorge Keishing 394a28061a0SGeorge KeishingGet GPU Temperature Via REST 395a28061a0SGeorge Keishing [Documentation] Return the temperature in degrees C of the warmest GPU 396a28061a0SGeorge Keishing ... as reportd by REST. 397a28061a0SGeorge Keishing 398a28061a0SGeorge Keishing # NOTE: This endpoint path is not defined until system has been powered-on. 399a28061a0SGeorge Keishing ${temperature_objs}= Read Properties ${SENSORS_URI}temperature/enumerate 400a28061a0SGeorge Keishing ... timeout=30 quiet=1 401a28061a0SGeorge Keishing 402a28061a0SGeorge Keishing ${core_temperatures_list}= Catenate {k:v for k,v in $temperature_objs.items() 403a28061a0SGeorge Keishing ... if re.match('${SENSORS_URI}temperature/.*_core_temp', k)} 404a28061a0SGeorge Keishing ${gpu_temperature_objs_dict}= Evaluate ${core_temperatures_list} modules=re 405a28061a0SGeorge Keishing 406a28061a0SGeorge Keishing # Create a list containing all of the GPU temperatures. 407a28061a0SGeorge Keishing ${gpu_temperatures}= Evaluate 408a28061a0SGeorge Keishing ... [ x['Value'] for x in $gpu_temperature_objs_dict.values() ] 409a28061a0SGeorge Keishing 410a28061a0SGeorge Keishing # Find the max temperature value and divide by 1000 to get just the integer 411a28061a0SGeorge Keishing # portion. 412a28061a0SGeorge Keishing ${max_gpu_temperature}= Evaluate 413a28061a0SGeorge Keishing ... int(max(map(int, $gpu_temperatures))/1000) 414a28061a0SGeorge Keishing 415a28061a0SGeorge Keishing [Return] ${max_gpu_temperature} 416a28061a0SGeorge Keishing 417a28061a0SGeorge Keishing 418a28061a0SGeorge KeishingGet GPU Clock Limit 419a28061a0SGeorge Keishing [Documentation] Get NVIDIA GPU maximum permitted graphics clock. 420a28061a0SGeorge Keishing 421a28061a0SGeorge Keishing # nvidia-smi --query-gpu=clocks.max.gr --format=csv returns 422a28061a0SGeorge Keishing # 1530 MHz 423a28061a0SGeorge Keishing # 1530 MHz 424a28061a0SGeorge Keishing # 1530 MHz 425a28061a0SGeorge Keishing # 1530 MHz 426a28061a0SGeorge Keishing 427a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=clocks.max.gr 428a28061a0SGeorge Keishing ... --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 429a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 430a28061a0SGeorge Keishing [Return] ${nvidia_out} 431a28061a0SGeorge Keishing 432a28061a0SGeorge Keishing 433a28061a0SGeorge KeishingGet GPU Clock 434a28061a0SGeorge Keishing [Documentation] Get the highest assigned value of the GPU graphics clock. 435a28061a0SGeorge Keishing 436a28061a0SGeorge Keishing # nvidia-smi --query-gpu=clocks.gr --format=csv returns 437a28061a0SGeorge Keishing # 1230 MHz 438a28061a0SGeorge Keishing # 1230 MHz 439a28061a0SGeorge Keishing # 135 MHz 440a28061a0SGeorge Keishing # 150 MHz 441a28061a0SGeorge Keishing 442a28061a0SGeorge Keishing ${cmd}= Catenate nvidia-smi --query-gpu=clocks.gr 443a28061a0SGeorge Keishing ... --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 444a28061a0SGeorge Keishing ${nvidia_out} ${stderr} ${rc}= OS Execute Command ${cmd} 445a28061a0SGeorge Keishing [Return] ${nvidia_out} 446a28061a0SGeorge Keishing 447a28061a0SGeorge Keishing 448a28061a0SGeorge KeishingCount GPUs From BMC 449a28061a0SGeorge Keishing [Documentation] Determine number of GPUs from the BMC. Hostboot 450a28061a0SGeorge Keishing ... needs to have been run previously because the BMC gets GPU data 451a28061a0SGeorge Keishing ... from Hostboot. 452a28061a0SGeorge Keishing 453a28061a0SGeorge Keishing # Example of gv* endpoint data: 454a28061a0SGeorge Keishing # "/xyz/openbmc_project/inventory/system/chassis/motherboard/gv100card0": { 455a28061a0SGeorge Keishing # "Functional": 1, 456a28061a0SGeorge Keishing # "Present": 1, 457a28061a0SGeorge Keishing # "PrettyName": "" 458a28061a0SGeorge Keishing # }, 459a28061a0SGeorge Keishing 460a28061a0SGeorge Keishing ${num_bmc_gpus}= Set Variable ${0} 461a28061a0SGeorge Keishing 462a28061a0SGeorge Keishing ${gpu_list}= Get Endpoint Paths 463a28061a0SGeorge Keishing ... ${HOST_INVENTORY_URI}system/chassis/motherboard gv* 464a28061a0SGeorge Keishing 465a28061a0SGeorge Keishing FOR ${gpu_uri} IN @{gpu_list} 466a28061a0SGeorge Keishing ${present}= Read Attribute ${gpu_uri} Present 467a28061a0SGeorge Keishing ${state}= Read Attribute ${gpu_uri} Functional 468a28061a0SGeorge Keishing Rpvars gpu_uri present state 469a28061a0SGeorge Keishing ${num_bmc_gpus}= Run Keyword If ${present} and ${state} 470a28061a0SGeorge Keishing ... Evaluate ${num_bmc_gpus}+${1} 471a28061a0SGeorge Keishing END 472a28061a0SGeorge Keishing [Return] ${num_bmc_gpus} 473a28061a0SGeorge Keishing 474a28061a0SGeorge Keishing 475a28061a0SGeorge KeishingCreate Default MDT Profile 476a28061a0SGeorge Keishing [Documentation] Create default mdt.bu profile and run. 477a28061a0SGeorge Keishing 478a28061a0SGeorge Keishing Print Timen Create HTX mdt profile. 479a28061a0SGeorge Keishing 480a28061a0SGeorge Keishing ${profile} ${stderr} ${rc}= OS Execute Command 481a28061a0SGeorge Keishing ... htxcmdline -createmdt 482a28061a0SGeorge Keishing Printn ${profile} 483a28061a0SGeorge Keishing Should Contain ${profile} mdts are created successfully 484a28061a0SGeorge Keishing ... msg=Create MDT profile failed. response=${profile} 485a28061a0SGeorge Keishing 486a28061a0SGeorge Keishing 487a28061a0SGeorge KeishingRun MDT Profile 488a28061a0SGeorge Keishing [Documentation] Load user pre-defined MDT profile. 489a28061a0SGeorge Keishing [Arguments] ${HTX_MDT_PROFILE}=${HTX_MDT_PROFILE} 490a28061a0SGeorge Keishing # Description of argument(s): 491a28061a0SGeorge Keishing # HTX_MDT_PROFILE MDT profile to be executed (e.g. "mdt.bu"). 492a28061a0SGeorge Keishing 493a28061a0SGeorge Keishing Print Timen Start HTX mdt profile execution. 494a28061a0SGeorge Keishing ${htx_run} ${stderr} ${rc}= OS Execute Command 495a28061a0SGeorge Keishing ... htxcmdline -run -mdt ${HTX_MDT_PROFILE} 496a28061a0SGeorge Keishing Printn ${htx_run} 497a28061a0SGeorge Keishing Should Contain ${htx_run} Activated 498a28061a0SGeorge Keishing ... msg=htxcmdline run mdt did not return "Activated" status. 499a28061a0SGeorge Keishing 500a28061a0SGeorge Keishing 501a28061a0SGeorge KeishingCheck HTX Run Status 502a28061a0SGeorge Keishing [Documentation] Get HTX exerciser status and check for error. 503a28061a0SGeorge Keishing [Arguments] ${sleep_time}=${0} 504a28061a0SGeorge Keishing 505a28061a0SGeorge Keishing # Description of argument(s): 506a28061a0SGeorge Keishing # sleep_time The amount of time to sleep after checking status, 507a28061a0SGeorge Keishing # for example "3s" or "2m". 508a28061a0SGeorge Keishing 509a28061a0SGeorge Keishing Print Timen Check HTX mdt Status and error. 510a28061a0SGeorge Keishing ${htx_status} ${stderr} ${rc}= OS Execute Command 511a28061a0SGeorge Keishing ... htxcmdline -status -mdt ${HTX_MDT_PROFILE} 512a28061a0SGeorge Keishing Printn ${htx_status} 513a28061a0SGeorge Keishing 514a28061a0SGeorge Keishing ${htx_errlog} ${stderr} ${rc}= OS Execute Command 515a28061a0SGeorge Keishing ... htxcmdline -geterrlog 516a28061a0SGeorge Keishing Printn ${htx_errlog} 517a28061a0SGeorge Keishing 518a28061a0SGeorge Keishing Should Contain ${htx_errlog} file </tmp/htx/htxerr> is empty 519a28061a0SGeorge Keishing ... msg=HTX geterrorlog was not empty. 520a28061a0SGeorge Keishing 521a28061a0SGeorge Keishing Return From Keyword If "${sleep_time}" == "${0}" 522a28061a0SGeorge Keishing 523a28061a0SGeorge Keishing Run Key U Sleep \ ${sleep_time} 524a28061a0SGeorge Keishing 525a28061a0SGeorge Keishing 526a28061a0SGeorge KeishingShutdown HTX Exerciser 527a28061a0SGeorge Keishing [Documentation] Shut down HTX exerciser run. 528a28061a0SGeorge Keishing 529a28061a0SGeorge Keishing Print Timen Shutdown HTX Run 530a28061a0SGeorge Keishing ${shutdown} ${stderr} ${rc}= OS Execute Command 531a28061a0SGeorge Keishing ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE} 532a28061a0SGeorge Keishing 533a28061a0SGeorge Keishing Printn ${shutdown} 534a28061a0SGeorge Keishing 535a28061a0SGeorge Keishing ${down1}= Evaluate 'shutdown successfully' in $shutdown 536a28061a0SGeorge Keishing Return From Keyword If '${down1}' == 'True' 537a28061a0SGeorge Keishing ${down2}= Evaluate 'No MDT is currently running' in $shutdown 538a28061a0SGeorge Keishing Return From Keyword If '${down2}' == 'True' 539a28061a0SGeorge Keishing Fail msg=Shutdown returned unexpected message. 540a28061a0SGeorge Keishing 541a28061a0SGeorge Keishing 542a28061a0SGeorge KeishingCreate JSON Inventory File 543a28061a0SGeorge Keishing [Documentation] Create a JSON inventory file, and make a YAML copy. 544a28061a0SGeorge Keishing [Arguments] ${json_file_path} 545a28061a0SGeorge Keishing # Description of argument: 546*9bfdf8a7SGeorge Keishing # json_file_path Where the inventory file is written to. 547a28061a0SGeorge Keishing 548a28061a0SGeorge Keishing Login To OS 549a28061a0SGeorge Keishing Compile Inventory JSON 550a28061a0SGeorge Keishing Run json2yaml ${json_tmp_file_path} ${yaml_file_path} 551a28061a0SGeorge Keishing # Format to JSON pretty print to file. 552a28061a0SGeorge Keishing Run python -m json.tool ${json_tmp_file_path} > ${json_file_path} 553a28061a0SGeorge Keishing OperatingSystem.File Should Exist ${json_file_path} 554a28061a0SGeorge Keishing ... msg=File ${json_file_path} does not exist. 555a28061a0SGeorge Keishing 556a28061a0SGeorge Keishing 557a28061a0SGeorge KeishingCompile Inventory JSON 558a28061a0SGeorge Keishing [Documentation] Compile the Inventory into a JSON file. 559a28061a0SGeorge Keishing Create File ${json_tmp_file_path} 560a28061a0SGeorge Keishing Write New JSON List ${json_tmp_file_path} Inventory 561a28061a0SGeorge Keishing Retrieve HW Info And Write processor ${json_tmp_file_path} 562a28061a0SGeorge Keishing Retrieve HW Info And Write memory ${json_tmp_file_path} 563a28061a0SGeorge Keishing Retrieve HW Info And Write List ${I/O} ${json_tmp_file_path} I/O last 564a28061a0SGeorge Keishing Close New JSON List ${json_tmp_file_path} 565a28061a0SGeorge Keishing 566a28061a0SGeorge Keishing 567a28061a0SGeorge KeishingWrite New JSON List 568a28061a0SGeorge Keishing [Documentation] Start a new JSON list element in file. 569a28061a0SGeorge Keishing [Arguments] ${json_tmp_file_path} ${json_field_name} 570a28061a0SGeorge Keishing # Description of argument(s): 571a28061a0SGeorge Keishing # json_tmp_file_path Name of file to write to. 572a28061a0SGeorge Keishing # json_field_name Name to give json list element. 573a28061a0SGeorge Keishing Append to File ${json_tmp_file_path} { "${json_field_name}" : [ 574a28061a0SGeorge Keishing 575a28061a0SGeorge Keishing 576a28061a0SGeorge KeishingClose New JSON List 577a28061a0SGeorge Keishing [Documentation] Close JSON list element in file. 578a28061a0SGeorge Keishing [Arguments] ${json_tmp_file_path} 579a28061a0SGeorge Keishing # Description of argument(s): 580a28061a0SGeorge Keishing # json_tmp_file_path Path of file to write to. 581a28061a0SGeorge Keishing Append to File ${json_tmp_file_path} ]} 582a28061a0SGeorge Keishing 583a28061a0SGeorge Keishing 584a28061a0SGeorge KeishingRetrieve HW Info And Write 585a28061a0SGeorge Keishing [Documentation] Retrieve and write info, add a comma if not last item. 586a28061a0SGeorge Keishing [Arguments] ${class} ${json_tmp_file_path} ${last}=false 587a28061a0SGeorge Keishing # Description of argument(s): 588a28061a0SGeorge Keishing # class Device class to retrieve with lshw. 589a28061a0SGeorge Keishing # json_tmp_file_path Path of file to write to. 590a28061a0SGeorge Keishing # last Is this the last element in the parent JSON? 591a28061a0SGeorge Keishing Write New JSON List ${json_tmp_file_path} ${class} 592a28061a0SGeorge Keishing ${output}= Retrieve Hardware Info ${class} 593a28061a0SGeorge Keishing ${output}= Clean Up String ${output} 594a28061a0SGeorge Keishing Run Keyword if ${output.__class__ is not type(None)} 595a28061a0SGeorge Keishing ... Append To File ${json_tmp_file_path} ${output} 596a28061a0SGeorge Keishing Close New JSON List ${json_tmp_file_path} 597a28061a0SGeorge Keishing Run Keyword if '${last}' == 'false' 598a28061a0SGeorge Keishing ... Append to File ${json_tmp_file_path} , 599a28061a0SGeorge Keishing 600a28061a0SGeorge Keishing 601a28061a0SGeorge KeishingRetrieve HW Info And Write List 602a28061a0SGeorge Keishing [Documentation] Does a Retrieve/Write with a list of classes and 603a28061a0SGeorge Keishing ... encapsulates them into one large JSON element. 604a28061a0SGeorge Keishing [Arguments] ${list} ${json_tmp_file_path} ${json_field_name} 605a28061a0SGeorge Keishing ... ${last}=false 606a28061a0SGeorge Keishing # Description of argument(s): 607a28061a0SGeorge Keishing # list The list of devices classes to retrieve with lshw. 608a28061a0SGeorge Keishing # json_tmp_file_path Path of file to write to. 609a28061a0SGeorge Keishing # json_field_name Name of the JSON element to encapsulate this list. 610a28061a0SGeorge Keishing # last Is this the last element in the parent JSON? 611a28061a0SGeorge Keishing Write New JSON List ${json_tmp_file_path} ${json_field_name} 612a28061a0SGeorge Keishing FOR ${class} IN @{list} 613a28061a0SGeorge Keishing ${tail} Get From List ${list} -1 614a28061a0SGeorge Keishing Run Keyword if '${tail}' == '${class}' 615a28061a0SGeorge Keishing ... Retrieve HW Info And Write ${class} ${json_tmp_file_path} true 616a28061a0SGeorge Keishing ... ELSE Retrieve HW Info And Write ${class} ${json_tmp_file_path} 617a28061a0SGeorge Keishing END 618a28061a0SGeorge Keishing Close New JSON List ${json_tmp_file_path} 619a28061a0SGeorge Keishing Run Keyword if '${last}' == 'false' 620a28061a0SGeorge Keishing ... Append to File ${json_tmp_file_path} , 621a28061a0SGeorge Keishing 622a28061a0SGeorge Keishing 623a28061a0SGeorge KeishingRetrieve Hardware Info 624a28061a0SGeorge Keishing [Documentation] Retrieves the lshw output of the device class as JSON. 625a28061a0SGeorge Keishing [Arguments] ${class} 626a28061a0SGeorge Keishing # Description of argument(s): 627a28061a0SGeorge Keishing # class Device class to retrieve with lshw. 628a28061a0SGeorge Keishing ${output} ${stderr} ${rc}= OS Execute Command lshw -c ${class} -json 629a28061a0SGeorge Keishing ${output}= Verify JSON string ${output} 630a28061a0SGeorge Keishing [Return] ${output} 631a28061a0SGeorge Keishing 632a28061a0SGeorge Keishing 633a28061a0SGeorge KeishingVerify JSON String 634a28061a0SGeorge Keishing [Documentation] Ensure the JSON string content is separated by commas. 635a28061a0SGeorge Keishing [Arguments] ${unver_string} 636a28061a0SGeorge Keishing # Description of argument(s): 637a28061a0SGeorge Keishing # unver_string JSON String we will be checking for lshw comma errors. 638a28061a0SGeorge Keishing ${unver_string}= Convert to String ${unver_string} 639a28061a0SGeorge Keishing ${ver_string}= Replace String Using Regexp ${unver_string} }\\s*{ },{ 640a28061a0SGeorge Keishing [Return] ${ver_string} 641a28061a0SGeorge Keishing 642a28061a0SGeorge Keishing 643a28061a0SGeorge KeishingClean Up String 644a28061a0SGeorge Keishing [Documentation] Remove extra whitespace and trailing commas. 645a28061a0SGeorge Keishing [Arguments] ${dirty_string} 646a28061a0SGeorge Keishing # Description of argument(s): 647a28061a0SGeorge Keishing # dirty_string String that will be space stripped and have comma removed. 648a28061a0SGeorge Keishing ${clean_string}= Strip String ${dirty_string} 649a28061a0SGeorge Keishing ${last_char}= Get Substring ${clean_string} -1 650a28061a0SGeorge Keishing ${trimmed_string}= Get Substring ${clean_string} 0 -1 651a28061a0SGeorge Keishing ${clean_string}= Set Variable If '${last_char}' == ',' 652a28061a0SGeorge Keishing ... ${trimmed_string} ${clean_string} 653a28061a0SGeorge Keishing [Return] ${clean_string} 654a28061a0SGeorge Keishing 655a28061a0SGeorge Keishing 656a28061a0SGeorge KeishingGet OS Network Interface Names 657a28061a0SGeorge Keishing [Documentation] Return a list of interface names on the OS. 658a28061a0SGeorge Keishing 659a28061a0SGeorge Keishing ${stdout} ${stderr} ${rc}= OS Execute Command ls /sys/class/net 660a28061a0SGeorge Keishing @{interface_names}= Split String ${stdout} 661a28061a0SGeorge Keishing [Return] @{interface_names} 662a28061a0SGeorge Keishing 663a28061a0SGeorge Keishing 664a28061a0SGeorge KeishingRun Soft Bootme 665a28061a0SGeorge Keishing [Documentation] Run a soft bootme for a period of an hour. 666a28061a0SGeorge Keishing [Arguments] ${bootme_period}=3 667a28061a0SGeorge Keishing # Description of argument(s): 668a28061a0SGeorge Keishing # bootme_time Bootme period to be rebooting the system. 669a28061a0SGeorge Keishing 670a28061a0SGeorge Keishing ${output} ${stderr} ${rc}= OS Execute Command 671a28061a0SGeorge Keishing ... htxcmdline -bootme on mode:soft period:${bootme_period} 672a28061a0SGeorge Keishing Should Contain ${output} bootme on is completed successfully 673a28061a0SGeorge Keishing 674a28061a0SGeorge Keishing 675a28061a0SGeorge KeishingShutdown Bootme 676a28061a0SGeorge Keishing [Documentation] Stop the bootme process. 677a28061a0SGeorge Keishing 678a28061a0SGeorge Keishing ${output} ${stderr} ${rc}= OS Execute Command htxcmdline -bootme off 679a28061a0SGeorge Keishing Should Contain ${output} bootme off is completed successfully 680