1*** Settings *** 2 3Documentation Stress the system using HTX exerciser. 4 5# Test Parameters: 6# OPENBMC_HOST The BMC host name or IP address. 7# OS_HOST The OS host name or IP Address. 8# OS_USERNAME The OS login userid (usually root). 9# OS_PASSWORD The password for the OS login. 10# HTX_DURATION Duration of HTX run, for example, 2h, or 30m. 11# HTX_LOOP The number of times to loop HTX. 12# HTX_INTERVAL The time delay between consecutive checks of HTX 13# status, for example, 15m. 14# In summary: Run HTX for $HTX_DURATION, looping 15# $HTX_LOOP times checking for errors every 16# $HTX_INTERVAL. Then allow extra time for OS 17# Boot, HTX startup, shutdown. 18# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to 19# continue running after an error was found. 20# CHECK_INVENTORY If set to 0 or False, OS inventory checking before 21# and after each HTX run will be disabled. This 22# parameter is optional. The default value is True. 23# PREV_INV_FILE_PATH The file path and name of an initial previous 24# inventory snapshot file in JSON format. Inventory 25# snapshots taken before and after each HTX run will 26# be compared to this file. 27# This parameter is optional. If not specified an 28# initial inventory snapshot will be taken before 29# HTX startup. 30# INV_IGNORE_LIST A comma-delimited list of strings that 31# indicate what to ignore if there are inventory 32# differences, (e.g., processor "size"). 33# If differences are found during inventory checking 34# and those items are in this list, the 35# differences will be ignored. This parameter is 36# optional. If not specified the default value is 37# "size". 38 39Resource ../lib/os_utilities.robot 40Resource ../lib/openbmc_ffdc_utils.robot 41Resource ../lib/logging_utils.robot 42Resource ../lib/code_update_utils.robot 43Resource ../lib/esel_utils.robot 44Resource ../lib/htx_resource.robot 45Library ../lib/os_utils_keywords.py 46Library ../lib/utils_files.py 47Library ../lib/logging_utils.py 48Library ../lib/os_utilities.py 49 50Suite Setup Run Keyword And Ignore Error Start SOL Console Logging 51Test Setup Test Setup Execution 52Test Teardown Test Teardown Execution 53 54Test Tags HTX_Hardbootme 55 56*** Variables **** 57 58${stack_mode} skip 59${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json 60${json_final_file_path} ${EXECDIR}/os_inventory_final.json 61${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json 62${CHECK_INVENTORY} True 63${INV_IGNORE_LIST} size 64${PREV_INV_FILE_PATH} NONE 65 66${rest_keyword} REST 67 68# Error log Severities to ignore when checking Error Logs. 69@{ESEL_IGNORE_LIST} 70... xyz.openbmc_project.Logging.Entry.Level.Informational 71 72 73 74*** Test Cases *** 75 76Hard Bootme Test 77 [Documentation] Stress the system using HTX exerciser. 78 [Tags] Hard_Bootme_Test 79 80 Printn 81 Rprint Vars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY 82 ... INV_IGNORE_LIST PREV_INV_FILE_PATH 83 84 IF '${PREV_INV_FILE_PATH}' != 'NONE' 85 OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH} 86 END 87 88 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 89 Set Suite Variable ${INV_IGNORE_LIST} children=true 90 91 # Set up the iteration (loop) counter. 92 Set Suite Variable ${iteration} ${0} children=true 93 94 # Estimate the time required for a single iteration loop. 95 # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown. 96 ${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m 97 Set Suite Variable ${loop_body_seconds} children=true 98 # And save it in printable (compact) format. 99 ${estimated_loop_time}= Convert Time 100 ... ${loop_body_seconds} result_format=compact 101 Set Suite Variable ${estimated_loop_time} children=true 102 103 # Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m 104 ${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60) 105 Set Suite Variable ${est_seconds_left} children=true 106 107 Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser 108 109 110*** Keywords *** 111 112 113Run HTX Exerciser 114 [Documentation] Run HTX exerciser. 115 # Test Flow: 116 # - Power on. 117 # - Establish SSH connection session. 118 # - Do inventory collection, compare with 119 # previous inventory run if applicable. 120 # - Create HTX mdt profile. 121 # - Run HTX exerciser. 122 # - Check HTX status for errors. 123 # - Do inventory collection, compare with 124 # previous inventory run. 125 # - Power off. 126 127 Set Suite Variable ${iteration} ${iteration + 1} 128 ${loop_count}= Catenate Starting iteration: ${iteration} 129 ${estimated_time_remaining}= Convert Time 130 ... ${est_seconds_left} result_format=compact 131 Printn 132 Rprint Vars loop_count estimated_loop_time estimated_time_remaining 133 134 Run Keyword ${rest_keyword} Power On stack_mode=skip 135 Run Key U Sleep \ 15s 136 137 # Post Power off and on, the OS SSH session needs to be established. 138 Login To OS 139 140 IF '${CHECK_INVENTORY}' == 'True' 141 Do Inventory And Compare ${json_initial_file_path} ${PREV_INV_FILE_PATH} 142 END 143 144 IF '${HTX_MDT_PROFILE}' == 'mdt.bu' Create Default MDT Profile 145 146 Run MDT Profile 147 148 Loop HTX Health Check 149 150 Shutdown HTX Exerciser 151 152 IF '${CHECK_INVENTORY}' == 'True' 153 Do Inventory And Compare ${json_final_file_path} ${PREV_INV_FILE_PATH} 154 END 155 156 Run Keyword ${rest_keyword} Power Off 157 158 # Close all SSH and REST active sessions. 159 Close All Connections 160 Flush REST Sessions 161 162 Print Timen HTX Test ran for: ${HTX_DURATION} 163 164 ${loop_count}= Catenate Ending iteration: ${iteration} 165 166 ${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds} 167 Set Suite Variable ${est_seconds_left} children=true 168 ${estimated_time_remaining}= Convert Time 169 ... ${est_seconds_left} result_format=compact 170 171 Rprint Vars loop_count estimated_time_remaining 172 173 174Do Inventory And Compare 175 [Documentation] Do inventory and compare. 176 [Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH} 177 # Description of argument(s): 178 # inventory_file_path The file to receive the inventory snapshot. 179 # PREV_INV_FILE_PATH The previous inventory to compare with. 180 181 Create JSON Inventory File ${inventory_file_path} 182 IF '${PREV_INV_FILE_PATH}' != 'NONE' 183 Compare Json Inventory Files ${inventory_file_path} ${PREV_INV_FILE_PATH} 184 END 185 ${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path} 186 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 187 188 189Compare Json Inventory Files 190 [Documentation] Compare JSON inventory files. 191 [Arguments] ${file1} ${file2} 192 # Description of argument(s): 193 # file1 A file that has an inventory snapshot in JSON format. 194 # file2 A file that has an inventory snapshot, to compare with file1. 195 196 ${diff_rc}= File_Diff ${file1} 197 ... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST} 198 IF '${diff_rc}' != '${0}' 199 Report Inventory Mismatch ${diff_rc} ${json_diff_file_path} 200 ELSE 201 Print Timen Inventoy check: No differences found. 202 END 203 204Report Inventory Mismatch 205 [Documentation] Report inventory mismatch. 206 [Arguments] ${diff_rc} ${json_diff_file_path} 207 # Description of argument(s): 208 # diff_rc The failing return code from the difference check. 209 # json_diff_file_path The file that has the latest inventory snapshot. 210 211 Log To Console Significant difference in inventory found, rc=${diff_rc} 212 Log To Console Differences are listed in file: no_newline=true 213 Log To Console ${json_diff_file_path} 214 Log To Console File Contents: 215 Wait Until Created ${json_diff_file_path} 216 ${file_contents}= OperatingSystem.Get File ${json_diff_file_path} 217 Log ${file_contents} level=WARN 218 Fail Significant difference in inventory found, rc=${diff_rc} 219 220 221Loop HTX Health Check 222 [Documentation] Run until HTX exerciser fails. 223 Repeat Keyword ${HTX_DURATION} 224 ... Run Keywords Check HTX Run Status 225 ... AND Check For Error Logs ${ESEL_IGNORE_LIST} 226 ... AND Sleep ${HTX_INTERVAL} 227 228 229Test Setup Execution 230 [Documentation] Do the initial test setup. 231 232 ${bmc_version} ${stderr} ${rc}= BMC Execute Command 233 ... cat /etc/os-release 234 Printn 235 Rprint Vars bmc_version 236 237 ${fw_version}= Get BMC Version 238 Rprint Vars fw_version 239 240 ${is_redfish}= Run Keyword And Return Status Redfish.Login 241 ${rest_keyword}= Set Variable If ${is_redfish} Redfish REST 242 Rprint Vars rest_keyword 243 Set Suite Variable ${rest_keyword} children=true 244 245 Run Keyword ${rest_keyword} Power On stack_mode=skip 246 247 Run Key U Sleep \ 15s 248 Run Keyword And Ignore Error Delete All Error Logs 249 Run Keyword And Ignore Error Redfish Purge Event Log 250 Tool Exist htxcmdline 251 252 ${os_release_info}= os_utilities.Get OS Release Info uname 253 Rprint Vars os_release_info fmt=1 254 255 # Shutdown if HTX is running. 256 ${status}= Is HTX Running 257 IF '${status}' == 'True' Shutdown HTX Exerciser 258 259 260Test Teardown Execution 261 [Documentation] Do the post-test teardown. 262 263 # Keep HTX running if user set HTX_KEEP_RUNNING to 1. 264 IF '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0} 265 Shutdown HTX Exerciser 266 END 267 268 ${keyword_buf}= Catenate Stop SOL Console Logging 269 ... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log 270 Run Keyword And Ignore Error ${keyword_buf} 271 272 FFDC On Test Case Fail 273 Close All Connections 274