1*** Settings *** 2 3Documentation Stress the system using HTX exerciser. 4 5# Test Parameters: 6# OPENBMC_HOST The BMC host name or IP address. 7# OS_HOST The OS host name or IP Address. 8# OS_USERNAME The OS login userid (usually root). 9# OS_PASSWORD The password for the OS login. 10# HTX_DURATION Duration of HTX run, for example, 2h, or 30m. 11# HTX_LOOP The number of times to loop HTX. 12# HTX_INTERVAL The time delay between consecutive checks of HTX 13# status, for example, 15m. 14# In summary: Run HTX for $HTX_DURATION, looping 15# $HTX_LOOP times checking for errors every 16# $HTX_INTERVAL. Then allow extra time for OS 17# Boot, HTX startup, shutdown. 18# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to 19# continue running after an error was found. 20# CHECK_INVENTORY If set to 0 or False, OS inventory checking before 21# and after each HTX run will be disabled. This 22# parameter is optional. The default value is True. 23# PREV_INV_FILE_PATH The file path and name of an initial previous 24# inventory snapshot file in JSON format. Inventory 25# snapshots taken before and after each HTX run will 26# be compared to this file. 27# This parameter is optional. If not specified an 28# initial inventory snapshot will be taken before 29# HTX startup. 30# INV_IGNORE_LIST A comma-delimited list of strings that 31# indicate what to ignore if there are inventory 32# differences, (e.g., processor "size"). 33# If differences are found during inventory checking 34# and those items are in this list, the 35# differences will be ignored. This parameter is 36# optional. If not specified the default value is 37# "size". 38 39Resource ../syslib/utils_os.robot 40Resource ../lib/openbmc_ffdc_utils.robot 41Resource ../lib/logging_utils.robot 42Library ../syslib/utils_keywords.py 43Library ../lib/utils_files.py 44Library ../lib/logging_utils.py 45Library ../syslib/utils_os.py 46 47Suite Setup Run Keyword Start SOL Console Logging 48Test Setup Test Setup Execution 49Test Teardown Test Teardown Execution 50 51 52*** Variables **** 53 54${stack_mode} skip 55${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json 56${json_final_file_path} ${EXECDIR}/os_inventory_final.json 57${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json 58${CHECK_INVENTORY} True 59${INV_IGNORE_LIST} size 60${PREV_INV_FILE_PATH} NONE 61 62 63*** Test Cases *** 64 65Hard Bootme Test 66 [Documentation] Stress the system using HTX exerciser. 67 [Tags] Hard_Bootme_Test 68 69 Rprintn 70 Rpvars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY 71 ... INV_IGNORE_LIST PREV_INV_FILE_PATH 72 73 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 74 ... OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH} 75 76 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 77 Set Suite Variable ${INV_IGNORE_LIST} children=true 78 79 # Set up the iteration (loop) counter. 80 Set Suite Variable ${iteration} ${0} children=true 81 82 # Estimate the time required for a single iteration loop. 83 # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown. 84 ${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m 85 Set Suite Variable ${loop_body_seconds} children=true 86 # And save it in printable (compact) format. 87 ${estimated_loop_time}= Convert Time 88 ... ${loop_body_seconds} result_format=compact 89 Set Suite Variable ${estimated_loop_time} children=true 90 91 # Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m 92 ${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60) 93 Set Suite Variable ${est_seconds_left} children=true 94 95 Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser 96 97 98*** Keywords *** 99 100 101Run HTX Exerciser 102 [Documentation] Run HTX exerciser. 103 # Test Flow: 104 # - Power on. 105 # - Establish SSH connection session. 106 # - Do inventory collection, compare with 107 # previous inventory run if applicable. 108 # - Create HTX mdt profile. 109 # - Run HTX exerciser. 110 # - Check HTX status for errors. 111 # - Do inventory collection, compare with 112 # previous inventory run. 113 # - Power off. 114 115 Set Suite Variable ${iteration} ${iteration + 1} 116 ${loop_count}= Catenate Starting iteration: ${iteration} 117 ${estimated_time_remaining}= Convert Time 118 ... ${est_seconds_left} result_format=compact 119 Rprintn 120 Rpvars loop_count estimated_loop_time estimated_time_remaining 121 122 REST Power On stack_mode=skip 123 Run Key U Sleep \ 15s 124 125 # Post Power off and on, the OS SSH session needs to be established. 126 Login To OS 127 128 Run Keyword If '${CHECK_INVENTORY}' == 'True' 129 ... Do Inventory And Compare ${json_initial_file_path} 130 ... ${PREV_INV_FILE_PATH} 131 132 Run Keyword If '${HTX_MDT_PROFILE}' == 'mdt.bu' 133 ... Create Default MDT Profile 134 135 Run MDT Profile 136 137 Loop HTX Health Check 138 139 Shutdown HTX Exerciser 140 141 Run Keyword If '${CHECK_INVENTORY}' == 'True' 142 ... Do Inventory And Compare ${json_final_file_path} 143 ... ${PREV_INV_FILE_PATH} 144 145 # Terminate run if there are any BMC error logs. 146 ${error_logs}= Get Error Logs 147 ${num_logs}= Get Length ${error_logs} 148 Run Keyword If ${num_logs} != 0 Run Keywords 149 ... Print Error Logs ${error_logs} 150 ... AND Fail msg=Terminating run due to BMC error log(s). 151 152 Power Off Host 153 154 # Close all SSH and REST active sessions. 155 Close All Connections 156 Flush REST Sessions 157 158 Rprint Timen HTX Test ran for: ${HTX_DURATION} 159 160 ${loop_count}= Catenate Ending iteration: ${iteration} 161 162 ${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds} 163 Set Suite Variable ${est_seconds_left} children=true 164 ${estimated_time_remaining}= Convert Time 165 ... ${est_seconds_left} result_format=compact 166 167 Rpvars loop_count estimated_time_remaining 168 169 170Do Inventory And Compare 171 [Documentation] Do inventory and compare. 172 [Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH} 173 # Description of argument(s): 174 # inventory_file_path The file to receive the inventory snapshot. 175 # PREV_INV_FILE_PATH The previous inventory to compare with. 176 177 Create JSON Inventory File ${inventory_file_path} 178 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 179 ... Compare Json Inventory Files ${inventory_file_path} 180 ... ${PREV_INV_FILE_PATH} 181 ${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path} 182 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 183 184 185Compare Json Inventory Files 186 [Documentation] Compare JSON inventory files. 187 [Arguments] ${file1} ${file2} 188 # Description of argument(s): 189 # file1 A file that has an inventory snapshot in JSON format. 190 # file2 A file that has an inventory snapshot, to compare with file1. 191 192 ${diff_rc}= File_Diff ${file1} 193 ... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST} 194 Run Keyword If '${diff_rc}' != '${0}' 195 ... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path} 196 ... ELSE Rprint Timen Inventoy check: No differences found. 197 198 199Report Inventory Mismatch 200 [Documentation] Report inventory mismatch. 201 [Arguments] ${diff_rc} ${json_diff_file_path} 202 # Description of argument(s): 203 # diff_rc The failing return code from the difference check. 204 # json_diff_file_path The file that has the latest inventory snapshot. 205 206 Log To Console Significant difference in inventory found, rc=${diff_rc} 207 Log To Console Differences are listed in file: no_newline=true 208 Log To Console ${json_diff_file_path} 209 Log To Console File Contents: 210 Wait Until Created ${json_diff_file_path} 211 ${file_contents}= OperatingSystem.Get File ${json_diff_file_path} 212 Log ${file_contents} level=WARN 213 Fail Significant difference in inventory found, rc=${diff_rc} 214 215 216Loop HTX Health Check 217 [Documentation] Run until HTX exerciser fails. 218 Repeat Keyword ${HTX_DURATION} 219 ... Run Keywords Check HTX Run Status 220 ... AND Sleep ${HTX_INTERVAL} 221 222 223Test Setup Execution 224 [Documentation] Do the initial test setup. 225 226 ${bmc_version} ${stderr} ${rc}= BMC Execute Command 227 ... cat /etc/os-release 228 Rprintn 229 Rpvars bmc_version 230 231 ${pnor_version} ${stderr} ${rc}= BMC Execute Command 232 ... sed -re 's/\t//g' /var/lib/phosphor-software-manager/pnor/ro/VERSION 233 Rpvars pnor_version 234 235 REST Power On stack_mode=skip 236 Run Key U Sleep \ 15s 237 Delete All Error Logs 238 Tool Exist htxcmdline 239 240 ${os_release_info}= Get OS Release Info 241 Rpvars 1 os_release_info 242 243 # Shutdown if HTX is running. 244 ${status}= Is HTX Running 245 Run Keyword If '${status}' == 'True' 246 ... Shutdown HTX Exerciser 247 248 249Test Teardown Execution 250 [Documentation] Do the post-test teardown. 251 252 # Keep HTX running if user set HTX_KEEP_RUNNING to 1. 253 Run Keyword If 254 ... '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0} 255 ... Shutdown HTX Exerciser 256 257 ${keyword_buf}= Catenate Stop SOL Console Logging 258 ... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log 259 Run Key ${keyword_buf} 260 261 FFDC On Test Case Fail 262 Close All Connections 263