1*** Settings *** 2 3Documentation Stress the system using HTX exerciser. 4 5# Test Parameters: 6# OPENBMC_HOST The BMC host name or IP address. 7# OS_HOST The OS host name or IP Address. 8# OS_USERNAME The OS login userid (usually root). 9# OS_PASSWORD The password for the OS login. 10# HTX_DURATION Duration of HTX run, for example, 2h, or 30m. 11# HTX_LOOP The number of times to loop HTX. 12# HTX_INTERVAL The time delay between consecutive checks of HTX 13# status, for example, 15m. 14# In summary: Run HTX for $HTX_DURATION, looping 15# $HTX_LOOP times checking for errors every 16# $HTX_INTERVAL. Then allow extra time for OS 17# Boot, HTX startup, shutdown. 18# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to 19# continue running after an error was found. 20# CHECK_INVENTORY If set to 0 or False, OS inventory checking before 21# and after each HTX run will be disabled. This 22# parameter is optional. The default value is True. 23# PREV_INV_FILE_PATH The file path and name of an initial previous 24# inventory snapshot file in JSON format. Inventory 25# snapshots taken before and after each HTX run will 26# be compared to this file. 27# This parameter is optional. If not specified an 28# initial inventory snapshot will be taken before 29# HTX startup. 30# INV_IGNORE_LIST A comma-delimited list of strings that 31# indicate what to ignore if there are inventory 32# differences, (e.g., processor "size"). 33# If differences are found during inventory checking 34# and those items are in this list, the 35# differences will be ignored. This parameter is 36# optional. If not specified the default value is 37# "size". 38 39Resource ../syslib/utils_os.robot 40Resource ../lib/openbmc_ffdc_utils.robot 41Resource ../lib/logging_utils.robot 42Library ../syslib/utils_keywords.py 43Library ../lib/utils_files.py 44Library ../lib/logging_utils.py 45 46Suite Setup Run Keyword Start SOL Console Logging 47Test Setup Test Setup Execution 48Test Teardown Test Teardown Execution 49 50 51*** Variables **** 52 53${stack_mode} skip 54${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json 55${json_final_file_path} ${EXECDIR}/os_inventory_final.json 56${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json 57${CHECK_INVENTORY} True 58${INV_IGNORE_LIST} size 59${PREV_INV_FILE_PATH} NONE 60 61 62*** Test Cases *** 63 64Hard Bootme Test 65 [Documentation] Stress the system using HTX exerciser. 66 [Tags] Hard_Bootme_Test 67 68 Rprintn 69 Rpvars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY 70 ... INV_IGNORE_LIST PREV_INV_FILE_PATH 71 72 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 73 ... OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH} 74 75 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 76 Set Suite Variable ${INV_IGNORE_LIST} children=true 77 78 # Set up the iteration (loop) counter. 79 Set Suite Variable ${iteration} ${0} children=true 80 81 # Estimate the time required for a single iteration loop. 82 # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown. 83 ${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m 84 Set Suite Variable ${loop_body_seconds} children=true 85 # And save it in printable (compact) format. 86 ${estimated_loop_time}= Convert Time 87 ... ${loop_body_seconds} result_format=compact 88 Set Suite Variable ${estimated_loop_time} children=true 89 90 # Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m 91 ${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60) 92 Set Suite Variable ${est_seconds_left} children=true 93 94 Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser 95 96 97*** Keywords *** 98 99 100Run HTX Exerciser 101 [Documentation] Run HTX exerciser. 102 # Test Flow: 103 # - Power on. 104 # - Establish SSH connection session. 105 # - Do inventory collection, compare with 106 # previous inventory run if applicable. 107 # - Create HTX mdt profile. 108 # - Run HTX exerciser. 109 # - Check HTX status for errors. 110 # - Do inventory collection, compare with 111 # previous inventory run. 112 # - Power off. 113 114 Set Suite Variable ${iteration} ${iteration + 1} 115 ${loop_count}= Catenate Starting iteration: ${iteration} 116 ${estimated_time_remaining}= Convert Time 117 ... ${est_seconds_left} result_format=compact 118 Rprintn 119 Rpvars loop_count estimated_loop_time estimated_time_remaining 120 121 REST Power On stack_mode=skip 122 123 # Post Power off and on, the OS SSH session needs to be established. 124 Login To OS 125 126 Run Keyword If '${CHECK_INVENTORY}' == 'True' 127 ... Do Inventory And Compare ${json_initial_file_path} 128 ... ${PREV_INV_FILE_PATH} 129 130 Run Keyword If '${HTX_MDT_PROFILE}' == 'mdt.bu' 131 ... Create Default MDT Profile 132 133 Run MDT Profile 134 135 Loop HTX Health Check 136 137 Shutdown HTX Exerciser 138 139 Run Keyword If '${CHECK_INVENTORY}' == 'True' 140 ... Do Inventory And Compare ${json_final_file_path} 141 ... ${PREV_INV_FILE_PATH} 142 143 # Terminate run if there are any BMC error logs. 144 ${error_logs}= Get Error Logs 145 ${num_logs}= Get Length ${error_logs} 146 Run Keyword If ${num_logs} != 0 Run Keywords 147 ... Print Error Logs ${error_logs} 148 ... AND Fail msg=Terminating run due to BMC error log(s). 149 150 Power Off Host 151 152 # Close all SSH and REST active sessions. 153 Close All Connections 154 Flush REST Sessions 155 156 Rprint Timen HTX Test ran for: ${HTX_DURATION} 157 158 ${loop_count}= Catenate Ending iteration: ${iteration} 159 160 ${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds} 161 Set Suite Variable ${est_seconds_left} children=true 162 ${estimated_time_remaining}= Convert Time 163 ... ${est_seconds_left} result_format=compact 164 165 Rpvars loop_count estimated_time_remaining 166 167 168Do Inventory And Compare 169 [Documentation] Do inventory and compare. 170 [Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH} 171 # Description of argument(s): 172 # inventory_file_path The file to receive the inventory snapshot. 173 # PREV_INV_FILE_PATH The previous inventory to compare with. 174 175 Create JSON Inventory File ${inventory_file_path} 176 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 177 ... Compare Json Inventory Files ${inventory_file_path} 178 ... ${PREV_INV_FILE_PATH} 179 ${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path} 180 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 181 182 183Compare Json Inventory Files 184 [Documentation] Compare JSON inventory files. 185 [Arguments] ${file1} ${file2} 186 # Description of argument(s): 187 # file1 A file that has an inventory snapshot in JSON format. 188 # file2 A file that has an inventory snapshot, to compare with file1. 189 190 ${diff_rc}= File_Diff ${file1} 191 ... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST} 192 Run Keyword If '${diff_rc}' != '${0}' 193 ... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path} 194 ... ELSE Rprint Timen Inventoy check: No differences found. 195 196 197Report Inventory Mismatch 198 [Documentation] Report inventory mismatch. 199 [Arguments] ${diff_rc} ${json_diff_file_path} 200 # Description of argument(s): 201 # diff_rc The failing return code from the difference check. 202 # json_diff_file_path The file that has the latest inventory snapshot. 203 204 Log To Console Significant difference in inventory found, rc=${diff_rc} 205 Log To Console Differences are listed in file: no_newline=true 206 Log To Console ${json_diff_file_path} 207 Log To Console File Contents: 208 Wait Until Created ${json_diff_file_path} 209 ${file_contents}= OperatingSystem.Get File ${json_diff_file_path} 210 Log ${file_contents} level=WARN 211 Fail Significant difference in inventory found, rc=${diff_rc} 212 213 214Loop HTX Health Check 215 [Documentation] Run until HTX exerciser fails. 216 Repeat Keyword ${HTX_DURATION} 217 ... Run Keywords Check HTX Run Status 218 ... AND Sleep ${HTX_INTERVAL} 219 220 221Test Setup Execution 222 [Documentation] Do the initial test setup. 223 224 REST Power On stack_mode=skip 225 Delete All Error Logs 226 Tool Exist htxcmdline 227 228 # Shutdown if HTX is running. 229 ${status}= Is HTX Running 230 Run Keyword If '${status}' == 'True' 231 ... Shutdown HTX Exerciser 232 233 234Test Teardown Execution 235 [Documentation] Do the post test teardown. 236 # 1. Shut down HTX exerciser if test Failed. 237 # 2. Capture FFDC on test failure. 238 # 3. Close all open SSH connections. 239 240 # Keep HTX running if user set HTX_KEEP_RUNNING to 1. 241 Run Keyword If 242 ... '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0} 243 ... Shutdown HTX Exerciser 244 245 ${keyword_buf}= Catenate Stop SOL Console Logging 246 ... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log 247 Run Key ${keyword_buf} 248 249 FFDC On Test Case Fail 250 Close All Connections 251