1*** Settings *** 2 3Documentation Stress the system using HTX exerciser. 4 5# Test Parameters: 6# OPENBMC_HOST The BMC host name or IP address. 7# OS_HOST The OS host name or IP Address. 8# OS_USERNAME The OS login userid (usually root). 9# OS_PASSWORD The password for the OS login. 10# HTX_DURATION Duration of HTX run, for example, 2h, or 30m. 11# HTX_LOOP The number of times to loop HTX. 12# HTX_INTERVAL The time delay between consecutive checks of HTX 13# status, for example, 15m. 14# In summary: Run HTX for $HTX_DURATION, looping 15# $HTX_LOOP times checking for errors every $HTX_INTERVAL. 16# Then allow extra time for OS Boot, HTX startup, shutdown. 17# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to 18# continue running after an error was found. 19# CHECK_INVENTORY If set to 0 or False, OS inventory checking before 20# and after each HTX run will be disabled. This 21# parameter is optional. The default value is True. 22# PREV_INV_FILE_PATH The file path and name of an initial previous 23# inventory snapshot file in JSON format. Inventory 24# snapshots taken before and after each HTX run will 25# be compared to this file. 26# This parameter is optional. If not specified an 27# initial inventory snapshot will be taken before 28# HTX startup. 29# INV_IGNORE_LIST A comma-delimited list of strings that 30# indicate what to ignore if there are inventory 31# differences, (e.g., processor "size"). 32# If differences are found during inventory checking 33# and those items are in this list, the 34# differences will be ignored. This parameter is 35# optional. If not specified the default value is 36# "size". 37 38Resource ../syslib/utils_os.robot 39Resource ../lib/openbmc_ffdc_utils.robot 40Library ../syslib/utils_keywords.py 41Library ../lib/utils_files.py 42 43Suite Setup Run Keyword Start SOL Console Logging 44Test Setup Test Setup Execution 45Test Teardown Test Teardown Execution 46 47 48*** Variables **** 49 50${stack_mode} skip 51${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json 52${json_final_file_path} ${EXECDIR}/os_inventory_final.json 53${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json 54${CHECK_INVENTORY} True 55${INV_IGNORE_LIST} size 56${PREV_INV_FILE_PATH} NONE 57 58 59*** Test Cases *** 60 61Hard Bootme Test 62 [Documentation] Stress the system using HTX exerciser. 63 [Tags] Hard_Bootme_Test 64 65 Rprintn 66 Rpvars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY 67 ... INV_IGNORE_LIST PREV_INV_FILE_PATH 68 69 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 70 ... OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH} 71 72 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 73 Set Suite Variable ${INV_IGNORE_LIST} children=true 74 75 # Set up the iteration (loop) counter. 76 Set Suite Variable ${iteration} ${0} children=true 77 78 # Estimate the time required for a single iteration loop. 79 # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown. 80 ${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m 81 Set Suite Variable ${loop_body_seconds} children=true 82 # And save it in printable (compact) format. 83 ${estimated_loop_time}= Convert Time 84 ... ${loop_body_seconds} result_format=compact 85 Set Suite Variable ${estimated_loop_time} children=true 86 87 # Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m 88 ${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60) 89 Set Suite Variable ${est_seconds_left} children=true 90 91 Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser 92 93 94*** Keywords *** 95 96 97Run HTX Exerciser 98 [Documentation] Run HTX exerciser. 99 # Test Flow: 100 # - Power on. 101 # - Establish SSH connection session. 102 # - Do inventory collection, compare with 103 # previous inventory run if applicable. 104 # - Create HTX mdt profile. 105 # - Run HTX exerciser. 106 # - Check HTX status for errors. 107 # - Do inventory collection, compare with 108 # previous inventory run. 109 # - Power off. 110 111 Set Suite Variable ${iteration} ${iteration + 1} 112 ${loop_count}= Catenate Starting iteration: ${iteration} 113 ${estimated_time_remaining}= Convert Time 114 ... ${est_seconds_left} result_format=compact 115 Rprintn 116 Rpvars loop_count estimated_loop_time estimated_time_remaining 117 118 Boot To OS 119 120 # Post Power off and on, the OS SSH session needs to be established. 121 Login To OS 122 123 Run Keyword If '${CHECK_INVENTORY}' == 'True' 124 ... Do Inventory And Compare ${json_initial_file_path} 125 ... ${PREV_INV_FILE_PATH} 126 127 Run Keyword If '${HTX_MDT_PROFILE}' == 'mdt.bu' 128 ... Create Default MDT Profile 129 130 Run MDT Profile 131 132 Loop HTX Health Check 133 134 Shutdown HTX Exerciser 135 136 Run Keyword If '${CHECK_INVENTORY}' == 'True' 137 ... Do Inventory And Compare ${json_final_file_path} 138 ... ${PREV_INV_FILE_PATH} 139 140 Error Logs Should Not Exist 141 Power Off Host 142 143 # Close all SSH and REST active sessions. 144 Close All Connections 145 Flush REST Sessions 146 147 Rprint Timen HTX Test ran for: ${HTX_DURATION} 148 149 ${loop_count}= Catenate Ending iteration: ${iteration} 150 151 ${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds} 152 Set Suite Variable ${est_seconds_left} children=true 153 ${estimated_time_remaining}= Convert Time 154 ... ${est_seconds_left} result_format=compact 155 156 Rpvars loop_count estimated_time_remaining 157 158 159Do Inventory And Compare 160 [Documentation] Do inventory and compare. 161 [Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH} 162 # Description of argument(s): 163 # inventory_file_path The file to receive the inventory snapshot. 164 # PREV_INV_FILE_PATH The previous inventory to compare with. 165 166 Create JSON Inventory File ${inventory_file_path} 167 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE' 168 ... Compare Json Inventory Files ${inventory_file_path} 169 ... ${PREV_INV_FILE_PATH} 170 ${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path} 171 Set Suite Variable ${PREV_INV_FILE_PATH} children=true 172 173 174Compare Json Inventory Files 175 [Documentation] Compare JSON inventory files. 176 [Arguments] ${file1} ${file2} 177 # Description of argument(s): 178 # file1 A file that has an inventory snapshot in JSON format. 179 # file2 A file that has an inventory snapshot, to compare with file1. 180 181 ${diff_rc}= File_Diff ${file1} 182 ... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST} 183 Run Keyword If '${diff_rc}' != '${0}' 184 ... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path} 185 186 187Report Inventory Mismatch 188 [Documentation] Report inventory mismatch. 189 [Arguments] ${diff_rc} ${json_diff_file_path} 190 # Description of argument(s): 191 # diff_rc The failing return code from the difference check. 192 # json_diff_file_path The file that has the latest inventory snapshot. 193 194 Log To Console Significant difference in inventory found, rc=${diff_rc} 195 Log To Console Differences are listed in file: no_newline=true 196 Log To Console ${json_diff_file_path} 197 Log To Console File Contents: 198 Wait Until Created ${json_diff_file_path} 199 ${file_contents}= OperatingSystem.Get File ${json_diff_file_path} 200 Log ${file_contents} level=WARN 201 Fail Significant difference in inventory found, rc=${diff_rc} 202 203 204Loop HTX Health Check 205 [Documentation] Run until HTX exerciser fails. 206 Repeat Keyword ${HTX_DURATION} 207 ... Run Keywords Check HTX Run Status 208 ... AND Sleep ${HTX_INTERVAL} 209 210 211Test Teardown Execution 212 [Documentation] Do the post test teardown. 213 # 1. Shut down HTX exerciser if test Failed. 214 # 2. Capture FFDC on test failure. 215 # 3. Close all open SSH connections. 216 217 # Keep HTX running if user set HTX_KEEP_RUNNING to 1. 218 Run Keyword If 219 ... '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0} 220 ... Shutdown HTX Exerciser 221 222 ${keyword_buf}= Catenate Stop SOL Console Logging 223 ... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log 224 Run Key ${keyword_buf} 225 226 FFDC On Test Case Fail 227 Close All Connections 228