1*** Settings ***
2
3Documentation  Stress the system using HTX exerciser.
4
5# Test Parameters:
6# OPENBMC_HOST        The BMC host name or IP address.
7# OS_HOST             The OS host name or IP Address.
8# OS_USERNAME         The OS login userid (usually root).
9# OS_PASSWORD         The password for the OS login.
10# HTX_DURATION        Duration of HTX run, for example, 2h, or 30m.
11# HTX_LOOP            The number of times to loop HTX.
12# HTX_INTERVAL        The time delay between consecutive checks of HTX
13#                     status, for example, 15m.
14#                     In summary: Run HTX for $HTX_DURATION, looping
15#                     $HTX_LOOP times checking for errors every
16#                     $HTX_INTERVAL.  Then allow extra time for OS
17#                     Boot, HTX startup, shutdown.
18# HTX_KEEP_RUNNING    If set to 1, this indicates that the HTX is to
19#                     continue running after an error was found.
20# CHECK_INVENTORY     If set to 0 or False, OS inventory checking before
21#                     and after each HTX run will be disabled.  This
22#                     parameter is optional.  The default value is True.
23# PREV_INV_FILE_PATH  The file path and name of an initial previous
24#                     inventory snapshot file in JSON format.  Inventory
25#                     snapshots taken before and after each HTX run will
26#                     be compared to this file.
27#                     This parameter is optional.  If not specified an
28#                     initial inventory snapshot will be taken before
29#                     HTX startup.
30# INV_IGNORE_LIST     A comma-delimited list of strings that
31#                     indicate what to ignore if there are inventory
32#                     differences, (e.g., processor "size").
33#                     If differences are found during inventory checking
34#                     and those items are in this list, the
35#                     differences will be ignored.  This parameter is
36#                     optional.  If not specified the default value is
37#                     "size".
38
39Resource        ../syslib/utils_os.robot
40Resource        ../lib/openbmc_ffdc_utils.robot
41Resource        ../lib/logging_utils.robot
42Resource        ../lib/code_update_utils.robot
43Resource        ../lib/esel_utils.robot
44Resource        ../syslib/resource.robot
45Library         ../syslib/utils_keywords.py
46Library         ../lib/utils_files.py
47Library         ../lib/logging_utils.py
48Library         ../syslib/utils_os.py
49
50Suite Setup     Run Keyword And Ignore Error  Start SOL Console Logging
51Test Setup      Test Setup Execution
52Test Teardown   Test Teardown Execution
53
54Force Tags      HTX_Hardbootme
55
56*** Variables ****
57
58${stack_mode}                skip
59${json_initial_file_path}    ${EXECDIR}/os_inventory_initial.json
60${json_final_file_path}      ${EXECDIR}/os_inventory_final.json
61${json_diff_file_path}       ${EXECDIR}/os_inventory_diff.json
62${CHECK_INVENTORY}           True
63${INV_IGNORE_LIST}           size
64${PREV_INV_FILE_PATH}        NONE
65
66${rest_keyword}              REST
67
68# Error log Severities to ignore when checking Error Logs.
69@{ESEL_IGNORE_LIST}
70...  xyz.openbmc_project.Logging.Entry.Level.Informational
71
72
73
74*** Test Cases ***
75
76Hard Bootme Test
77    [Documentation]  Stress the system using HTX exerciser.
78    [Tags]  Hard_Bootme_Test
79
80    Printn
81    Rprint Vars  HTX_DURATION  HTX_LOOP  HTX_INTERVAL  CHECK_INVENTORY
82    ...  INV_IGNORE_LIST  PREV_INV_FILE_PATH
83
84    Run Keyword If  '${PREV_INV_FILE_PATH}' != 'NONE'
85    ...  OperatingSystem.File Should Exist  ${PREV_INV_FILE_PATH}
86
87    Set Suite Variable  ${PREV_INV_FILE_PATH}  children=true
88    Set Suite Variable  ${INV_IGNORE_LIST}  children=true
89
90    # Set up the iteration (loop) counter.
91    Set Suite Variable  ${iteration}  ${0}  children=true
92
93    # Estimate the time required for a single iteration loop.
94    # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown.
95    ${loop_body_seconds}=  Add Time To Time  ${HTX_DURATION}  10m
96    Set Suite Variable  ${loop_body_seconds}  children=true
97    # And save it in printable (compact) format.
98    ${estimated_loop_time}=  Convert Time
99    ...  ${loop_body_seconds}  result_format=compact
100    Set Suite Variable  ${estimated_loop_time}  children=true
101
102    # Estimated time remaining =  loop_body_seconds * HTX_LOOP + 5m
103    ${est_seconds_left}=  Evaluate  ${loop_body_seconds}*${HTX_LOOP}+(5*60)
104    Set Suite Variable  ${est_seconds_left}  children=true
105
106    Repeat Keyword  ${HTX_LOOP} times  Run HTX Exerciser
107
108
109*** Keywords ***
110
111
112Run HTX Exerciser
113    [Documentation]  Run HTX exerciser.
114    # Test Flow:
115    # - Power on.
116    # - Establish SSH connection session.
117    # - Do inventory collection, compare with
118    #   previous inventory run if applicable.
119    # - Create HTX mdt profile.
120    # - Run HTX exerciser.
121    # - Check HTX status for errors.
122    # - Do inventory collection, compare with
123    #   previous inventory run.
124    # - Power off.
125
126    Set Suite Variable  ${iteration}  ${iteration + 1}
127    ${loop_count}=  Catenate  Starting iteration: ${iteration}
128    ${estimated_time_remaining}=  Convert Time
129    ...  ${est_seconds_left}  result_format=compact
130    Printn
131    Rprint Vars  loop_count  estimated_loop_time   estimated_time_remaining
132
133    Run Keyword  ${rest_keyword} Power On  stack_mode=skip
134    Run Key U  Sleep \ 15s
135
136    # Post Power off and on, the OS SSH session needs to be established.
137    Login To OS
138
139    Run Keyword If  '${CHECK_INVENTORY}' == 'True'
140    ...  Do Inventory And Compare  ${json_initial_file_path}
141    ...  ${PREV_INV_FILE_PATH}
142
143    Run Keyword If  '${HTX_MDT_PROFILE}' == 'mdt.bu'
144    ...  Create Default MDT Profile
145
146    Run MDT Profile
147
148    Loop HTX Health Check
149
150    Shutdown HTX Exerciser
151
152    Run Keyword If  '${CHECK_INVENTORY}' == 'True'
153    ...  Do Inventory And Compare  ${json_final_file_path}
154    ...  ${PREV_INV_FILE_PATH}
155
156    Run Keyword  ${rest_keyword} Power Off
157
158    # Close all SSH and REST active sessions.
159    Close All Connections
160    Flush REST Sessions
161
162    Print Timen  HTX Test ran for: ${HTX_DURATION}
163
164    ${loop_count}=  Catenate  Ending iteration: ${iteration}
165
166    ${est_seconds_left}=  Evaluate  ${est_seconds_left}-${loop_body_seconds}
167    Set Suite Variable  ${est_seconds_left}  children=true
168    ${estimated_time_remaining}=  Convert Time
169    ...  ${est_seconds_left}  result_format=compact
170
171    Rprint Vars  loop_count  estimated_time_remaining
172
173
174Do Inventory And Compare
175    [Documentation]  Do inventory and compare.
176    [Arguments]  ${inventory_file_path}  ${PREV_INV_FILE_PATH}
177    # Description of argument(s):
178    # inventory_file_path  The file to receive the inventory snapshot.
179    # PREV_INV_FILE_PATH   The previous inventory to compare with.
180
181    Create JSON Inventory File  ${inventory_file_path}
182    Run Keyword If  '${PREV_INV_FILE_PATH}' != 'NONE'
183    ...  Compare Json Inventory Files  ${inventory_file_path}
184    ...  ${PREV_INV_FILE_PATH}
185    ${PREV_INV_FILE_PATH}=   Set Variable  ${inventory_file_path}
186    Set Suite Variable  ${PREV_INV_FILE_PATH}  children=true
187
188
189Compare Json Inventory Files
190    [Documentation]  Compare JSON inventory files.
191    [Arguments]  ${file1}  ${file2}
192    # Description of argument(s):
193    # file1   A file that has an inventory snapshot in JSON format.
194    # file2   A file that has an inventory snapshot, to compare with file1.
195
196    ${diff_rc}=  File_Diff  ${file1}
197     ...  ${file2}  ${json_diff_file_path}  ${INV_IGNORE_LIST}
198    Run Keyword If  '${diff_rc}' != '${0}'
199    ...  Report Inventory Mismatch  ${diff_rc}  ${json_diff_file_path}
200    ...  ELSE  Print Timen  Inventoy check: No differences found.
201
202
203Report Inventory Mismatch
204    [Documentation]  Report inventory mismatch.
205    [Arguments]  ${diff_rc}  ${json_diff_file_path}
206    # Description of argument(s):
207    # diff_rc              The failing return code from the difference check.
208    # json_diff_file_path  The file that has the latest inventory snapshot.
209
210    Log To Console  Significant difference in inventory found, rc=${diff_rc}
211    Log To Console  Differences are listed in file:  no_newline=true
212    Log To Console  ${json_diff_file_path}
213    Log To Console  File Contents:
214    Wait Until Created  ${json_diff_file_path}
215    ${file_contents}=  OperatingSystem.Get File  ${json_diff_file_path}
216    Log  ${file_contents}  level=WARN
217    Fail  Significant difference in inventory found, rc=${diff_rc}
218
219
220Loop HTX Health Check
221    [Documentation]  Run until HTX exerciser fails.
222    Repeat Keyword  ${HTX_DURATION}
223    ...  Run Keywords  Check HTX Run Status
224    ...  AND  Check For Error Logs  ${ESEL_IGNORE_LIST}
225    ...  AND  Sleep  ${HTX_INTERVAL}
226
227
228Test Setup Execution
229    [Documentation]  Do the initial test setup.
230
231    ${bmc_version}  ${stderr}  ${rc}=  BMC Execute Command
232    ...  cat /etc/os-release
233    Printn
234    Rprint Vars  bmc_version
235
236    ${fw_version}=  Get BMC Version
237    Rprint Vars  fw_version
238
239    ${is_redfish}=  Run Keyword And Return Status  Redfish.Login
240    ${rest_keyword}=  Set Variable If  ${is_redfish}  Redfish  REST
241    Rprint Vars  rest_keyword
242    Set Suite Variable  ${rest_keyword}  children=true
243
244    Run Keyword  ${rest_keyword} Power On  stack_mode=skip
245
246    Run Key U  Sleep \ 15s
247    Run Keyword And Ignore Error  Delete All Error Logs
248    Run Keyword And Ignore Error  Redfish Purge Event Log
249    Tool Exist  htxcmdline
250
251    ${os_release_info}=  utils_os.Get OS Release Info  uname
252    Rprint Vars  os_release_info  fmt=1
253
254    # Shutdown if HTX is running.
255    ${status}=  Is HTX Running
256    Run Keyword If  '${status}' == 'True'
257    ...  Shutdown HTX Exerciser
258
259
260Test Teardown Execution
261    [Documentation]  Do the post-test teardown.
262
263    # Keep HTX running if user set HTX_KEEP_RUNNING to 1.
264    Run Keyword If
265    ...  '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0}
266    ...  Shutdown HTX Exerciser
267
268    ${keyword_buf}=  Catenate  Stop SOL Console Logging
269    ...  \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log
270    Run Keyword And Ignore Error   ${keyword_buf}
271
272    FFDC On Test Case Fail
273    Close All Connections
274