1*** Settings ***
2
3Documentation  Stress the system using HTX exerciser.
4
5# Test Parameters:
6# OPENBMC_HOST        The BMC host name or IP address.
7# OS_HOST             The OS host name or IP Address.
8# OS_USERNAME         The OS login userid (usually root).
9# OS_PASSWORD         The password for the OS login.
10# HTX_DURATION        Duration of HTX run, for example, 2h, or 30m.
11# HTX_LOOP            The number of times to loop HTX.
12# HTX_INTERVAL        The time delay between consecutive checks of HTX
13#                     status, for example, 15m.
14#                     In summary: Run HTX for $HTX_DURATION, looping
15#                     $HTX_LOOP times checking for errors every $HTX_INTERVAL.
16#                     Then allow extra time for OS Boot, HTX startup, shutdown.
17# HTX_KEEP_RUNNING    If set to 1, this indicates that the HTX is to
18#                     continue running after an error was found.
19# CHECK_INVENTORY     If set to 0 or False, OS inventory checking before
20#                     and after each HTX run will be disabled.  This
21#                     parameter is optional.  The default value is True.
22# PREV_INV_FILE_PATH  The file path and name of an initial previous
23#                     inventory snapshot file in JSON format.  Inventory
24#                     snapshots taken before and after each HTX run will
25#                     be compared to this file.
26#                     This parameter is optional.  If not specified an
27#                     initial inventory snapshot will be taken before
28#                     HTX startup.
29# INV_IGNORE_LIST     A comma-delimited list of strings that
30#                     indicate what to ignore if there are inventory
31#                     differences, (e.g., processor "size").
32#                     If differences are found during inventory checking
33#                     and those items are in this list, the
34#                     differences will be ignored.  This parameter is
35#                     optional.  If not specified the default value is
36#                     "size".
37
38Resource        ../syslib/utils_os.robot
39Library         ../syslib/utils_keywords.py
40Library         ../lib/utils_files.py
41
42Suite Setup     Run Keyword  Start SOL Console Logging
43Test Setup      Pre Test Case Execution
44Test Teardown   Post Test Case Execution
45
46
47*** Variables ****
48
49${stack_mode}                skip
50${json_initial_file_path}    ${EXECDIR}/os_inventory_initial.json
51${json_final_file_path}      ${EXECDIR}/os_inventory_final.json
52${json_diff_file_path}       ${EXECDIR}/os_inventory_diff.json
53${CHECK_INVENTORY}           True
54${INV_IGNORE_LIST}           size
55${PREV_INV_FILE_PATH}        NONE
56
57
58*** Test Cases ***
59
60Hard Bootme Test
61    [Documentation]  Stress the system using HTX exerciser.
62    [Tags]  Hard_Bootme_Test
63
64    Rprintn
65    Rpvars  HTX_DURATION  HTX_LOOP  HTX_INTERVAL  CHECK_INVENTORY
66    ...  INV_IGNORE_LIST  PREV_INV_FILE_PATH
67
68    Run Keyword If  '${PREV_INV_FILE_PATH}' != 'NONE'
69    ...  OperatingSystem.File Should Exist  ${PREV_INV_FILE_PATH}
70
71    Set Suite Variable  ${PREV_INV_FILE_PATH}  children=true
72    Set Suite Variable  ${INV_IGNORE_LIST}  children=true
73
74    # Set up the iteration (loop) counter.
75    Set Suite Variable  ${iteration}  ${0}  children=true
76
77    # Estimate the time required for a single iteration loop.
78    # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown.
79    ${loop_body_seconds}=  Add Time To Time  ${HTX_DURATION}  10m
80    Set Suite Variable  ${loop_body_seconds}  children=true
81    # And save it in printable (compact) format.
82    ${estimated_loop_time}=  Convert Time
83    ...  ${loop_body_seconds}  result_format=compact
84    Set Suite Variable  ${estimated_loop_time}  children=true
85
86    # Estimated time remaining =  loop_body_seconds * HTX_LOOP + 5m
87    ${est_seconds_left}=  Evaluate  ${loop_body_seconds}*${HTX_LOOP}+(5*60)
88    Set Suite Variable  ${est_seconds_left}  children=true
89
90    Repeat Keyword  ${HTX_LOOP} times  Run HTX Exerciser
91
92
93*** Keywords ***
94
95
96Run HTX Exerciser
97    [Documentation]  Run HTX exerciser.
98    # Test Flow:
99    # - Power on.
100    # - Establish SSH connection session.
101    # - Do inventory collection, compare with
102    #   previous inventory run if applicable.
103    # - Create HTX mdt profile.
104    # - Run HTX exerciser.
105    # - Check HTX status for errors.
106    # - Do inventory collection, compare with
107    #   previous inventory run.
108    # - Power off.
109
110    Set Suite Variable  ${iteration}  ${iteration + 1}
111    ${loop_count}=  Catenate  Starting iteration: ${iteration}
112    ${estimated_time_remaining}=  Convert Time
113    ...  ${est_seconds_left}  result_format=compact
114    Rprintn
115    Rpvars  loop_count  estimated_loop_time   estimated_time_remaining
116
117    Boot To OS
118
119    # Post Power off and on, the OS SSH session needs to be established.
120    Login To OS
121
122    Run Keyword If  '${CHECK_INVENTORY}' == 'True'
123    ...  Do Inventory And Compare  ${json_initial_file_path}
124    ...  ${PREV_INV_FILE_PATH}
125
126    Run Keyword If  '${HTX_MDT_PROFILE}' == 'mdt.bu'
127    ...  Create Default MDT Profile
128
129    Run MDT Profile
130
131    Loop HTX Health Check
132
133    Shutdown HTX Exerciser
134
135    Run Keyword If  '${CHECK_INVENTORY}' == 'True'
136    ...  Do Inventory And Compare  ${json_final_file_path}
137    ...  ${PREV_INV_FILE_PATH}
138
139    Power Off Host
140
141    # Close all SSH and REST active sessions.
142    Close All Connections
143    Flush REST Sessions
144
145    Rprint Timen  HTX Test ran for: ${HTX_DURATION}
146
147    ${loop_count}=  Catenate  Ending iteration: ${iteration}
148
149    ${est_seconds_left}=  Evaluate  ${est_seconds_left}-${loop_body_seconds}
150    Set Suite Variable  ${est_seconds_left}  children=true
151    ${estimated_time_remaining}=  Convert Time
152    ...  ${est_seconds_left}  result_format=compact
153
154    Rpvars  loop_count  estimated_time_remaining
155
156
157Do Inventory And Compare
158    [Documentation]  Do inventory and compare.
159    [Arguments]  ${inventory_file_path}  ${PREV_INV_FILE_PATH}
160    # Description of argument(s):
161    # inventory_file_path  The file to receive the inventory snapshot.
162    # PREV_INV_FILE_PATH   The previous inventory to compare with.
163
164    Create JSON Inventory File  ${inventory_file_path}
165    Run Keyword If  '${PREV_INV_FILE_PATH}' != 'NONE'
166    ...  Compare Json Inventory Files  ${inventory_file_path}
167    ...  ${PREV_INV_FILE_PATH}
168    ${PREV_INV_FILE_PATH}=   Set Variable  ${inventory_file_path}
169    Set Suite Variable  ${PREV_INV_FILE_PATH}  children=true
170
171
172Compare Json Inventory Files
173    [Documentation]  Compare JSON inventory files.
174    [Arguments]  ${file1}  ${file2}
175    # Description of argument(s):
176    # file1   A file that has an inventory snapshot in JSON format.
177    # file2   A file that has an inventory snapshot, to compare with file1.
178
179    ${diff_rc}=  File_Diff  ${file1}
180     ...  ${file2}  ${json_diff_file_path}  ${INV_IGNORE_LIST}
181    Run Keyword If  '${diff_rc}' != '${0}'
182    ...  Report Inventory Mismatch  ${diff_rc}  ${json_diff_file_path}
183
184
185Report Inventory Mismatch
186    [Documentation]  Report inventory mismatch.
187    [Arguments]  ${diff_rc}  ${json_diff_file_path}
188    # Description of argument(s):
189    # diff_rc              The failing return code from the difference check.
190    # json_diff_file_path  The file that has the latest inventory snapshot.
191
192    Log To Console  Significant difference in inventory found, rc=${diff_rc}
193    Log To Console  Differences are listed in file:  no_newline=true
194    Log To Console  ${json_diff_file_path}
195    Log To Console  File Contents:
196    Wait Until Created  ${json_diff_file_path}
197    ${file_contents}=  OperatingSystem.Get File  ${json_diff_file_path}
198    Log  ${file_contents}  level=WARN
199    Fail  Significant difference in inventory found, rc=${diff_rc}
200
201
202Loop HTX Health Check
203    [Documentation]  Run until HTX exerciser fails.
204    Repeat Keyword  ${HTX_DURATION}
205    ...  Run Keywords  Check HTX Run Status
206    ...  AND  Sleep  ${HTX_INTERVAL}
207
208
209Post Test Case Execution
210    [Documentation]  Do the post test teardown.
211    # 1. Shut down HTX exerciser if test Failed.
212    # 2. Capture FFDC on test failure.
213    # 3. Close all open SSH connections.
214
215    # Keep HTX running if user set HTX_KEEP_RUNNING to 1.
216    Run Keyword If
217    ...  '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0}
218    ...  Shutdown HTX Exerciser
219
220    ${keyword_buf}=  Catenate  Stop SOL Console Logging
221    ...  \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log
222    Run Key  ${keyword_buf}
223
224    FFDC On Test Case Fail
225    Close All Connections
226