1*** Settings ***
2Documentation       Utility for RAS test scenarios through HOST & BMC.
3Resource            ../../lib/utils.robot
4Resource            ../../lib/openbmc_ffdc.robot
5Resource            ../../lib/openbmc_ffdc_utils.robot
6Resource            ../../lib/openbmc_ffdc_methods.robot
7Resource            ../../lib/ras/host_utils.robot
8Resource            ../../lib/resource.robot
9Resource            ../../lib/state_manager.robot
10Resource            ../../lib/boot_utils.robot
11Variables           ../../lib/ras/variables.py
12Variables           ../../data/variables.py
13Resource            ../../lib/dump_utils.robot
14
15Library             DateTime
16Library             OperatingSystem
17Library             random
18Library             Collections
19
20*** Variables ***
21${stack_mode}       normal
22
23*** Keywords ***
24
25Verify And Clear Gard Records On HOST
26    [Documentation]  Verify And Clear gard records on HOST.
27
28    ${output}=  Gard Operations On OS  list
29    Should Not Contain  ${output}  No GARD
30    Gard Operations On OS  clear all
31
32Verify Error Log Entry
33    [Documentation]  Verify error log entry & signature description.
34    [Arguments]  ${signature_desc}  ${log_prefix}
35    # Description of argument(s):
36    # signature_desc  Error log signature description.
37    # log_prefix      Log path prefix.
38
39    # TODO: Need to move this keyword to common utility.
40
41    Error Logs Should Exist
42
43    Collect eSEL Log  ${log_prefix}
44    ${error_log_file_path}=  Catenate  ${log_prefix}esel.txt
45    ${rc}  ${output}=  Run and Return RC and Output
46    ...  grep -i ${signature_desc} ${error_log_file_path}
47    Should Be Equal  ${rc}  ${0}
48    Should Not Be Empty  ${output}
49
50Inject Recoverable Error With Threshold Limit
51    [Documentation]  Inject and verify recoverable error on processor through
52    ...              host.
53    ...              Test sequence:
54    ...              1. Enable Auto Reboot Setting.
55    ...              2. Inject Error on processor/centaur.
56    ...              3. Check If HOST is running.
57    ...              4. Verify error log entry & signature description.
58    ...              4. Verify & clear gard records.
59    [Arguments]      ${interface_type}  ${fir}  ${chip_address}  ${threshold_limit}
60    ...              ${signature_desc}  ${log_prefix}
61    # Description of argument(s):
62    # interface_type      Inject error through 'BMC' or 'HOST'.
63    # fir                 FIR (Fault isolation register) value (e.g. 2011400).
64    # chip_address        Chip address (e.g 2000000000000000).
65    # threshold_limit     Threshold limit (e.g 1, 5, 32).
66    # signature_desc      Error log signature description.
67    # log_prefix          Log path prefix.
68
69    Set Auto Reboot  1
70    Run Keyword If  '${interface_type}' == 'HOST'
71    ...     Inject Error Through HOST  ${fir}  ${chip_address}  ${threshold_limit}
72    ...     ${master_proc_chip}
73    ...  ELSE
74    ...     Inject Error Through BMC  ${fir}  ${chip_address}  ${threshold_limit}
75    ...     ${master_proc_chip}
76
77    Is Host Running
78    ${output}=  Gard Operations On OS  list
79    Should Contain  ${output}  No GARD
80    Verify Error Log Entry  ${signature_desc}  ${log_prefix}
81    # TODO: Verify SOL console logs.
82
83
84Inject Unrecoverable Error Through Host
85    [Documentation]  Inject and verify recoverable error on processor through
86    ...              host.
87    ...              Test sequence:
88    ...              1. Enable Auto Reboot Setting.
89    ...              2. Inject Error on processor/centaur.
90    ...              3. Check If HOST is rebooted.
91    ...              4. Verify & clear gard records.
92    ...              5. Verify error log entry & signature description.
93    ...              6. Verify & clear dump entry.
94    [Arguments]      ${fir}  ${chip_address}  ${threshold_limit}
95    ...              ${signature_desc}  ${log_prefix}
96    # Description of argument(s):
97    # fir                 FIR (Fault isolation register) value (e.g. 2011400).
98    # chip_address        Chip address (e.g 2000000000000000).
99    # threshold_limit     Threshold limit (e.g 1, 5, 32).
100    # signature_desc      Error Log signature description.
101    #                     (e.g 'mcs(n0p0c0) (MCFIR[0]) mc internal recoverable')
102    # log_prefix          Log path prefix.
103
104    Set Auto Reboot  1
105    Inject Error Through HOST  ${fir}  ${chip_address}  ${threshold_limit}
106    ...  ${master_proc_chip}
107    Wait Until Keyword Succeeds  500 sec  20 sec  Is Host Rebooted
108    Wait for OS
109    Verify Error Log Entry  ${signature_desc}  ${log_prefix}
110    ${resp}=  OpenBMC Get Request  ${DUMP_ENTRY_URI}list
111    Should Not Be Equal As Strings  ${resp.status_code}  ${HTTP_NOT_FOUND}
112    Delete All BMC Dump
113    Verify And Clear Gard Records On HOST
114
115Fetch FIR Address Translation Value
116    [Documentation]  Fetch FIR address translation value through HOST.
117    [Arguments]  ${fir}  ${target_type}
118    # Description of argument(s):
119    # fir                  FIR (Fault isolation register) value (e.g. '2011400').
120    # core_id              Core ID (e.g. '9').
121    # target_type          Target type (e.g. 'EX', 'EQ', 'C').
122
123    Login To OS Host
124    Copy Address Translation Utils To HOST OS
125
126    # Fetch processor chip IDs.
127    ${proc_chip_id}=  Get ProcChipId From OS  Processor  ${master_proc_chip}
128    # Example output:
129    # 00000000
130
131    ${core_ids}=  Get Core IDs From OS  ${proc_chip_id[-1]}
132    # Example output:
133    #./probe_cpus.sh | grep 'CHIP ID: 0' | cut -c21-22
134    # ['14', '15', '16', '17']
135
136    # Ignoring master core ID.
137    ${output}=  Get Slice From List  ${core_ids}  1
138    # Feth random non-master core ID.
139    ${core_ids_sub_list}=   Evaluate  random.sample(${core_ids}, 1)  random
140    ${core_id}=  Get From List  ${core_ids_sub_list}  0
141    ${translated_fir_addr}=  FIR Address Translation Through HOST
142    ...  ${fir}  ${core_id}  ${target_type}
143
144    [Return]  ${translated_fir_addr}
145
146RAS Test SetUp
147    [Documentation]  Validates input parameters.
148
149    Should Not Be Empty
150    ...  ${OS_HOST}  msg=You must provide DNS name/IP of the OS host.
151    Should Not Be Empty
152    ...  ${OS_USERNAME}  msg=You must provide OS host user name.
153    Should Not Be Empty
154    ...  ${OS_PASSWORD}  msg=You must provide OS host user password.
155
156    # Boot to OS.
157    REST Power On  quiet=${1}
158    # Adding delay after host bring up.
159    Sleep  60s
160
161RAS Suite Setup
162    [Documentation]  Create RAS log directory to store all RAS test logs.
163
164    ${RAS_LOG_DIR_PATH}=  Catenate  ${EXECDIR}/RAS_logs/
165    Set Suite Variable  ${RAS_LOG_DIR_PATH}
166    Set Suite Variable  ${master_proc_chip}  False
167
168    Create Directory  ${RAS_LOG_DIR_PATH}
169    OperatingSystem.Directory Should Exist  ${RAS_LOG_DIR_PATH}
170    Empty Directory  ${RAS_LOG_DIR_PATH}
171
172    Should Not Be Empty  ${ESEL_BIN_PATH}
173    Set Environment Variable  PATH  %{PATH}:${ESEL_BIN_PATH}
174
175    # Boot to Os.
176    REST Power On  quiet=${1}
177
178    # Check Opal-PRD service enabled on host.
179    ${opal_prd_state}=  Is Opal-PRD Service Enabled
180    Run Keyword If  '${opal_prd_state}' == 'disabled'
181    ...  Enable Opal-PRD Service On HOST
182
183RAS Suite Cleanup
184    [Documentation]  Perform RAS suite cleanup and verify that host
185    ...              boots after test suite run.
186
187    # Boot to OS.
188    REST Power On  quiet=${1}
189    Delete Error Logs
190    Gard Operations On OS  clear all
191