1*** Settings *** 2Documentation Error logging utility keywords. 3 4Resource rest_client.robot 5Resource bmc_redfish_utils.robot 6Variables ../data/variables.py 7Variables ../data/pel_variables.py 8 9*** Variables *** 10 11 12# Define variables for use by callers of 'Get Error Logs'. 13${low_severity_errlog_regex} \\.(Informational|Notice|Debug|OK)$ 14&{low_severity_errlog_filter} Severity=${low_severity_errlog_regex} 15&{low_severity_errlog_filter_args} filter_dict=${low_severity_errlog_filter} regex=${True} invert=${True} 16# The following is equivalent to &{low_severity_errlog_filter_args} but the name may be more intuitive for 17# users. Example usage: 18# ${err_logs}= Get Error Logs &{filter_low_severity_errlogs} 19&{filter_low_severity_errlogs} &{low_severity_errlog_filter_args} 20 21*** Keywords *** 22 23Get Logging Entry List 24 [Documentation] Get logging entry and return the object list. 25 26 ${entry_list}= Create List 27 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}list quiet=${1} 28 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND} 29 ${jsondata}= To JSON ${resp.content} 30 31 FOR ${entry} IN @{jsondata["data"]} 32 Continue For Loop If '${entry.rsplit('/', 1)[1]}' == 'callout' 33 Append To List ${entry_list} ${entry} 34 END 35 36 # Logging entries list. 37 # ['/xyz/openbmc_project/logging/entry/14', 38 # '/xyz/openbmc_project/logging/entry/15'] 39 [Return] ${entry_list} 40 41 42Logging Entry Should Exist 43 [Documentation] Find the matching message id and return the entry id. 44 [Arguments] ${message_id} 45 46 # Description of argument(s): 47 # message_id Logging message string. 48 # Example: "xyz.openbmc_project.Common.Error.InternalFailure" 49 50 @{elog_entries}= Get Logging Entry List 51 52 FOR ${entry} IN @{elog_entries} 53 ${resp}= Read Properties ${entry} 54 ${status}= Run Keyword And Return Status 55 ... Should Be Equal As Strings ${message_id} ${resp["Message"]} 56 Return From Keyword If ${status} == ${TRUE} ${entry} 57 END 58 59 Fail No ${message_id} logging entry found. 60 61 62Get Error Logs 63 [Documentation] Return the BMC error logs as a dictionary. 64 [Arguments] ${quiet}=1 &{filter_struct_args} 65 66 # Example of call using pre-defined filter args (defined above). 67 68 # ${err_logs}= Get Error Logs &{filter_low_severity_errlogs} 69 70 # In this example, all error logs with "Severity" fields that are neither Informational, Debug nor 71 # Notice will be returned. 72 73 # Description of argument(s): 74 # quiet Indicates whether this keyword should run without any output to the 75 # console, 0 = verbose, 1 = quiet. 76 # filter_struct_args filter_struct args (e.g. filter_dict, regex, etc.) to be passed directly 77 # to the Filter Struct keyword. See its prolog for details. 78 79 # The length of the returned dictionary indicates how many logs there are. 80 81 # Use 'Print Error Logs' to print. Example: 82 83 # Print Error Logs ${error_logs} Message. 84 85 ${status} ${error_logs}= Run Keyword And Ignore Error Read Properties 86 ... /xyz/openbmc_project/logging/entry/enumerate timeout=30 quiet=${quiet} 87 Return From Keyword If '${status}' == 'FAIL' &{EMPTY} 88 ${num_filter_struct_args}= Get Length ${filter_struct_args} 89 Return From Keyword If '${num_filter_struct_args}' == '${0}' ${error_logs} 90 ${filtered_error_logs}= Filter Struct ${error_logs} &{filter_struct_args} 91 [Return] ${filtered_error_logs} 92 93 94Get IPMI SEL Setting 95 [Documentation] Returns status for given IPMI SEL setting. 96 [Arguments] ${setting} 97 # Description of argument(s): 98 # setting SEL setting which needs to be read(e.g. "Last Add Time"). 99 100 ${resp}= Run IPMI Standard Command sel info 101 102 ${setting_line}= Get Lines Containing String ${resp} ${setting} 103 ... case-insensitive 104 ${setting_status}= Fetch From Right ${setting_line} :${SPACE} 105 106 [Return] ${setting_status} 107 108 109Verify Watchdog Errorlog Content 110 [Documentation] Verify watchdog errorlog content. 111 # Example: 112 # "/xyz/openbmc_project/logging/entry/1": 113 # { 114 # "AdditionalData": [], 115 # "Id": 1, 116 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut", 117 # "Resolved": 0, 118 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", 119 # "Timestamp": 1492715244828, 120 # "Associations": [] 121 # }, 122 123 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} 124 ${elog}= Read Properties ${elog_entry[0]} 125 Should Be Equal As Strings 126 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut 127 ... msg=Watchdog timeout error log was not found. 128 Should Be Equal As Strings 129 ... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Error 130 ... msg=Watchdog timeout severity unexpected value. 131 132 133Logging Test Binary Exist 134 [Documentation] Verify existence of prerequisite logging-test. 135 ${stdout} ${stderr} ${rc}= 136 ... BMC Execute Command test -f /tmp/tarball/bin/logging-test print_out=1 137 Should Be Empty ${stderr} msg=Logging Test stderr is non-empty. 138 139 140Clear Existing Error Logs 141 [Documentation] If error log isn't empty, reboot the BMC to clear the log. 142 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} 143 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND} 144 Initiate BMC Reboot 145 Wait Until Keyword Succeeds 10 min 10 sec 146 ... Is BMC Ready 147 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} 148 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} 149 ... msg=Could not clear BMC error logs. 150 151 152Create Test PEL Log 153 [Documentation] Generate test PEL log. 154 [Arguments] ${pel_type}=Unrecoverable Error 155 156 # Description of argument(s): 157 # pel_type The PEL type (e.g. Internal Failure, FRU Callout, Procedural Callout). 158 159 # Test PEL log entry example: 160 # { 161 # "0x5000002D": { 162 # "SRC": "BD8D1002", 163 # "Message": "An application had an internal failure", 164 # "PLID": "0x5000002D", 165 # "CreatorID": "BMC", 166 # "Subsystem": "BMC Firmware", 167 # "Commit Time": "02/25/2020 04:47:09", 168 # "Sev": "Unrecoverable Error", 169 # "CompID": "0x1000" 170 # } 171 # } 172 173 Run Keyword If '${pel_type}' == 'Internal Failure' 174 ... BMC Execute Command ${CMD_INTERNAL_FAILURE} 175 ... ELSE IF '${pel_type}' == 'FRU Callout' 176 ... BMC Execute Command ${CMD_FRU_CALLOUT} 177 ... ELSE IF '${pel_type}' == 'Procedure And Symbolic FRU Callout' 178 ... BMC Execute Command ${CMD_PROCEDURAL_SYMBOLIC_FRU_CALLOUT} 179 ... ELSE IF '${pel_type}' == 'Unrecoverable Error' 180 ... BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} 181 182 183Create Test Error Log 184 [Documentation] Generate test error log. 185 # Test error log entry example: 186 # "/xyz/openbmc_project/logging/entry/1": { 187 # "AdditionalData": [ 188 # "STRING=FOO" 189 # ], 190 # "Id": 1, 191 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", 192 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", 193 # "Timestamp": 1487743963328, 194 # "Associations": [] 195 # } 196 BMC Execute Command /tmp/tarball/bin/logging-test -c AutoTestSimple 197 198Count Error Entries 199 [Documentation] Count Error entries. 200 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY} 201 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} 202 ... msg=Failed to get error logs. 203 ${jsondata}= To JSON ${resp.content} 204 ${count}= Get Length ${jsondata["data"]} 205 [Return] ${count} 206 207Verify Test Error Log 208 [Documentation] Verify test error log entries. 209 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} 210 ${entry_id}= Read Attribute ${elog_entry[0]} Message 211 Should Be Equal ${entry_id} 212 ... example.xyz.openbmc_project.Example.Elog.AutoTestSimple 213 ... msg=Error log not from AutoTestSimple. 214 ${entry_id}= Read Attribute ${elog_entry[0]} Severity 215 Should Be Equal ${entry_id} 216 ... xyz.openbmc_project.Logging.Entry.Level.Error 217 ... msg=Error log severity mismatch. 218 219Delete Error Logs And Verify 220 [Documentation] Delete all error logs and verify. 221 Delete All Error Logs 222 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}list quiet=${1} 223 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} 224 ... msg=Error logs not deleted as expected. 225 226 227Install Tarball 228 [Documentation] Install tarball on BMC. 229 Should Not Be Empty ${DEBUG_TARBALL_PATH} 230 ... msg=Debug tarball path value is required. 231 BMC Execute Command rm -rf /tmp/tarball 232 Install Debug Tarball On BMC ${DEBUG_TARBALL_PATH} 233 234 235Get Event Logs 236 [Documentation] Get all available EventLog entries. 237 238 #{ 239 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 240 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 241 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 242 # "Description": "Collection of System Event Log Entries", 243 # "Members": [ 244 # { 245 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 246 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 247 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 248 # "Created": "2019-05-29T13:19:27+00:00", 249 # "EntryType": "Event", 250 # "Id": "1", 251 # "Message": "org.open_power.Host.Error.Event", 252 # "Name": "System DBus Event Log Entry", 253 # "Severity": "Critical" 254 # } 255 # ], 256 # "Members@odata.count": 1, 257 # "Name": "System Event Log Entries" 258 #} 259 260 ${members}= Redfish.Get Attribute ${EVENT_LOG_URI}Entries Members 261 [Return] ${members} 262 263 264Get Redfish Event Logs 265 [Documentation] Pack the list of all available EventLog entries in dictionary. 266 [Arguments] ${quiet}=1 &{filter_struct_args} 267 268 # Description of argument(s): 269 # quiet Indicates whether this keyword should run without any output to the 270 # console, 0 = verbose, 1 = quiet. 271 # filter_struct_args filter_struct args (e.g. filter_dict, regex, etc.) to be passed 272 # directly to the Filter Struct keyword. See its prolog for details. 273 274 ${packed_dict}= Create Dictionary 275 ${error_logs}= Get Event Logs 276 277 FOR ${idx} IN @{error_logs} 278 Set To Dictionary ${packed_dict} ${idx['@odata.id']}=${idx} 279 END 280 281 ${num_filter_struct_args}= Get Length ${filter_struct_args} 282 Return From Keyword If '${num_filter_struct_args}' == '${0}' &{packed_dict} 283 ${filtered_error_logs}= Filter Struct ${packed_dict} &{filter_struct_args} 284 285 [Return] ${filtered_error_logs} 286 287 288Get Event Logs Not Ok 289 [Documentation] Get all event logs where the 'Severity' is not 'OK'. 290 291 ${members}= Get Event Logs 292 ${severe_logs}= Evaluate [elog for elog in $members if elog['Severity'] != 'OK'] 293 [Return] ${severe_logs} 294 295 296Get Number Of Event Logs 297 [Documentation] Return the number of EventLog members. 298 299 ${members}= Get Event Logs 300 ${num_members}= Get Length ${members} 301 [Return] ${num_members} 302 303 304Redfish Purge Event Log 305 [Documentation] Do Redfish EventLog purge. 306 307 ${target_action}= redfish_utils.Get Target Actions 308 ... /redfish/v1/Systems/system/LogServices/EventLog/ LogService.ClearLog 309 Redfish.Post ${target_action} body={'target': '${target_action}'} 310 ... valid_status_codes=[${HTTP_OK}, ${HTTP_NO_CONTENT}] 311 312 313Event Log Should Not Exist 314 [Documentation] Event log entries should not exist. 315 316 ${elogs}= Get Event Logs 317 Should Be Empty ${elogs} msg=System event log entry is not empty. 318