1*** Settings *** 2Documentation Inventory of hardware resources under systems. 3 4Resource ../../../lib/bmc_redfish_resource.robot 5Resource ../../../lib/bmc_redfish_utils.robot 6Resource ../../../lib/logging_utils.robot 7Resource ../../../lib/openbmc_ffdc.robot 8Resource ../../../lib/ipmi_client.robot 9Library ../../../lib/logging_utils.py 10 11Suite Setup Suite Setup Execution 12Suite Teardown Suite Teardown Execution 13Test Setup Test Setup Execution 14Test Teardown Test Teardown Execution 15 16Force Tags Event_Logging 17 18** Variables *** 19 20${max_num_event_logs} ${200} 21 22*** Test Cases *** 23 24Event Log Check After BMC Reboot 25 [Documentation] Check event log after BMC rebooted. 26 [Tags] Event_Log_Check_After_BMC_Reboot 27 28 Redfish Purge Event Log 29 Event Log Should Not Exist 30 31 Redfish OBMC Reboot (off) 32 33 Redfish.Login 34 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 35 36 Event Log Should Not Exist 37 38 39Event Log Check After Host Poweron 40 [Documentation] Check event log after host has booted. 41 [Tags] Event_Log_Check_After_Host_Poweron 42 43 Redfish Purge Event Log 44 Event Log Should Not Exist 45 46 Redfish Power On 47 48 Redfish.Login 49 Event Log Should Not Exist 50 51 52Create Test Event Log And Verify 53 [Documentation] Create event logs and verify via redfish. 54 [Tags] Create_Test_Event_Log_And_Verify 55 56 Create Test Error Log 57 Event Log Should Exist 58 59 60Delete Redfish Event Log And Verify 61 [Documentation] Delete Redfish event log and verify via Redfish. 62 [Tags] Delete_Redfish_Event_Log_And_Verify 63 64 Redfish.Login 65 Redfish Purge Event Log 66 Create Test PEL Log 67 ${elog_entry}= Get Event Logs 68 69 Redfish.Delete /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries/${elog_entry[0]["Id"]} 70 71 ${error_entries}= Get Redfish Error Entries 72 Should Be Empty ${error_entries} 73 74 75Test Event Log Persistency On Restart 76 [Documentation] Restart logging service and verify event logs. 77 [Tags] Test_Event_Log_Persistency_On_Restart 78 79 Create Test Error Log 80 Event Log Should Exist 81 82 BMC Execute Command 83 ... systemctl restart xyz.openbmc_project.Logging.service 84 Sleep 10s reason=Wait for logging service to restart properly. 85 86 Event Log Should Exist 87 88 89Test Event Entry Numbering Reset On Restart 90 [Documentation] Restart logging service and verify event logs entry starts 91 ... from entry "Id" 1. 92 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart 93 [Setup] Redfish Power Off stack_mode=skip 94 95 #{ 96 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 97 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 98 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 99 # "Description": "Collection of System Event Log Entries", 100 # "Members": [ 101 # { 102 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 103 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 104 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 105 # "Created": "2019-05-29T13:19:27+00:00", 106 # "EntryType": "Event", 107 # "Id": "1", <----- Event log ID 108 # "Message": "org.open_power.Host.Error.Event", 109 # "Name": "System DBus Event Log Entry", 110 # "Severity": "Critical" 111 # } 112 # ], 113 # "Members@odata.count": 1, 114 # "Name": "System Event Log Entries" 115 #} 116 117 Create Test PEL Log 118 Create Test PEL Log 119 Event Log Should Exist 120 121 Redfish Purge Event Log 122 Event Log Should Not Exist 123 124 BMC Execute Command 125 ... systemctl restart xyz.openbmc_project.Logging.service 126 Sleep 10s reason=Wait for logging service to restart properly. 127 128 Create Test PEL Log 129 ${elogs}= Get Event Logs 130 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1 131 132 133Test Event Log Persistency On Reboot 134 [Documentation] Reboot BMC and verify event log. 135 [Tags] Test_Event_Log_Persistency_On_Reboot 136 137 Redfish Purge Event Log 138 Create Test Error Log 139 Event Log Should Exist 140 141 Redfish OBMC Reboot (off) 142 143 Redfish.Login 144 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 145 146 Event Log Should Exist 147 148 149Create Test Event Log And Verify Time Stamp 150 [Documentation] Create event logs and verify time stamp. 151 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp 152 153 #{ 154 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 155 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 156 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 157 # "Description": "Collection of System Event Log Entries", 158 # "Members": [ 159 # { 160 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 161 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 162 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 163 # "Created": "2023-05-10T10:26:02.186+00:00", <--- Time stamp 164 # "EntryType": "Event", 165 # "Id": "1", 166 # "Message": "org.open_power.Host.Error.Event", 167 # "Name": "System DBus Event Log Entry", 168 # "Severity": "Critical" 169 # } 170 # ], 171 # "Members@odata.count": 1, 172 # "Name": "System Event Log Entries" 173 #} 174 175 Redfish Purge Event Log 176 177 Create Test Error Log 178 Sleep 2s 179 Create Test Error Log 180 181 ${elog_entry}= Get Event Logs 182 183 # The event log generated is associated with the epoc time and unique 184 # for every error and in increasing time stamp. 185 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch 186 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"].split('.')[0]} epoch 187 188 Should Be True ${time_stamp2} > ${time_stamp1} 189 190 191# TODO: openbmc/openbmc-test-automation#1789 192Verify Setting Error Log As Resolved 193 [Documentation] Verify modified field of error log is updated when error log is marked resolved. 194 [Tags] Verify_Setting_Error_Log_As_Resolved 195 196 Create Test PEL Log 197 ${elog_entry}= Get Event Logs 198 199 # Wait for 5 seconds after creating error log. 200 Sleep 5s 201 202 # Mark error log as resolved by setting it to true. 203 Redfish.Patch ${EVENT_LOG_URI}Entries/${elog_entry[0]["Id"]} body={'Resolved':True} 204 205 ${elog_entry}= Get Event Logs 206 207 # Example error log with resolve field set to true: 208 # { 209 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/2045", 210 # "@odata.type": "#LogEntry.v1_8_0.LogEntry", 211 # "AdditionalDataURI": "/redfish/v1/Systems/system/LogServices/EventLog/attachment/2045", 212 # "Created": "2023-05-10T10:26:02.186+00:00", 213 # "EntryType": "Event", 214 # "Id": "2045", 215 # "Message": "xyz.openbmc_project.Host.Error.Event", 216 # "Modified": "2023-05-10T10:26:02.186+00:00", 217 # "Name": "System Event Log Entry", 218 # "Resolved": true, 219 # "Severity": "OK" 220 # } 221 222 Should Be Equal As Strings ${elog_entry[0]["Resolved"]} True 223 224 # Difference created and modified time of error log should be around 5 seconds. 225 ${creation_time}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch 226 ${modification_time}= Convert Date ${elog_entry[0]["Modified"].split('.')[0]} epoch 227 228 ${diff}= Subtract Date From Date ${modification_time} ${creation_time} 229 ${diff}= Convert To Number ${diff} 230 Should Be True 4 < ${diff} < 8 231 232 233Verify IPMI SEL Delete 234 [Documentation] Verify IPMI SEL delete operation. 235 [Tags] Verify_IPMI_SEL_Delete 236 237 Redfish Purge Event Log 238 Create Test Error Log 239 240 ${sel_list}= Run IPMI Standard Command sel list 241 Should Not Be Equal As Strings ${sel_list} SEL has no entries 242 243 # Example of SEL List: 244 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted 245 246 ${sel_entry}= Fetch from Left ${sel_list} | 247 ${sel_entry}= Evaluate $sel_entry.replace(' ','') 248 ${sel_entry}= Convert To Integer 0x${sel_entry} 249 250 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry} 251 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry} 252 ... case_insensitive=True 253 254 ${sel_list}= Run IPMI Standard Command sel list 255 Should Be Equal As Strings ${sel_list} SEL has no entries 256 ... case_insensitive=True 257 258 259Verify IPMI SEL Event Entries 260 [Documentation] Verify IPMI SEL's entries info. 261 [Tags] Verify_IPMI_SEL_Event_Entries 262 263 # Generate error logs of random count. 264 ${count}= Evaluate random.randint(1, 5) modules=random 265 Repeat Keyword ${count} Create Test Error Log 266 267 ${sel_entries_count}= Get IPMI SEL Setting Entries 268 Should Be Equal As Strings ${sel_entries_count} ${count} 269 270 271Create Test Event Log And Delete 272 [Documentation] Create an event log and delete it. 273 [Tags] Create_Test_Event_Log_And_Delete 274 275 Create Test Error Log 276 Redfish Purge Event Log 277 Event Log Should Not Exist 278 279 280Create Multiple Test Event Logs And Delete All 281 [Documentation] Create multiple event logs and delete all. 282 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All 283 284 Create Test Error Log 285 Create Test Error Log 286 Create Test Error Log 287 Redfish Purge Event Log 288 Event Log Should Not Exist 289 290 291Create Two Test Event Logs And Delete One 292 [Documentation] Create two event logs and delete the first entry. 293 [Tags] Create_Two_Test_Event_Logs_And_Delete_One 294 [Setup] Redfish Power Off stack_mode=skip 295 296 Redfish Purge Event Log 297 Create Test PEL Log 298 Create Test PEL Log 299 ${error_entries_before}= Get Redfish Error Entries 300 Redfish.Delete /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries/${error_entries_before[0]} 301 302 ${error_entries_after}= Get Redfish Error Entries 303 Should Not Contain ${error_entries_after} ${error_entries_before[0]} 304 Should Contain ${error_entries_after} ${error_entries_before[1]} 305 306 307Verify Watchdog Timedout Event 308 [Documentation] Trigger watchdog timed out and verify event log generated. 309 [Tags] Verify_Watchdog_Timedout_Event 310 [Teardown] Run Keywords Test Teardown Execution AND Redfish Power Off stack_mode=skip 311 312 Redfish Power Off stack_mode=skip 313 314 # Clear errors if there are any. 315 Redfish.Login 316 Redfish Purge Event Log 317 318 # Reference: [Old legacy REST code] Trigger Host Watchdog Error 319 # Currently, no known redfish interface to set to trigger watchdog timer. 320 321 Redfish Initiate Auto Reboot 1000 322 323 # Logging takes time to generate the timeout error. 324 Wait Until Keyword Succeeds 3 min 20 sec Verify Watchdog EventLog Content 325 326 327Verify Event Logs Capping 328 [Documentation] Verify event logs capping. 329 [Tags] Verify_Event_Logs_Capping 330 331 Redfish Purge Event Log 332 333 ${cmd}= Catenate uptime; for i in {1..201}; do /tmp/tarball/bin/logging-test -c 334 ... AutoTestSimple;sleep 1;done; uptime 335 BMC Execute Command ${cmd} 336 337 ${elogs}= Get Event Logs 338 ${count}= Get Length ${elogs} 339 Run Keyword If ${count} > 200 340 ... Fail Error logs created exceeded max capacity 200. 341 342 343Test Event Log Wrapping 344 [Documentation] Verify event log entries wraps when 200 max cap is reached. 345 [Tags] Test_Event_Log_Wrapping 346 347 # Restarting logging service in order to clear logs and get the next log 348 # ID set to 1. 349 BMC Execute Command 350 ... systemctl restart xyz.openbmc_project.Logging.service 351 Sleep 10s reason=Wait for logging service to restart properly. 352 353 # Create ${max_num_event_logs} event logs. 354 ${cmd}= Catenate uptime; for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c 355 ... AutoTestSimple;sleep 1;done; uptime 356 BMC Execute Command ${cmd} 357 358 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist. 359 ${event_log}= Get Event Logs 360 361 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 362 Rprint Vars log_entries 363 Should Be Equal As Strings ${log_entries[0]["Id"]} 1 364 365 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')] 366 Rprint Vars log_entries 367 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs} 368 369 # Create event log and verify the entry ID, ${max_num_event_logs + 1}. 370 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1} 371 372 Create Test Error Log 373 374 ${event_log}= Get Event Logs 375 376 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')] 377 Rprint Vars log_entries 378 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id} 379 380 # Event log 1 should be wrapped. 381 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 382 Rprint Vars log_entries 383 384 ${length_log_entries} Get Length ${log_entries} 385 Should Be Equal As Integers ${length_log_entries} 0 386 ... msg=The event log should have wrapped such that entry ID 1 is now purged. 387 388 389Verify Default Value Of Resolved Field Is False For An Error Log Via Redfish 390 [Documentation] Verify the Resolve field status is false for an error log from Redfish. 391 [Tags] Verify_Default_Value_Of_Resolved_Field_Is_False_For_An_Error_Log_Via_Redfish 392 393 Redfish Purge Event Log 394 Create Test Error Log 395 396 # Check resolve field value of created error log. 397 ${elog_entry}= Get Event Logs 398 Should Be Equal ${elog_entry[0]["Resolved"]} ${False} 399 400 401*** Keywords *** 402 403Suite Setup Execution 404 [Documentation] Do test case setup tasks. 405 406 Redfish.Login 407 408 Redfish Purge Event Log 409 410 ${status}= Run Keyword And Return Status Logging Test Binary Exist 411 Run Keyword If ${status} == ${False} Install Tarball 412 413 414Suite Teardown Execution 415 [Documentation] Do the post suite teardown. 416 417 Redfish.Logout 418 419 420Test Setup Execution 421 [Documentation] Do test case setup tasks. 422 423 Redfish Purge Event Log 424 425 ${status}= Run Keyword And Return Status Logging Test Binary Exist 426 Run Keyword If ${status} == ${False} Install Tarball 427 428 429Test Teardown Execution 430 [Documentation] Do the post test teardown. 431 432 FFDC On Test Case Fail 433 Redfish.Login 434 Redfish Purge Event Log 435 436 437Get Redfish Error Entries 438 [Documentation] Return Redfish error ids list. 439 ${error_uris}= redfish_utils.get_member_list /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries 440 ${error_ids}= Create List 441 442 FOR ${error_uri} IN @{error_uris} 443 ${error_id}= Fetch From Right ${error_uri} / 444 Append To List ${error_ids} ${error_id} 445 END 446 447 [Return] ${error_ids} 448 449 450Event Log Should Not Exist 451 [Documentation] Event log entries should not exist. 452 453 ${elogs}= Get Event Logs 454 Should Be Empty ${elogs} msg=System event log entry is not empty. 455 456 457Event Log Should Exist 458 [Documentation] Event log entries should exist. 459 460 ${elogs}= Get Event Logs 461 Should Not Be Empty ${elogs} msg=System event log entry is not empty. 462 463 464Verify Watchdog EventLog Content 465 [Documentation] Verify watchdog event log content. 466 467 # Example: 468 # { 469 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 470 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31", 471 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 472 # "Created": "2019-05-31T18:41:33+00:00", 473 # "EntryType": "Event", 474 # "Id": "31", 475 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut", 476 # "Name": "System DBus Event Log Entry", 477 # "Severity": "Critical" 478 # } 479 480 ${elog_list}= Get Event Logs 481 482 Rprint Vars elog_list 483 484 FOR ${entry} IN @{elog_list} 485 ${found_match}= Run Keyword And Return Status Is Watchdog Error Found ${entry} 486 Exit For Loop If '${found_match}' == 'True' 487 END 488 489 Run Keyword If '${found_match}' == 'False' Fail msg=No watchdog error logged. 490 491 492Is Watchdog Error Found 493 [Documentation] Check if the give log entry matches specific watchdog error. 494 [Arguments] ${elog} 495 496 # Description of argument(s): 497 # elog Error log entry dictionary data. 498 499 Should Contain Any 500 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut 501 ... CEC Hardware - Hostboot-Service Processor Interface 502 ... msg=Watchdog timeout event log was not found. 503 504 Log To Console Matched Found: ${elog} 505