1*** Settings *** 2Documentation Inventory of hardware resources under systems. 3 4Resource ../../../lib/bmc_redfish_resource.robot 5Resource ../../../lib/bmc_redfish_utils.robot 6Resource ../../../lib/logging_utils.robot 7Resource ../../../lib/openbmc_ffdc.robot 8Resource ../../../lib/ipmi_client.robot 9Library ../../../lib/logging_utils.py 10 11Suite Setup Suite Setup Execution 12Suite Teardown Suite Teardown Execution 13Test Setup Test Setup Execution 14Test Teardown Test Teardown Execution 15 16Force Tags Event_Logging 17 18** Variables *** 19 20${max_num_event_logs} ${200} 21 22*** Test Cases *** 23 24Event Log Check After BMC Reboot 25 [Documentation] Check event log after BMC rebooted. 26 [Tags] Event_Log_Check_After_BMC_Reboot 27 28 Redfish Purge Event Log 29 Event Log Should Not Exist 30 31 Redfish OBMC Reboot (off) 32 33 Redfish.Login 34 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 35 36 Event Log Should Not Exist 37 38 39Event Log Check After Host Poweron 40 [Documentation] Check event log after host has booted. 41 [Tags] Event_Log_Check_After_Host_Poweron 42 43 Redfish Purge Event Log 44 Event Log Should Not Exist 45 46 Redfish Power On 47 48 Redfish.Login 49 Event Log Should Not Exist 50 51 52Create Test Event Log And Verify 53 [Documentation] Create event logs and verify via redfish. 54 [Tags] Create_Test_Event_Log_And_Verify 55 56 Create Test Error Log 57 Event Log Should Exist 58 59 60Delete Redfish Event Log And Verify 61 [Documentation] Delete Redfish event log and verify via Redfish. 62 [Tags] Delete_Redfish_Event_Log_And_Verify 63 64 Redfish.Login 65 Redfish Purge Event Log 66 Create Test PEL Log 67 ${elog_entry}= Get Event Logs 68 69 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${elog_entry[0]["Id"]} 70 71 ${error_entries}= Get Redfish Error Entries 72 Should Be Empty ${error_entries} 73 74 75Test Event Log Persistency On Restart 76 [Documentation] Restart logging service and verify event logs. 77 [Tags] Test_Event_Log_Persistency_On_Restart 78 79 Create Test Error Log 80 Event Log Should Exist 81 82 BMC Execute Command 83 ... systemctl restart xyz.openbmc_project.Logging.service 84 Sleep 10s reason=Wait for logging service to restart properly. 85 86 Event Log Should Exist 87 88 89Test Event Entry Numbering Reset On Restart 90 [Documentation] Restart logging service and verify event logs entry starts 91 ... from entry "Id" 1. 92 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart 93 [Setup] Redfish Power Off stack_mode=skip 94 95 #{ 96 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 97 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 98 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 99 # "Description": "Collection of System Event Log Entries", 100 # "Members": [ 101 # { 102 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 103 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 104 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 105 # "Created": "2019-05-29T13:19:27+00:00", 106 # "EntryType": "Event", 107 # "Id": "1", <----- Event log ID 108 # "Message": "org.open_power.Host.Error.Event", 109 # "Name": "System DBus Event Log Entry", 110 # "Severity": "Critical" 111 # } 112 # ], 113 # "Members@odata.count": 1, 114 # "Name": "System Event Log Entries" 115 #} 116 117 Create Test PEL Log 118 Create Test PEL Log 119 Event Log Should Exist 120 121 Redfish Purge Event Log 122 Event Log Should Not Exist 123 124 BMC Execute Command 125 ... systemctl restart xyz.openbmc_project.Logging.service 126 Sleep 10s reason=Wait for logging service to restart properly. 127 128 Create Test PEL Log 129 ${elogs}= Get Event Logs 130 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1 131 132 133Test Event Log Persistency On Reboot 134 [Documentation] Reboot BMC and verify event log. 135 [Tags] Test_Event_Log_Persistency_On_Reboot 136 137 Redfish Purge Event Log 138 Create Test Error Log 139 Event Log Should Exist 140 141 Redfish OBMC Reboot (off) 142 143 Redfish.Login 144 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 145 146 Event Log Should Exist 147 148 149Create Test Event Log And Verify Time Stamp 150 [Documentation] Create event logs and verify time stamp. 151 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp 152 153 #{ 154 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 155 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 156 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 157 # "Description": "Collection of System Event Log Entries", 158 # "Members": [ 159 # { 160 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 161 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 162 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 163 # "Created": "2023-05-10T10:26:02.186+00:00", <--- Time stamp 164 # "EntryType": "Event", 165 # "Id": "1", 166 # "Message": "org.open_power.Host.Error.Event", 167 # "Name": "System DBus Event Log Entry", 168 # "Severity": "Critical" 169 # } 170 # ], 171 # "Members@odata.count": 1, 172 # "Name": "System Event Log Entries" 173 #} 174 175 Redfish Purge Event Log 176 177 Create Test Error Log 178 Sleep 2s 179 Create Test Error Log 180 181 ${elog_entry}= Get Event Logs 182 183 # The event log generated is associated with the epoc time and unique 184 # for every error and in increasing time stamp. 185 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch 186 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"].split('.')[0]} epoch 187 188 Should Be True ${time_stamp2} > ${time_stamp1} 189 190 191# TODO: openbmc/openbmc-test-automation#1789 192Verify Setting Error Log As Resolved 193 [Documentation] Verify modified field of error log is updated when error log is marked resolved. 194 [Tags] Verify_Setting_Error_Log_As_Resolved 195 196 Create Test PEL Log 197 ${elog_entry}= Get Event Logs 198 199 # Wait for 5 seconds after creating error log. 200 Sleep 5s 201 202 # Mark error log as resolved by setting it to true. 203 Redfish.Patch ${EVENT_LOG_URI}Entries/${elog_entry[0]["Id"]} body={'Resolved':True} 204 205 ${elog_entry}= Get Event Logs 206 207 # Example error log with resolve field set to true: 208 # { 209 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/2045", 210 # "@odata.type": "#LogEntry.v1_8_0.LogEntry", 211 # "AdditionalDataURI": "/redfish/v1/Systems/system/LogServices/EventLog/attachment/2045", 212 # "Created": "2023-05-10T10:26:02.186+00:00", 213 # "EntryType": "Event", 214 # "Id": "2045", 215 # "Message": "xyz.openbmc_project.Host.Error.Event", 216 # "Modified": "2023-05-10T10:26:02.186+00:00", 217 # "Name": "System Event Log Entry", 218 # "Resolved": true, 219 # "Severity": "OK" 220 # } 221 222 Should Be Equal As Strings ${elog_entry[0]["Resolved"]} True 223 224 # Difference created and modified time of error log should be around 5 seconds. 225 ${creation_time}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch 226 ${modification_time}= Convert Date ${elog_entry[0]["Modified"].split('.')[0]} epoch 227 228 ${diff}= Subtract Date From Date ${modification_time} ${creation_time} 229 ${diff}= Convert To Number ${diff} 230 Should Be True 4 < ${diff} < 8 231 232 233Verify IPMI SEL Delete 234 [Documentation] Verify IPMI SEL delete operation. 235 [Tags] Verify_IPMI_SEL_Delete 236 237 Redfish Purge Event Log 238 Create Test Error Log 239 240 ${sel_list}= Run IPMI Standard Command sel list 241 Should Not Be Equal As Strings ${sel_list} SEL has no entries 242 243 # Example of SEL List: 244 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted 245 246 ${sel_entry}= Fetch from Left ${sel_list} | 247 ${sel_entry}= Evaluate $sel_entry.replace(' ','') 248 ${sel_entry}= Convert To Integer 0x${sel_entry} 249 250 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry} 251 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry} 252 ... case_insensitive=True 253 254 ${sel_list}= Run IPMI Standard Command sel list 255 Should Be Equal As Strings ${sel_list} SEL has no entries 256 ... case_insensitive=True 257 258 259Delete Non Existing SEL Event Entry 260 [Documentation] Delete non existing SEL event entry. 261 [Tags] Delete_Non_Existing_SEL_Event_Entry 262 263 ${sel_delete}= Run Keyword And Expect Error * 264 ... Run IPMI Standard Command sel delete 100 265 Should Contain ${sel_delete} Unable to delete entry 266 ... case_insensitive=True 267 268 269Delete Invalid SEL Event Entry 270 [Documentation] Delete invalid SEL event entry. 271 [Tags] Delete_Invalid_SEL_Event_Entry 272 273 ${sel_delete}= Run Keyword And Expect Error * 274 ... Run IPMI Standard Command sel delete abc 275 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid 276 ... case_insensitive=True 277 278 279Verify IPMI SEL Event Entries 280 [Documentation] Verify IPMI SEL's entries info. 281 [Tags] Verify_IPMI_SEL_Event_Entries 282 283 # Generate error logs of random count. 284 ${count}= Evaluate random.randint(1, 5) modules=random 285 Repeat Keyword ${count} Create Test Error Log 286 287 ${sel_entries_count}= Get IPMI SEL Setting Entries 288 Should Be Equal As Strings ${sel_entries_count} ${count} 289 290 291Verify IPMI SEL Event Last Add Time 292 [Documentation] Verify IPMI SEL's last added timestamp. 293 [Tags] Verify_IPMI_SEL_Event_Last_Add_Time 294 295 Create Test Error Log 296 ${sel_time}= Run IPMI Standard Command sel time get 297 ${sel_time}= Convert Date ${sel_time} 298 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True 299 300 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time 301 ${sel_last_add_time}= Convert Date ${sel_last_add_time} 302 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True 303 304 ${time_diff}= 305 ... Subtract Date From Date ${sel_last_add_time} ${sel_time} 306 307 # Verify if the delay in current time check and last add SEL time 308 # is less or equals to 2 seconds. 309 Should Be True ${time_diff} <= 2 310 311 312Create Test Event Log And Delete 313 [Documentation] Create an event log and delete it. 314 [Tags] Create_Test_Event_Log_And_Delete 315 316 Create Test Error Log 317 Redfish Purge Event Log 318 Event Log Should Not Exist 319 320 321Create Multiple Test Event Logs And Delete All 322 [Documentation] Create multiple event logs and delete all. 323 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All 324 325 Create Test Error Log 326 Create Test Error Log 327 Create Test Error Log 328 Redfish Purge Event Log 329 Event Log Should Not Exist 330 331 332Create Two Test Event Logs And Delete One 333 [Documentation] Create two event logs and delete the first entry. 334 [Tags] Create_Two_Test_Event_Logs_And_Delete_One 335 [Setup] Redfish Power Off stack_mode=skip 336 337 Redfish Purge Event Log 338 Create Test PEL Log 339 Create Test PEL Log 340 ${error_entries_before}= Get Redfish Error Entries 341 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${error_entries_before[0]} 342 343 ${error_entries_after}= Get Redfish Error Entries 344 Should Not Contain ${error_entries_after} ${error_entries_before[0]} 345 Should Contain ${error_entries_after} ${error_entries_before[1]} 346 347 348Verify Watchdog Timedout Event 349 [Documentation] Trigger watchdog timed out and verify event log generated. 350 [Tags] Verify_Watchdog_Timedout_Event 351 [Teardown] Run Keywords Test Teardown Execution AND Redfish Power Off stack_mode=skip 352 353 Redfish Power Off stack_mode=skip 354 355 # Clear errors if there are any. 356 Redfish.Login 357 Redfish Purge Event Log 358 359 # Reference: [Old legacy REST code] Trigger Host Watchdog Error 360 # Currently, no known redfish interface to set to trigger watchdog timer. 361 362 Redfish Initiate Auto Reboot 1000 363 364 # Logging takes time to generate the timeout error. 365 Wait Until Keyword Succeeds 3 min 20 sec Verify Watchdog EventLog Content 366 367 368Verify Event Logs Capping 369 [Documentation] Verify event logs capping. 370 [Tags] Verify_Event_Logs_Capping 371 372 Redfish Purge Event Log 373 374 ${cmd}= Catenate uptime; for i in {1..201}; do /tmp/tarball/bin/logging-test -c 375 ... AutoTestSimple;sleep 1;done; uptime 376 BMC Execute Command ${cmd} 377 378 ${elogs}= Get Event Logs 379 ${count}= Get Length ${elogs} 380 Run Keyword If ${count} > 200 381 ... Fail Error logs created exceeded max capacity 200. 382 383 384Test Event Log Wrapping 385 [Documentation] Verify event log entries wraps when 200 max cap is reached. 386 [Tags] Test_Event_Log_Wrapping 387 388 # Restarting logging service in order to clear logs and get the next log 389 # ID set to 1. 390 BMC Execute Command 391 ... systemctl restart xyz.openbmc_project.Logging.service 392 Sleep 10s reason=Wait for logging service to restart properly. 393 394 # Create ${max_num_event_logs} event logs. 395 ${cmd}= Catenate uptime; for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c 396 ... AutoTestSimple;sleep 1;done; uptime 397 BMC Execute Command ${cmd} 398 399 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist. 400 ${event_log}= Get Event Logs 401 402 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 403 Rprint Vars log_entries 404 Should Be Equal As Strings ${log_entries[0]["Id"]} 1 405 406 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')] 407 Rprint Vars log_entries 408 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs} 409 410 # Create event log and verify the entry ID, ${max_num_event_logs + 1}. 411 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1} 412 413 Create Test Error Log 414 415 ${event_log}= Get Event Logs 416 417 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')] 418 Rprint Vars log_entries 419 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id} 420 421 # Event log 1 should be wrapped. 422 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 423 Rprint Vars log_entries 424 425 ${length_log_entries} Get Length ${log_entries} 426 Should Be Equal As Integers ${length_log_entries} 0 427 ... msg=The event log should have wrapped such that entry ID 1 is now purged. 428 429 430Verify Default Value Of Resolved Field Is False For An Error Log Via Redfish 431 [Documentation] Verify the Resolve field status is false for an error log from Redfish. 432 [Tags] Verify_Default_Value_Of_Resolved_Field_Is_False_For_An_Error_Log_Via_Redfish 433 434 Redfish Purge Event Log 435 Create Test Error Log 436 437 # Check resolve field value of created error log. 438 ${elog_entry}= Get Event Logs 439 Should Be Equal ${elog_entry[0]["Resolved"]} ${False} 440 441 442*** Keywords *** 443 444Suite Setup Execution 445 [Documentation] Do test case setup tasks. 446 447 Redfish.Login 448 449 Redfish Purge Event Log 450 451 ${status}= Run Keyword And Return Status Logging Test Binary Exist 452 Run Keyword If ${status} == ${False} Install Tarball 453 454 455Suite Teardown Execution 456 [Documentation] Do the post suite teardown. 457 458 Redfish.Logout 459 460 461Test Setup Execution 462 [Documentation] Do test case setup tasks. 463 464 Redfish Purge Event Log 465 466 ${status}= Run Keyword And Return Status Logging Test Binary Exist 467 Run Keyword If ${status} == ${False} Install Tarball 468 469 470Test Teardown Execution 471 [Documentation] Do the post test teardown. 472 473 FFDC On Test Case Fail 474 Redfish.Login 475 Redfish Purge Event Log 476 477 478Get Redfish Error Entries 479 [Documentation] Return Redfish error ids list. 480 ${error_uris}= redfish_utils.get_member_list /redfish/v1/Systems/system/LogServices/EventLog/Entries 481 ${error_ids}= Create List 482 483 FOR ${error_uri} IN @{error_uris} 484 ${error_id}= Fetch From Right ${error_uri} / 485 Append To List ${error_ids} ${error_id} 486 END 487 488 [Return] ${error_ids} 489 490 491Event Log Should Not Exist 492 [Documentation] Event log entries should not exist. 493 494 ${elogs}= Get Event Logs 495 Should Be Empty ${elogs} msg=System event log entry is not empty. 496 497 498Event Log Should Exist 499 [Documentation] Event log entries should exist. 500 501 ${elogs}= Get Event Logs 502 Should Not Be Empty ${elogs} msg=System event log entry is not empty. 503 504 505Verify Watchdog EventLog Content 506 [Documentation] Verify watchdog event log content. 507 508 # Example: 509 # { 510 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 511 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31", 512 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 513 # "Created": "2019-05-31T18:41:33+00:00", 514 # "EntryType": "Event", 515 # "Id": "31", 516 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut", 517 # "Name": "System DBus Event Log Entry", 518 # "Severity": "Critical" 519 # } 520 521 ${elog_list}= Get Event Logs 522 523 Rprint Vars elog_list 524 525 FOR ${entry} IN @{elog_list} 526 ${found_match}= Run Keyword And Return Status Is Watchdog Error Found ${entry} 527 Exit For Loop If '${found_match}' == 'True' 528 END 529 530 Run Keyword If '${found_match}' == 'False' Fail msg=No watchdog error logged. 531 532 533Is Watchdog Error Found 534 [Documentation] Check if the give log entry matches specific watchdog error. 535 [Arguments] ${elog} 536 537 # Description of argument(s): 538 # elog Error log entry dictionary data. 539 540 Should Contain Any 541 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut 542 ... CEC Hardware - Hostboot-Service Processor Interface 543 ... msg=Watchdog timeout event log was not found. 544 545 Log To Console Matched Found: ${elog} 546