1*** Settings *** 2Documentation Inventory of hardware resources under systems. 3 4Resource ../../../lib/bmc_redfish_resource.robot 5Resource ../../../lib/bmc_redfish_utils.robot 6Resource ../../../lib/logging_utils.robot 7Resource ../../../lib/openbmc_ffdc.robot 8Resource ../../../lib/ipmi_client.robot 9Library ../../../lib/logging_utils.py 10 11Test Setup Test Setup Execution 12Test Teardown Test Teardown Execution 13Suite Teardown Suite Teardown Execution 14 15** Variables *** 16 17${max_num_event_logs} ${200} 18 19*** Test Cases *** 20 21Event Log Check After BMC Reboot 22 [Documentation] Check event log after BMC rebooted. 23 [Tags] Event_Log_Check_After_BMC_Reboot 24 25 Redfish Purge Event Log 26 Event Log Should Not Exist 27 28 Redfish OBMC Reboot (off) 29 30 Redfish.Login 31 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 32 33 Event Log Should Not Exist 34 35 36Event Log Check After Host Poweron 37 [Documentation] Check event log after host has booted. 38 [Tags] Event_Log_Check_After_Host_Poweron 39 40 Redfish Purge Event Log 41 Event Log Should Not Exist 42 43 Redfish Power On 44 45 Redfish.Login 46 Event Log Should Not Exist 47 48 49Create Test Event Log And Verify 50 [Documentation] Create event logs and verify via redfish. 51 [Tags] Create_Test_Event_Log_And_Verify 52 53 Create Test Error Log 54 Event Log Should Exist 55 56 57Test Event Log Persistency On Restart 58 [Documentation] Restart logging service and verify event logs. 59 [Tags] Test_Event_Log_Persistency_On_Restart 60 61 Create Test Error Log 62 Event Log Should Exist 63 64 BMC Execute Command 65 ... systemctl restart xyz.openbmc_project.Logging.service 66 Sleep 10s reason=Wait for logging service to restart properly. 67 68 Event Log Should Exist 69 70 71Test Event Entry Numbering Reset On Restart 72 [Documentation] Restart logging service and verify event logs entry starts 73 ... from entry "Id" 1. 74 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart 75 76 #{ 77 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 78 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 79 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 80 # "Description": "Collection of System Event Log Entries", 81 # "Members": [ 82 # { 83 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 84 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 85 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 86 # "Created": "2019-05-29T13:19:27+00:00", 87 # "EntryType": "Event", 88 # "Id": "1", <----- Event log ID 89 # "Message": "org.open_power.Host.Error.Event", 90 # "Name": "System DBus Event Log Entry", 91 # "Severity": "Critical" 92 # } 93 # ], 94 # "Members@odata.count": 1, 95 # "Name": "System Event Log Entries" 96 #} 97 98 Create Test Error Log 99 Create Test Error Log 100 Event Log Should Exist 101 102 Redfish Purge Event Log 103 Event Log Should Not Exist 104 105 BMC Execute Command 106 ... systemctl restart xyz.openbmc_project.Logging.service 107 Sleep 10s reason=Wait for logging service to restart properly. 108 109 Create Test Error Log 110 ${elogs}= Get Event Logs 111 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1. 112 113 114Test Event Log Persistency On Reboot 115 [Documentation] Reboot BMC and verify event log. 116 [Tags] Test_Event_Log_Persistency_On_Reboot 117 118 Redfish Purge Event Log 119 Create Test Error Log 120 Event Log Should Exist 121 122 Redfish OBMC Reboot (off) 123 124 Redfish.Login 125 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries 126 127 Event Log Should Exist 128 129 130Create Test Event Log And Verify Time Stamp 131 [Documentation] Create event logs and verify time stamp. 132 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp 133 134 #{ 135 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", 136 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", 137 # "@odata.type": "#LogEntryCollection.LogEntryCollection", 138 # "Description": "Collection of System Event Log Entries", 139 # "Members": [ 140 # { 141 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 142 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", 143 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 144 # "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp 145 # "EntryType": "Event", 146 # "Id": "1", 147 # "Message": "org.open_power.Host.Error.Event", 148 # "Name": "System DBus Event Log Entry", 149 # "Severity": "Critical" 150 # } 151 # ], 152 # "Members@odata.count": 1, 153 # "Name": "System Event Log Entries" 154 #} 155 156 Redfish Purge Event Log 157 158 Create Test Error Log 159 Sleep 2s 160 Create Test Error Log 161 162 ${elog_entry}= Get Event Logs 163 164 # The event log generated is associated with the epoc time and unique 165 # for every error and in increasing time stamp. 166 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"]} epoch 167 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"]} epoch 168 169 Should Be True ${time_stamp2} > ${time_stamp1} 170 171 172Delete Non Existing SEL Event Entry 173 [Documentation] Delete non existing SEL event entry. 174 [Tags] Delete_Non_Existing_SEL_Event_Entry 175 176 ${sel_delete}= Run Keyword And Expect Error * 177 ... Run IPMI Standard Command sel delete 100 178 Should Contain ${sel_delete} Unable to delete entry 179 ... case_insensitive=True 180 181 182Delete Invalid SEL Event Entry 183 [Documentation] Delete invalid SEL event entry. 184 [Tags] Delete_Invalid_SEL_Event_Entry 185 186 ${sel_delete}= Run Keyword And Expect Error * 187 ... Run IPMI Standard Command sel delete abc 188 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid 189 ... case_insensitive=True 190 191 192Verify IPMI SEL Event Entries 193 [Documentation] Verify IPMI SEL's entries info. 194 [Tags] Verify_IPMI_SEL_Event_Entries 195 196 # Generate error logs of random count. 197 ${count}= Evaluate random.randint(1, 5) modules=random 198 Repeat Keyword ${count} Create Test Error Log 199 200 ${sel_entries_count}= Get IPMI SEL Setting Entries 201 Should Be Equal As Strings ${sel_entries_count} ${count} 202 203 204Verify IPMI SEL Event Last Add Time 205 [Documentation] Verify IPMI SEL's last added timestamp. 206 [Tags] Verify_IPMI_SEL_Event_Last_Add_Time 207 208 Create Test Error Log 209 ${sel_time}= Run IPMI Standard Command sel time get 210 ${sel_time}= Convert Date ${sel_time} 211 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True 212 213 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time 214 ${sel_last_add_time}= Convert Date ${sel_last_add_time} 215 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True 216 217 ${time_diff}= 218 ... Subtract Date From Date ${sel_last_add_time} ${sel_time} 219 220 # Verify if the delay in current time check and last add SEL time 221 # is less or equals to 2 seconds. 222 Should Be True ${time_diff} <= 2 223 224 225Create Test Event Log And Delete 226 [Documentation] Create an event log and delete it. 227 [Tags] Create_Test_Event_Log_And_Delete 228 229 Create Test Error Log 230 Redfish Purge Event Log 231 Event Log Should Not Exist 232 233 234Create Multiple Test Event Logs And Delete All 235 [Documentation] Create multiple event logs and delete all. 236 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All 237 238 Create Test Error Log 239 Create Test Error Log 240 Create Test Error Log 241 Redfish Purge Event Log 242 Event Log Should Not Exist 243 244 245Verify Watchdog Timedout Event 246 [Documentation] Trigger watchdog timed out and verify event log generated. 247 [Tags] Verify_Watchdog_Timedout_Event 248 249 Redfish Power On 250 251 # Clear errors if there are any. 252 Redfish.Login 253 Redfish Purge Event Log 254 255 Trigger Host Watchdog Error 256 257 # Logging takes time to generate the timeout error. 258 Wait Until Keyword Succeeds 2 min 30 sec 259 ... Verify Watchdog EventLog Content 260 261 Redfish Power Off 262 263 264Verify Event Logs Capping 265 [Documentation] Verify event logs capping. 266 [Tags] Verify_Event_Logs_Capping 267 268 Redfish Purge Event Log 269 270 ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c 271 ... AutoTestSimple; done 272 BMC Execute Command ${cmd} 273 274 ${elogs}= Get Event Logs 275 ${count}= Get Length ${elogs} 276 Run Keyword If ${count} > 200 277 ... Fail Error logs created exceeded max capacity 200. 278 279 280Test Event Log Wrapping 281 [Documentation] Verify event log entries wraps when 200 max cap is reached. 282 [Tags] Test_Event_Log_Wrapping 283 284 # Restarting logging service in order to clear logs and get the next log 285 # ID set to 1. 286 BMC Execute Command 287 ... systemctl restart xyz.openbmc_project.Logging.service 288 Sleep 10s reason=Wait for logging service to restart properly. 289 290 # Create ${max_num_event_logs} event logs. 291 ${cmd}= Catenate for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c 292 ... AutoTestSimple;done 293 BMC Execute Command ${cmd} 294 295 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist. 296 ${event_log}= Get Event Logs 297 298 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 299 Rprint Vars log_entries fmt=terse 300 Should Be Equal As Strings ${log_entries[0]["Id"]} 1 301 302 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')] 303 Rprint Vars log_entries fmt=terse 304 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs} 305 306 # Create event log and verify the entry ID, ${max_num_event_logs + 1}. 307 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1} 308 309 Create Test Error Log 310 311 ${event_log}= Get Event Logs 312 313 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')] 314 Rprint Vars log_entries fmt=terse 315 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id} 316 317 # Event log 1 should be wrapped. 318 ${log_entries}= Filter Struct ${event_log} [('Id', '1')] 319 Rprint Vars log_entries fmt=terse 320 321 ${length_log_entries} Get Length ${log_entries} 322 Should Be Equal As Integers ${length_log_entries} 0 323 ... msg=The event log should have wrapped such that entry ID 1 is now purged. 324 325 326*** Keywords *** 327 328Suite Teardown Execution 329 [Documentation] Do the post suite teardown. 330 331 Redfish.Logout 332 333 334Test Setup Execution 335 [Documentation] Do test case setup tasks. 336 337 Redfish.Login 338 339 Redfish Purge Event Log 340 341 ${status}= Run Keyword And Return Status Logging Test Binary Exist 342 Run Keyword If ${status} == ${False} Install Tarball 343 344 345Test Teardown Execution 346 [Documentation] Do the post test teardown. 347 348 FFDC On Test Case Fail 349 Redfish.Login 350 Redfish Purge Event Log 351 352 353Event Log Should Not Exist 354 [Documentation] Event log entries should not exist. 355 356 ${elogs}= Get Event Logs 357 Should Be Empty ${elogs} msg=System event log entry is not empty. 358 359 360Event Log Should Exist 361 [Documentation] Event log entries should exist. 362 363 ${elogs}= Get Event Logs 364 Should Not Be Empty ${elogs} msg=System event log entry is not empty. 365 366 367Verify Watchdog EventLog Content 368 [Documentation] Verify watchdog event log content. 369 370 # Example: 371 # { 372 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", 373 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31", 374 # "@odata.type": "#LogEntry.v1_4_0.LogEntry", 375 # "Created": "2019-05-31T18:41:33+00:00", 376 # "EntryType": "Event", 377 # "Id": "31", 378 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut", 379 # "Name": "System DBus Event Log Entry", 380 # "Severity": "Critical" 381 # } 382 383 ${elog}= Get Event Logs 384 Should Be Equal As Strings 385 ... ${elog[0]["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut 386 ... msg=Watchdog timeout event log was not found. 387 Should Be Equal As Strings 388 ... ${elog[0]["Severity"]} Critical 389 ... msg=Watchdog timeout severity unexpected value. 390