1*** Settings ***
2Documentation       Inventory of hardware resources under systems.
3
4Resource            ../../../lib/bmc_redfish_resource.robot
5Resource            ../../../lib/bmc_redfish_utils.robot
6Resource            ../../../lib/logging_utils.robot
7Resource            ../../../lib/openbmc_ffdc.robot
8Resource            ../../../lib/ipmi_client.robot
9Library             ../../../lib/logging_utils.py
10
11Suite Setup         Suite Setup Execution
12Suite Teardown      Suite Teardown Execution
13Test Setup          Test Setup Execution
14Test Teardown       Test Teardown Execution
15
16** Variables ***
17
18${max_num_event_logs}  ${200}
19
20*** Test Cases ***
21
22Event Log Check After BMC Reboot
23    [Documentation]  Check event log after BMC rebooted.
24    [Tags]  Event_Log_Check_After_BMC_Reboot
25
26    Redfish Purge Event Log
27    Event Log Should Not Exist
28
29    Redfish OBMC Reboot (off)
30
31    Redfish.Login
32    Wait Until Keyword Succeeds  1 mins  15 secs   Redfish.Get  ${EVENT_LOG_URI}Entries
33
34    Event Log Should Not Exist
35
36
37Event Log Check After Host Poweron
38    [Documentation]  Check event log after host has booted.
39    [Tags]  Event_Log_Check_After_Host_Poweron
40
41    Redfish Purge Event Log
42    Event Log Should Not Exist
43
44    Redfish Power On
45
46    Redfish.Login
47    Event Log Should Not Exist
48
49
50Create Test Event Log And Verify
51    [Documentation]  Create event logs and verify via redfish.
52    [Tags]  Create_Test_Event_Log_And_Verify
53
54    Create Test Error Log
55    Event Log Should Exist
56
57
58Test Event Log Persistency On Restart
59    [Documentation]  Restart logging service and verify event logs.
60    [Tags]  Test_Event_Log_Persistency_On_Restart
61
62    Create Test Error Log
63    Event Log Should Exist
64
65    BMC Execute Command
66    ...  systemctl restart xyz.openbmc_project.Logging.service
67    Sleep  10s  reason=Wait for logging service to restart properly.
68
69    Event Log Should Exist
70
71
72Test Event Entry Numbering Reset On Restart
73    [Documentation]  Restart logging service and verify event logs entry starts
74    ...  from entry "Id" 1.
75    [Tags]  Test_Event_Entry_Numbering_Reset_On_Restart
76
77    #{
78    #  "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
79    #  "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
80    #  "@odata.type": "#LogEntryCollection.LogEntryCollection",
81    #  "Description": "Collection of System Event Log Entries",
82    #  "Members": [
83    #  {
84    #    "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
85    #    "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
86    #    "@odata.type": "#LogEntry.v1_4_0.LogEntry",
87    #    "Created": "2019-05-29T13:19:27+00:00",
88    #    "EntryType": "Event",
89    #    "Id": "1",               <----- Event log ID
90    #    "Message": "org.open_power.Host.Error.Event",
91    #    "Name": "System DBus Event Log Entry",
92    #    "Severity": "Critical"
93    #  }
94    #  ],
95    #  "Members@odata.count": 1,
96    #  "Name": "System Event Log Entries"
97    #}
98
99    Create Test Error Log
100    Create Test Error Log
101    Event Log Should Exist
102
103    Redfish Purge Event Log
104    Event Log Should Not Exist
105
106    BMC Execute Command
107    ...  systemctl restart xyz.openbmc_project.Logging.service
108    Sleep  10s  reason=Wait for logging service to restart properly.
109
110    Create Test Error Log
111    ${elogs}=  Get Event Logs
112    Should Be Equal  ${elogs[0]["Id"]}  1  msg=Event log entry is not 1.
113
114
115Test Event Log Persistency On Reboot
116    [Documentation]  Reboot BMC and verify event log.
117    [Tags]  Test_Event_Log_Persistency_On_Reboot
118
119    Redfish Purge Event Log
120    Create Test Error Log
121    Event Log Should Exist
122
123    Redfish OBMC Reboot (off)
124
125    Redfish.Login
126    Wait Until Keyword Succeeds  1 mins  15 secs   Redfish.Get  ${EVENT_LOG_URI}Entries
127
128    Event Log Should Exist
129
130
131# TODO: openbmc/openbmc-test-automation#1789
132Create Test Event Log And Verify Resolved Field
133    [Documentation]  Create event log and verify "Resolved" field is 0.
134    [Tags]  Create_Test_Event_Log_And_Verify_Resolved_Field
135
136    # Example Error log:
137    #  "/xyz/openbmc_project/logging/entry/1": {
138    #    "AdditionalData": [
139    #        "STRING=FOO"
140    #    ],
141    #    "Id": 1,
142    #    "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
143    #    "Resolved": 0,
144    #    "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
145    #    "Timestamp": 1490817164983,
146    #    "Associations": []
147    # },
148
149    # To mark an error as resolved, without deleting the error, user would
150    # set this bool property.
151    # In this test context we are making sure "Resolved" field is "0"
152    # by default.
153
154    Redfish Purge Event Log
155    Create Test Error Log
156    ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
157    ${resolved}=  Read Attribute  ${elog_entry[0]}  Resolved
158    Should Be True  ${resolved} == 0
159
160
161Create Test Event Log And Verify Time Stamp
162    [Documentation]  Create event logs and verify time stamp.
163    [Tags]  Create_Test_Event_Log_And_Verify_Time_Stamp
164
165    #{
166    #  "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
167    #  "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
168    #  "@odata.type": "#LogEntryCollection.LogEntryCollection",
169    #  "Description": "Collection of System Event Log Entries",
170    #  "Members": [
171    #  {
172    #    "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
173    #    "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
174    #    "@odata.type": "#LogEntry.v1_4_0.LogEntry",
175    #    "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp
176    #    "EntryType": "Event",
177    #    "Id": "1",
178    #    "Message": "org.open_power.Host.Error.Event",
179    #    "Name": "System DBus Event Log Entry",
180    #    "Severity": "Critical"
181    #  }
182    #  ],
183    #  "Members@odata.count": 1,
184    #  "Name": "System Event Log Entries"
185    #}
186
187    Redfish Purge Event Log
188
189    Create Test Error Log
190    Sleep  2s
191    Create Test Error Log
192
193    ${elog_entry}=  Get Event Logs
194
195    # The event log generated is associated with the epoc time and unique
196    # for every error and in increasing time stamp.
197    ${time_stamp1}=  Convert Date  ${elog_entry[0]["Created"]}  epoch
198    ${time_stamp2}=  Convert Date  ${elog_entry[1]["Created"]}  epoch
199
200    Should Be True  ${time_stamp2} > ${time_stamp1}
201
202
203Verify IPMI SEL Delete
204    [Documentation]  Verify IPMI SEL delete operation.
205    [Tags]  Verify_IPMI_SEL_Delete
206
207    Redfish Purge Event Log
208    Create Test Error Log
209
210    ${sel_list}=  Run IPMI Standard Command  sel list
211    Should Not Be Equal As Strings  ${sel_list}  SEL has no entries
212
213    # Example of SEL List:
214    # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
215
216    ${sel_entry}=  Fetch from Left  ${sel_list}  |
217    ${sel_entry}=  Evaluate  $sel_entry.replace(' ','')
218    ${sel_entry}=  Convert To Integer  0x${sel_entry}
219
220    ${sel_delete}=  Run IPMI Standard Command  sel delete ${sel_entry}
221    Should Be Equal As Strings  ${sel_delete}  Deleted entry ${sel_entry}
222    ...  case_insensitive=True
223
224    ${sel_list}=  Run IPMI Standard Command  sel list
225    Should Be Equal As Strings  ${sel_list}  SEL has no entries
226    ...  case_insensitive=True
227
228
229Delete Non Existing SEL Event Entry
230    [Documentation]  Delete non existing SEL event entry.
231    [Tags]  Delete_Non_Existing_SEL_Event_Entry
232
233    ${sel_delete}=  Run Keyword And Expect Error  *
234    ...  Run IPMI Standard Command  sel delete 100
235    Should Contain  ${sel_delete}  Unable to delete entry
236    ...  case_insensitive=True
237
238
239Delete Invalid SEL Event Entry
240    [Documentation]  Delete invalid SEL event entry.
241    [Tags]  Delete_Invalid_SEL_Event_Entry
242
243    ${sel_delete}=  Run Keyword And Expect Error  *
244    ...  Run IPMI Standard Command  sel delete abc
245    Should Contain  ${sel_delete}  Given SEL ID 'abc' is invalid
246    ...  case_insensitive=True
247
248
249Verify IPMI SEL Event Entries
250    [Documentation]  Verify IPMI SEL's entries info.
251    [Tags]  Verify_IPMI_SEL_Event_Entries
252
253    # Generate error logs of random count.
254    ${count}=  Evaluate  random.randint(1, 5)  modules=random
255    Repeat Keyword  ${count}  Create Test Error Log
256
257    ${sel_entries_count}=  Get IPMI SEL Setting  Entries
258    Should Be Equal As Strings  ${sel_entries_count}  ${count}
259
260
261Verify IPMI SEL Event Last Add Time
262    [Documentation]  Verify IPMI SEL's last added timestamp.
263    [Tags]  Verify_IPMI_SEL_Event_Last_Add_Time
264
265    Create Test Error Log
266    ${sel_time}=  Run IPMI Standard Command  sel time get
267    ${sel_time}=  Convert Date  ${sel_time}
268    ...  date_format=%m/%d/%Y %H:%M:%S  exclude_millis=True
269
270    ${sel_last_add_time}=  Get IPMI SEL Setting  Last Add Time
271    ${sel_last_add_time}=  Convert Date  ${sel_last_add_time}
272    ...  date_format=%m/%d/%Y %H:%M:%S  exclude_millis=True
273
274    ${time_diff}=
275    ...  Subtract Date From Date  ${sel_last_add_time}  ${sel_time}
276
277    # Verify if the delay in current time check and last add SEL time
278    # is less or equals to 2 seconds.
279    Should Be True  ${time_diff} <= 2
280
281
282Create Test Event Log And Delete
283    [Documentation]  Create an event log and delete it.
284    [Tags]  Create_Test_Event_Log_And_Delete
285
286    Create Test Error Log
287    Redfish Purge Event Log
288    Event Log Should Not Exist
289
290
291Create Multiple Test Event Logs And Delete All
292    [Documentation]  Create multiple event logs and delete all.
293    [Tags]  Create_Multiple_Test_Event_Logs_And_Delete_All
294
295    Create Test Error Log
296    Create Test Error Log
297    Create Test Error Log
298    Redfish Purge Event Log
299    Event Log Should Not Exist
300
301
302# TODO: openbmc/openbmc-test-automation#1789
303Create Two Test Event Logs And Delete One
304    [Documentation]  Create two event logs and delete the first entry.
305    [Tags]  Create_Two_Test_Eevent_Logs_And_Delete_One
306
307    Redfish Purge Event Log
308    Create Test Error Log
309    ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
310    Create Test Error Log
311    Delete Error log Entry  ${elog_entry[0]}
312    ${resp}=  OpenBMC Get Request  ${elog_entry[0]}
313    Should Be Equal As Strings  ${resp.status_code}  ${HTTP_NOT_FOUND}
314
315
316Verify Watchdog Timedout Event
317    [Documentation]  Trigger watchdog timed out and verify event log generated.
318    [Tags]  Verify_Watchdog_Timedout_Event
319
320    Redfish Power On
321
322    # Clear errors if there are any.
323    Redfish.Login
324    Redfish Purge Event Log
325
326    Trigger Host Watchdog Error
327
328    # Logging takes time to generate the timeout error.
329    Wait Until Keyword Succeeds  2 min  30 sec
330    ...  Verify Watchdog EventLog Content
331
332
333Verify Event Logs Capping
334    [Documentation]  Verify event logs capping.
335    [Tags]  Verify_Event_Logs_Capping
336
337    Redfish Purge Event Log
338
339    ${cmd}=  Catenate  for i in {1..201}; do /tmp/tarball/bin/logging-test -c
340    ...  AutoTestSimple;sleep 1;done
341    BMC Execute Command  ${cmd}
342
343    ${elogs}=  Get Event Logs
344    ${count}=  Get Length  ${elogs}
345    Run Keyword If  ${count} > 200
346    ...  Fail  Error logs created exceeded max capacity 200.
347
348
349Test Event Log Wrapping
350    [Documentation]  Verify event log entries wraps when 200 max cap is reached.
351    [Tags]  Test_Event_Log_Wrapping
352
353    # Restarting logging service in order to clear logs and get the next log
354    # ID set to 1.
355    BMC Execute Command
356    ...  systemctl restart xyz.openbmc_project.Logging.service
357    Sleep  10s  reason=Wait for logging service to restart properly.
358
359    # Create ${max_num_event_logs} event logs.
360    ${cmd}=  Catenate  for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c
361    ...  AutoTestSimple;sleep 1;done
362    BMC Execute Command  ${cmd}
363
364    # Verify that event logs with IDs 1 and ${max_num_event_logs} exist.
365    ${event_log}=  Get Event Logs
366
367    ${log_entries}=  Filter Struct  ${event_log}  [('Id', '1')]
368    Rprint Vars  log_entries
369    Should Be Equal As Strings  ${log_entries[0]["Id"]}  1
370
371    ${log_entries}=  Filter Struct  ${event_log}  [('Id', '${max_num_event_logs}')]
372    Rprint Vars  log_entries
373    Should Be Equal As Strings  ${log_entries[0]["Id"]}  ${max_num_event_logs}
374
375    # Create event log and verify the entry ID, ${max_num_event_logs + 1}.
376    ${next_event_log_id}=  Set Variable  ${max_num_event_logs + 1}
377
378    Create Test Error Log
379
380    ${event_log}=  Get Event Logs
381
382    ${log_entries}=  Filter Struct  ${event_log}  [('Id', '${next_event_log_id}')]
383    Rprint Vars  log_entries
384    Should Be Equal As Strings  ${log_entries[0]["Id"]}  ${next_event_log_id}
385
386    # Event log 1 should be wrapped.
387    ${log_entries}=  Filter Struct  ${event_log}  [('Id', '1')]
388    Rprint Vars  log_entries
389
390    ${length_log_entries}  Get Length  ${log_entries}
391    Should Be Equal As Integers  ${length_log_entries}  0
392    ...  msg=The event log should have wrapped such that entry ID 1 is now purged.
393
394
395*** Keywords ***
396
397Suite Setup Execution
398   [Documentation]  Do test case setup tasks.
399
400    Redfish.Login
401
402    Redfish Purge Event Log
403
404    ${status}=  Run Keyword And Return Status  Logging Test Binary Exist
405    Run Keyword If  ${status} == ${False}  Install Tarball
406
407
408Suite Teardown Execution
409    [Documentation]  Do the post suite teardown.
410
411    Redfish.Logout
412
413
414Test Setup Execution
415   [Documentation]  Do test case setup tasks.
416
417    Redfish Purge Event Log
418
419    ${status}=  Run Keyword And Return Status  Logging Test Binary Exist
420    Run Keyword If  ${status} == ${False}  Install Tarball
421
422
423Test Teardown Execution
424    [Documentation]  Do the post test teardown.
425
426    FFDC On Test Case Fail
427    Redfish.Login
428    Redfish Purge Event Log
429
430
431Event Log Should Not Exist
432    [Documentation]  Event log entries should not exist.
433
434    ${elogs}=  Get Event Logs
435    Should Be Empty  ${elogs}  msg=System event log entry is not empty.
436
437
438Event Log Should Exist
439    [Documentation]  Event log entries should exist.
440
441    ${elogs}=  Get Event Logs
442    Should Not Be Empty  ${elogs}  msg=System event log entry is not empty.
443
444
445Verify Watchdog EventLog Content
446    [Documentation]  Verify watchdog event log content.
447
448    # Example:
449    # {
450    #    "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
451    #    "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31",
452    #    "@odata.type": "#LogEntry.v1_4_0.LogEntry",
453    #    "Created": "2019-05-31T18:41:33+00:00",
454    #    "EntryType": "Event",
455    #    "Id": "31",
456    #    "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
457    #    "Name": "System DBus Event Log Entry",
458    #    "Severity": "Critical"
459    # }
460
461    ${elog}=  Get Event Logs
462    Should Be Equal As Strings
463    ...  ${elog[0]["Message"]}  org.open_power.Host.Boot.Error.WatchdogTimedOut
464    ...  msg=Watchdog timeout event log was not found.
465    Should Be Equal As Strings
466    ...  ${elog[0]["Severity"]}  Critical
467    ...  msg=Watchdog timeout severity unexpected value.
468