1*** Settings *** 2 3Documentation Test BMC dump functionality of OpenBMC. 4 5Resource ../../lib/bmc_redfish_resource.robot 6Resource ../../lib/boot_utils.robot 7Resource ../../lib/dump_utils.robot 8Resource ../../lib/openbmc_ffdc.robot 9Variables ../../data/pel_variables.py 10 11Suite Setup Redfish.Login 12Test Setup Redfish Delete All BMC Dumps 13Test Teardown Test Teardown Execution 14 15*** Variables *** 16 17# Total size of the dump in kilo bytes 18${BMC_DUMP_TOTAL_SIZE} ${1024} 19 20# Minimum space required for one bmc dump in kilo bytes 21${BMC_DUMP_MIN_SPACE_REQD} ${20} 22${MAX_DUMP_COUNT} ${20} 23${BMC_DUMP_COLLECTOR_PATH} /var/lib/phosphor-debug-collector/dumps 24 25*** Test Cases *** 26 27Verify Error Response For Already Deleted Dump Id 28 [Documentation] Delete non existing BMC dump and expect an error. 29 [Tags] Verify_Error_Response_For_Already_Deleted_Dump_Id 30 31 Redfish Power Off stack_mode=skip 32 ${dump_id}= Create User Initiated BMC Dump Via Redfish 33 Redfish Delete BMC Dump ${dump_id} 34 Run Keyword And Expect Error ValueError: * Redfish Delete BMC Dump ${dump_id} 35 36 37Verify User Initiated BMC Dump When Host Powered Off 38 [Documentation] Create user initiated BMC dump at host off state and 39 ... verify dump entry for it. 40 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Powered_Off 41 42 Redfish Power Off stack_mode=skip 43 ${dump_id}= Create User Initiated BMC Dump Via Redfish 44 ${dump_entries}= Get BMC Dump Entries 45 Length Should Be ${dump_entries} 1 46 List Should Contain Value ${dump_entries} ${dump_id} 47 48 49Verify User Initiated BMC Dump Size 50 [Documentation] Verify user initiated BMC dump size is under 20 MB. 51 [Tags] Verify_User_Initiated_BMC_Dump_Size 52 53 ${dump_id}= Create User Initiated BMC Dump Via Redfish 54 ${resp}= Redfish.Get Properties /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_id} 55 56 # Example of response from above Redfish GET request. 57 # "@odata.type": "#LogEntry.v1_7_0.LogEntry", 58 # "AdditionalDataSizeBytes": 31644, 59 # "AdditionalDataURI": "/redfish/v1/Managers/bmc/LogServices/Dump/attachment/9", 60 # "Created": "2020-10-23T06:32:53+00:00", 61 # "DiagnosticDataType": "Manager", 62 # "EntryType": "Event", 63 # "Id": "9", 64 # "Name": "BMC Dump Entry" 65 66 # Max size for dump is 20 MB = 20x1024x1024 Byte. 67 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 68 69 70Verify Internal Failure Initiated BMC Dump Size 71 [Documentation] Verify that the internal failure initiated BMC dump size is under 20 MB. 72 [Tags] Verify_Internal_Failure_Initiated_BMC_Dump_Size 73 74 Redfish Delete All BMC Dumps 75 76 # Create an internal failure error log. 77 BMC Execute Command ${CMD_INTERNAL_FAILURE} 78 79 # Wait for BMC dump to get generated after injecting internal failure. 80 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 81 82 # Verify that only one BMC dump is generated after injecting error. 83 ${dump_entries}= Get BMC Dump Entries 84 ${length}= Get length ${dump_entries} 85 Should Be Equal As Integers ${length} ${1} 86 87 # Max size for dump is 20 MB = 20x1024x1024 Byte. 88 ${resp}= Redfish.Get Properties 89 ... /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_entries[0]} 90 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 91 92 93Verify Multiple BMC Dump Creation 94 [Documentation] Verify that multiple BMC dumps can be created one after 95 ... another successfully. 96 [Tags] Verify_Multiple_BMC_Dump_Creation 97 98 ${dump_count}= Evaluate random.randint(5, 10) modules=random 99 FOR ${INDEX} IN 1 ${dump_count} 100 Create User Initiated BMC Dump Via Redfish 101 END 102 103 104Verify BMC Dump Default Location In BMC 105 [Documentation] Verify that BMC dump is created in its default location of BMC. 106 [Tags] Verify_BMC_Dump_Default_Location_In_BMC 107 108 Redfish Delete All BMC Dumps 109 ${dump_id}= Create User Initiated BMC Dump Via Redfish 110 ${dump_file} ${stderr} ${rc}= BMC Execute Command 111 ... ls ${BMC_DUMP_COLLECTOR_PATH}/${dump_id} 112 Should Be True ${rc} == 0 113 Should Contain Any ${dump_file} BMCDUMP obmcdump 114 115 116Verify User Initiated BMC Dump When Host Booted 117 [Documentation] Create user initiated BMC dump at host booted state and 118 ... verify dump entry for it. 119 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Booted 120 121 Redfish Power On stack_mode=skip 122 ${dump_id}= Create User Initiated BMC Dump Via Redfish 123 ${dump_entries}= Get BMC Dump Entries 124 Length Should Be ${dump_entries} 1 125 List Should Contain Value ${dump_entries} ${dump_id} 126 127 128Verify User Initiated BMC Dump At Host Booting 129 [Documentation] Create and verify user initiated BMC dump during Host is powwering on 130 ... or when host booting is in progress. 131 [Tags] Verify_User_Initiated_BMC_Dump_At_Host_Booting 132 133 Redfish Power Off stack_mode=skip 134 Redfish Delete All BMC Dumps 135 136 # Initiate power on. 137 Redfish Power Operation On 138 Wait Until Keyword Succeeds 2 min 5 sec Is Boot Progress Changed 139 140 # Create user initiated BMC dump and verify only one dump is available. 141 Create User Initiated BMC Dump Via Redfish 142 ${dump_entries}= Get BMC Dump Entries 143 Length Should Be ${dump_entries} 1 144 145 146Verify Dump Persistency On Dump Service Restart 147 [Documentation] Create user dump, restart dump manager service and verify dump 148 ... persistency. 149 [Tags] Verify_Dump_Persistency_On_Dump_Service_Restart 150 151 Create User Initiated BMC Dump Via Redfish 152 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries 153 154 # Restart dump service. 155 BMC Execute Command systemctl restart xyz.openbmc_project.Dump.Manager.service 156 Sleep 10s reason=Wait for BMC dump service to restart properly 157 158 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries 159 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} 160 161 162Verify Dump Persistency On BMC Reset 163 [Documentation] Create user dump, reset BMC and verify dump persistency. 164 [Tags] Verify_Dump_Persistency_On_BMC_Reset 165 166 # Power off host so that dump is not offloaded to host OS. 167 Redfish Power Off stack_mode=skip 168 169 Create User Initiated BMC Dump Via Redfish 170 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries 171 172 # Reset BMC. 173 OBMC Reboot (off) stack_mode=skip 174 175 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries 176 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} 177 178 179Delete User Initiated BMC Dump And Verify 180 [Documentation] Delete user initiated BMC dump and verify. 181 [Tags] Delete_User_Initiated_BMC_Dump_And_Verify 182 183 ${dump_id}= Create User Initiated BMC Dump Via Redfish 184 Redfish Delete BMC Dump ${dump_id} 185 186 ${dump_entries}= Get BMC Dump Entries 187 Should Be Empty ${dump_entries} 188 189 190Delete All User Initiated BMC Dumps And Verify 191 [Documentation] Delete all user initiated BMC dumps and verify. 192 [Tags] Delete_All_User_Initiated_BMC_Dumps_And_Verify 193 194 # Power off host so that dump is not offloaded to host OS. 195 Redfish Power Off stack_mode=skip 196 197 # Create some BMC dump. 198 Create User Initiated BMC Dump Via Redfish 199 Create User Initiated BMC Dump Via Redfish 200 201 Redfish Delete All BMC Dumps 202 ${dump_entries}= Get BMC Dump Entries 203 Should Be Empty ${dump_entries} 204 205 206Create Two User Initiated BMC Dumps 207 [Documentation] Create two user initiated BMC dumps. 208 [Tags] Create_Two_User_Initiated_BMC_Dumps 209 210 ${dump_id1}= Create User Initiated BMC Dump Via Redfish 211 ${dump_id2}= Create User Initiated BMC Dump Via Redfish 212 213 ${dump_entries}= Get BMC Dump Entries 214 Length Should Be ${dump_entries} 2 215 Should Contain ${dump_entries} ${dump_id1} 216 Should Contain ${dump_entries} ${dump_id2} 217 218 219Create Two User Initiated BMC Dumps And Delete One 220 [Documentation] Create two dumps and delete the first. 221 [Tags] Create_Two_User_Initiated_BMC_Dumps_And_Delete_One 222 223 ${dump_id1}= Create User Initiated BMC Dump Via Redfish 224 ${dump_id2}= Create User Initiated BMC Dump Via Redfish 225 226 Redfish Delete BMC Dump ${dump_id1} 227 228 ${dump_entries}= Get BMC Dump Entries 229 Length Should Be ${dump_entries} 1 230 List Should Contain Value ${dump_entries} ${dump_id2} 231 232 233Create And Delete User Initiated BMC Dump Multiple Times 234 [Documentation] Create and delete user initiated BMC dump multiple times. 235 [Tags] Create_And_Delete_User_Initiated_BMC_Dump_Multiple_Times 236 237 FOR ${INDEX} IN 1 10 238 ${dump_id}= Create User Initiated BMC Dump Via Redfish 239 Redfish Delete BMC Dump ${dump_id} 240 END 241 242 243Verify Maximum BMC Dump Creation 244 [Documentation] Create maximum BMC dump and verify error when dump runs out of space. 245 [Tags] Verify_Maximum_BMC_Dump_Creation 246 [Teardown] Redfish Delete All BMC Dumps 247 248 # Maximum allowed space for dump is 1024 KB. BMC typically hold 8-14 dumps 249 # before running out of this dump space. So trying to create dumps in 20 250 # iterations to run out of space. 251 # User can key in the Maximum allowed space for bmc dump and how many iteration. 252 FOR ${n} IN RANGE 0 ${MAX_DUMP_COUNT} 253 Create User Initiated BMC Dump Via Redfish 254 ${dump_space}= Get Disk Usage For Dumps 255 Exit For Loop If ${dump_space} >= (${BMC_DUMP_TOTAL_SIZE} - ${BMC_DUMP_MIN_SPACE_REQD}) 256 END 257 258 # Check error while creating dump when dump size is full. 259 ${payload}= Create Dictionary DiagnosticDataType=Manager 260 Redfish.Post /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData 261 ... body=${payload} valid_status_codes=[${HTTP_INTERNAL_SERVER_ERROR}] 262 263 264Verify BMC Core Dump When Host Powered Off 265 [Documentation] Verify BMC core dump after application crash at host powered off state. 266 [Tags] Verify_BMC_Core_Dump_When_Host_Powered_Off 267 268 Redfish Power Off stack_mode=skip 269 270 # Ensure all dumps are cleaned out. 271 Redfish Delete All BMC Dumps 272 Trigger Core Dump 273 274 # Verify that BMC dump is available. 275 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 276 277 278Verify Core Dump Size 279 [Documentation] Verify BMC core dump size is under 20 MB. 280 [Tags] Verify_Core_Dump_Size 281 282 Redfish Power Off stack_mode=skip 283 284 # Ensure all dumps are cleaned out. 285 Redfish Delete All BMC Dumps 286 Trigger Core Dump 287 288 # Verify that BMC dump is available. 289 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 290 ${dump_entries}= Get BMC Dump Entries 291 ${resp}= Redfish.Get Properties 292 ... /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_entries[0]} 293 294 # Max size for dump is 20 MB = 20x1024x1024 Byte. 295 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 296 297 298Verify Error While Initiating BMC Dump During Dumping State 299 [Documentation] Verify error while initiating BMC dump during dumping state. 300 [Tags] Verify_Error_While_Initiating_BMC_Dump_During_Dumping_State 301 302 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} 303 304 # Check error while initiating BMC dump while dump in progress. 305 ${payload}= Create Dictionary DiagnosticDataType=Manager 306 Redfish.Post 307 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData 308 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}] 309 310 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. 311 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${task_id} 312 313 314Verify BMC Dump Create Errors While Another BMC Dump In Progress 315 [Documentation] Verify BMC dump creation error until older BMC dump completion. 316 [Tags] Verify_BMC_Dump_Create_Errors_While_Another_BMC_Dump_In_Progress 317 318 # Initiate a BMC dump that returns without completion. 319 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} 320 321 # Now continue to initiate multiple dump request which is not expected to be accepted 322 # till earlier BMC dump task is completed. A limit is set to avoid risk of infinite loop. 323 ${payload}= Create Dictionary DiagnosticDataType=Manager 324 WHILE True limit=1000 325 ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id} 326 IF '${task_dict['TaskState']}' == 'Completed' BREAK 327 ${resp}= Redfish.Post 328 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData 329 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}, ${HTTP_ACCEPTED}] 330 331 # Sample response of above POST request: 332 # 503 333 # Strict-Transport-Security max-age=31536000; includeSubdomains; preload 334 # X-Frame-Options DENY 335 # Pragma no-cache 336 # Cache-Control no-Store,no-Cache 337 # X-XSS-Protection 1; mode=block 338 # X-Content-Type-Options nosniff 339 # Content-Security-Policy default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 340 # 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; 341 # object-src 'none'; base-uri 'none' 342 # Content-Type application/json 343 # Date Thu, 16 Mar 2023 06:41:06 GMT 344 # Content-Length 573 345 # { 346 # "error": { 347 # "@Message.ExtendedInfo": [ 348 # { 349 # "@odata.type": "#Message.v1_1_1.Message", 350 # "Message": "The request could not be performed because the resource is in standby.", 351 # "MessageArgs": [], 352 # "MessageId": "Base.1.8.1.ResourceInStandby", 353 # "MessageSeverity": "Critical", 354 # "Resolution": "Ensure that the resource is in the correct power state and resubmit the request." 355 # } 356 # ], 357 # "code": "Base.1.8.1.ResourceInStandby", 358 # "message": "The request could not be performed because the resource is in standby." 359 # } 360 # } 361 362 # At this point the request HTTPS response should be HTTP_SERVICE_UNAVAILABLE. However, if the 363 # response contains the HTTPS response code HTTP_ACCEPTED, it means there is a dump initiation 364 # request which is completed. We verify this with below check. 365 ${resp}= Convert To String ${resp} 366 ${contains}= Run Keyword And Return Status Should Contain ${resp.split()[0]} ${HTTP_ACCEPTED} 367 IF ${contains} 368 ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id} 369 Should Be True '${task_dict['TaskState']}' == 'Completed' 370 END 371 END 372 373 # The next BMC dump initiation request should be accepted as earlier dump is completed. 374 ${resp}= Redfish.Post 375 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData 376 ... body=${payload} valid_status_codes=[${HTTP_ACCEPTED}] 377 378 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. 379 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${resp.dict['Id']} 380 381 382Verify Core Dump After Terminating Dump Manager Service 383 [Documentation] Verify initiate core dumps and kill Phosphor-dump-manager. 384 [Tags] Verify_Core_Dump_After_Terminating_Dump_Manager_Service 385 386 Redfish Power Off stack_mode=skip 387 388 # Remove all available dumps in BMC. 389 Redfish Delete All BMC Dumps 390 391 # Find the pid of the phosphor-dump-manage process and kill it. 392 ${cmd_buf}= Catenate kill -s SEGV $(pgrep phosphor-dump-manager) 393 ${cmd_output} ${stderr} ${rc}= BMC Execute Command ${cmd_buf} 394 Should Be Equal As Integers ${rc} ${0} 395 396 # Verify that BMC dump is available. 397 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 398 399 # Verifying that there is only one dump. 400 ${dump_entries}= Get BMC Dump Entries 401 ${length}= Get length ${dump_entries} 402 Should Be Equal As Integers ${length} ${1} 403 404 405Verify Error Log And Dump For Internal Failure 406 [Documentation] Verify error log and dump for internal failure. 407 [Tags] Verify_Error_Log_And_Dump_For_Internal_Failure 408 409 Redfish Purge Event Log 410 Redfish Delete All BMC Dumps 411 412 # Create an internal failure error log. 413 BMC Execute Command ${CMD_INTERNAL_FAILURE} 414 415 # With internal failure, an error log file is generated. Check if 416 # BMC has only one error log for this internal failure. 417 ${resp}= Redfish.Get /redfish/v1/Systems/system/LogServices/CELog/Entries 418 Should Be True ${resp.dict["Members@odata.count"]} == ${1} 419 420 # Wait for the BMC dump to become available and verify its presence. 421 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 422 ${dump_entries}= Get BMC Dump Entries 423 ${length}= Get length ${dump_entries} 424 Should Be Equal As Integers ${length} ${1} 425 426 427Verify Core Watchdog Initiated BMC Dump 428 [Documentation] Verify core watchdog timeout initiated BMC dump. 429 [Tags] Verify_Core_Watchdog_Initiated_BMC_Dump 430 431 Redfish Delete All BMC Dumps 432 Redfish Power Off stack_mode=skip 433 434 # Trigger watchdog timeout. 435 Redfish Initiate Auto Reboot 2000 436 437 # Wait for BMC dump to get generated after injecting watchdog timeout. 438 Wait Until Keyword Succeeds 4 min 20 sec Is BMC Dump Available 439 440 # Verify that only one BMC dump is available. 441 ${dump_entry_list}= Get BMC Dump Entries 442 ${length}= Get length ${dump_entry_list} 443 Should Be Equal As Integers ${length} ${1} 444 445 446*** Keywords *** 447 448Get BMC Dump Entries 449 [Documentation] Return BMC dump ids list. 450 451 ${dump_uris}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries 452 ${dump_ids}= Create List 453 454 FOR ${dump_uri} IN @{dump_uris} 455 ${dump_id}= Fetch From Right ${dump_uri} / 456 Append To List ${dump_ids} ${dump_id} 457 END 458 459 [Return] ${dump_ids} 460 461 462Is BMC Dump Available 463 [Documentation] Verify if BMC dump is available. 464 465 ${dump_entries}= Get BMC Dump Entries 466 467 # Verifying that BMC dump is available. 468 ${length}= Get length ${dump_entries} 469 Should Be True 0 < ${length} 470 471 472Get Disk Usage For Dumps 473 [Documentation] Return disk usage in kilobyte for BMC dumps. 474 475 ${usage_output} ${stderr} ${rc}= BMC Execute Command du -s ${BMC_DUMP_COLLECTOR_PATH} 476 477 # Example of output from above BMC cli command. 478 # $ du -s /var/lib/phosphor-debug-collector/dumps 479 # 516 /var/lib/phosphor-debug-collector/dumps 480 481 ${usage_output}= Fetch From Left ${usage_output} / 482 ${usage_output}= Convert To Integer ${usage_output} 483 484 [return] ${usage_output} 485 486 487Test Teardown Execution 488 [Documentation] Do test teardown operation. 489 490 FFDC On Test Case Fail 491 Close All Connections 492