1*** Settings *** 2 3Documentation Test BMC dump functionality of OpenBMC. 4 5Resource ../../lib/bmc_redfish_resource.robot 6Resource ../../lib/boot_utils.robot 7Resource ../../lib/dump_utils.robot 8Resource ../../lib/openbmc_ffdc.robot 9Variables ../../data/pel_variables.py 10 11Suite Setup Redfish.Login 12Test Setup Redfish Delete All BMC Dumps 13Test Teardown Test Teardown Execution 14 15*** Variables *** 16 17# Total size of the dump in kilo bytes 18${BMC_DUMP_TOTAL_SIZE} ${1024} 19 20# Minimum space required for one bmc dump in kilo bytes 21${BMC_DUMP_MIN_SPACE_REQD} ${20} 22${MAX_DUMP_COUNT} ${20} 23${BMC_DUMP_COLLECTOR_PATH} /var/lib/phosphor-debug-collector/dumps 24 25*** Test Cases *** 26 27Verify Error Response For Already Deleted Dump Id 28 [Documentation] Delete non existing BMC dump and expect an error. 29 [Tags] Verify_Error_Response_For_Already_Deleted_Dump_Id 30 31 Redfish Power Off stack_mode=skip 32 ${dump_id}= Create User Initiated BMC Dump Via Redfish 33 Redfish Delete BMC Dump ${dump_id} 34 Run Keyword And Expect Error ValueError: * Redfish Delete BMC Dump ${dump_id} 35 36 37Verify User Initiated BMC Dump When Host Powered Off 38 [Documentation] Create user initiated BMC dump at host off state and 39 ... verify dump entry for it. 40 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Powered_Off 41 42 Redfish Power Off stack_mode=skip 43 ${dump_id}= Create User Initiated BMC Dump Via Redfish 44 ${dump_entries}= Get BMC Dump Entries 45 Length Should Be ${dump_entries} 1 46 List Should Contain Value ${dump_entries} ${dump_id} 47 48 49Verify User Initiated BMC Dump Size 50 [Documentation] Verify user initiated BMC dump size is under 20 MB. 51 [Tags] Verify_User_Initiated_BMC_Dump_Size 52 53 Redfish Power Off stack_mode=skip 54 ${dump_id}= Create User Initiated BMC Dump Via Redfish 55 ${resp}= Redfish.Get Properties /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_id} 56 57 # Example of response from above Redfish GET request. 58 # "@odata.type": "#LogEntry.v1_7_0.LogEntry", 59 # "AdditionalDataSizeBytes": 31644, 60 # "AdditionalDataURI": "/redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/attachment/9", 61 # "Created": "2020-10-23T06:32:53+00:00", 62 # "DiagnosticDataType": "Manager", 63 # "EntryType": "Event", 64 # "Id": "9", 65 # "Name": "BMC Dump Entry" 66 67 # Max size for dump is 20 MB = 20x1024x1024 Byte. 68 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 69 70 71Verify Internal Failure Initiated BMC Dump Size 72 [Documentation] Verify that the internal failure initiated BMC dump size is under 20 MB. 73 [Tags] Verify_Internal_Failure_Initiated_BMC_Dump_Size 74 75 Redfish Delete All BMC Dumps 76 77 # Create an internal failure error log. 78 BMC Execute Command ${CMD_INTERNAL_FAILURE} 79 80 # Wait for BMC dump to get generated after injecting internal failure. 81 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 82 83 # Verify that only one BMC dump is generated after injecting error. 84 ${dump_entries}= Get BMC Dump Entries 85 ${length}= Get length ${dump_entries} 86 Should Be Equal As Integers ${length} ${1} 87 88 # Max size for dump is 20 MB = 20x1024x1024 Byte. 89 ${resp}= Redfish.Get Properties 90 ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_entries[0]} 91 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 92 93 94Verify Multiple BMC Dump Creation 95 [Documentation] Verify that multiple BMC dumps can be created one after 96 ... another successfully. 97 [Tags] Verify_Multiple_BMC_Dump_Creation 98 99 Redfish Power Off stack_mode=skip 100 ${dump_count}= Evaluate random.randint(5, 10) modules=random 101 FOR ${INDEX} IN 1 ${dump_count} 102 Create User Initiated BMC Dump Via Redfish 103 END 104 105 106Verify BMC Dump Default Location In BMC 107 [Documentation] Verify that BMC dump is created in its default location of BMC. 108 [Tags] Verify_BMC_Dump_Default_Location_In_BMC 109 110 Redfish Power Off stack_mode=skip 111 Redfish Delete All BMC Dumps 112 ${dump_id}= Create User Initiated BMC Dump Via Redfish 113 ${dump_file} ${stderr} ${rc}= BMC Execute Command 114 ... ls ${BMC_DUMP_COLLECTOR_PATH}/${dump_id} 115 Should Be True ${rc} == 0 116 Should Contain Any ${dump_file} BMCDUMP obmcdump 117 118 119Verify User Initiated BMC Dump When Host Booted 120 [Documentation] Create user initiated BMC dump at host booted state and 121 ... verify dump entry for it. 122 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Booted 123 124 Redfish Power On stack_mode=skip 125 ${dump_id}= Create User Initiated BMC Dump Via Redfish 126 ${dump_entries}= Get BMC Dump Entries 127 Length Should Be ${dump_entries} 1 128 List Should Contain Value ${dump_entries} ${dump_id} 129 130 131Verify User Initiated BMC Dump At Host Booting 132 [Documentation] Create and verify user initiated BMC dump during Host is powwering on 133 ... or when host booting is in progress. 134 [Tags] Verify_User_Initiated_BMC_Dump_At_Host_Booting 135 136 Redfish Power Off stack_mode=skip 137 Redfish Delete All BMC Dumps 138 139 # Initiate power on. 140 Redfish Power Operation On 141 Wait Until Keyword Succeeds 2 min 5 sec Is Boot Progress Changed 142 143 # Create user initiated BMC dump and verify only one dump is available. 144 Create User Initiated BMC Dump Via Redfish 145 ${dump_entries}= Get BMC Dump Entries 146 Length Should Be ${dump_entries} 1 147 148 149Verify Dump Persistency On Dump Service Restart 150 [Documentation] Create user dump, restart dump manager service and verify dump 151 ... persistency. 152 [Tags] Verify_Dump_Persistency_On_Dump_Service_Restart 153 154 Redfish Power Off stack_mode=skip 155 Create User Initiated BMC Dump Via Redfish 156 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries 157 158 # Restart dump service. 159 BMC Execute Command systemctl restart xyz.openbmc_project.Dump.Manager.service 160 Sleep 10s reason=Wait for BMC dump service to restart properly 161 162 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries 163 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} 164 165 166Verify Dump Persistency On BMC Reset 167 [Documentation] Create user dump, reset BMC and verify dump persistency. 168 [Tags] Verify_Dump_Persistency_On_BMC_Reset 169 170 # Power off host so that dump is not offloaded to host OS. 171 Redfish Power Off stack_mode=skip 172 173 Create User Initiated BMC Dump Via Redfish 174 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries 175 176 # Reset BMC. 177 OBMC Reboot (off) stack_mode=skip 178 179 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries 180 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} 181 182 183Delete User Initiated BMC Dump And Verify 184 [Documentation] Delete user initiated BMC dump and verify. 185 [Tags] Delete_User_Initiated_BMC_Dump_And_Verify 186 187 Redfish Power Off stack_mode=skip 188 ${dump_id}= Create User Initiated BMC Dump Via Redfish 189 Redfish Delete BMC Dump ${dump_id} 190 191 ${dump_entries}= Get BMC Dump Entries 192 Should Be Empty ${dump_entries} 193 194 195Delete All User Initiated BMC Dumps And Verify 196 [Documentation] Delete all user initiated BMC dumps and verify. 197 [Tags] Delete_All_User_Initiated_BMC_Dumps_And_Verify 198 199 # Power off host so that dump is not offloaded to host OS. 200 Redfish Power Off stack_mode=skip 201 202 # Create some BMC dump. 203 Create User Initiated BMC Dump Via Redfish 204 Create User Initiated BMC Dump Via Redfish 205 206 Redfish Delete All BMC Dumps 207 ${dump_entries}= Get BMC Dump Entries 208 Should Be Empty ${dump_entries} 209 210 211Create Two User Initiated BMC Dumps 212 [Documentation] Create two user initiated BMC dumps. 213 [Tags] Create_Two_User_Initiated_BMC_Dumps 214 215 Redfish Power Off stack_mode=skip 216 ${dump_id1}= Create User Initiated BMC Dump Via Redfish 217 ${dump_id2}= Create User Initiated BMC Dump Via Redfish 218 219 ${dump_entries}= Get BMC Dump Entries 220 Length Should Be ${dump_entries} 2 221 Should Contain ${dump_entries} ${dump_id1} 222 Should Contain ${dump_entries} ${dump_id2} 223 224 225Create Two User Initiated BMC Dumps And Delete One 226 [Documentation] Create two dumps and delete the first. 227 [Tags] Create_Two_User_Initiated_BMC_Dumps_And_Delete_One 228 229 Redfish Power Off stack_mode=skip 230 ${dump_id1}= Create User Initiated BMC Dump Via Redfish 231 ${dump_id2}= Create User Initiated BMC Dump Via Redfish 232 233 Redfish Delete BMC Dump ${dump_id1} 234 235 ${dump_entries}= Get BMC Dump Entries 236 Length Should Be ${dump_entries} 1 237 List Should Contain Value ${dump_entries} ${dump_id2} 238 239 240Create And Delete User Initiated BMC Dump Multiple Times 241 [Documentation] Create and delete user initiated BMC dump multiple times. 242 [Tags] Create_And_Delete_User_Initiated_BMC_Dump_Multiple_Times 243 244 Redfish Power Off stack_mode=skip 245 FOR ${INDEX} IN 1 10 246 ${dump_id}= Create User Initiated BMC Dump Via Redfish 247 Redfish Delete BMC Dump ${dump_id} 248 END 249 250 251Verify Maximum BMC Dump Creation 252 [Documentation] Create maximum BMC dump and verify error when dump runs out of space. 253 [Tags] Verify_Maximum_BMC_Dump_Creation 254 [Teardown] Redfish Delete All BMC Dumps 255 256 # Maximum allowed space for dump is 1024 KB. BMC typically hold 8-14 dumps 257 # before running out of this dump space. So trying to create dumps in 20 258 # iterations to run out of space. 259 # User can key in the Maximum allowed space for bmc dump and how many iteration. 260 FOR ${n} IN RANGE 0 ${MAX_DUMP_COUNT} 261 Create User Initiated BMC Dump Via Redfish 262 ${dump_space}= Get Disk Usage For Dumps 263 Exit For Loop If ${dump_space} >= (${BMC_DUMP_TOTAL_SIZE} - ${BMC_DUMP_MIN_SPACE_REQD}) 264 END 265 266 # Check error while creating dump when dump size is full. 267 ${payload}= Create Dictionary DiagnosticDataType=Manager 268 Redfish.Post /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData 269 ... body=${payload} valid_status_codes=[${HTTP_INTERNAL_SERVER_ERROR}] 270 271 272Verify BMC Core Dump When Host Powered Off 273 [Documentation] Verify BMC core dump after application crash at host powered off state. 274 [Tags] Verify_BMC_Core_Dump_When_Host_Powered_Off 275 276 Redfish Power Off stack_mode=skip 277 278 # Ensure all dumps are cleaned out. 279 Redfish Delete All BMC Dumps 280 Trigger Core Dump 281 282 # Verify that BMC dump is available. 283 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 284 285 286Verify Core Dump Size 287 [Documentation] Verify BMC core dump size is under 20 MB. 288 [Tags] Verify_Core_Dump_Size 289 290 Redfish Power Off stack_mode=skip 291 292 # Ensure all dumps are cleaned out. 293 Redfish Delete All BMC Dumps 294 Trigger Core Dump 295 296 # Verify that BMC dump is available. 297 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 298 ${dump_entries}= Get BMC Dump Entries 299 ${resp}= Redfish.Get Properties 300 ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_entries[0]} 301 302 # Max size for dump is 20 MB = 20x1024x1024 Byte. 303 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 304 305 306Verify Error While Initiating BMC Dump During Dumping State 307 [Documentation] Verify error while initiating BMC dump during dumping state. 308 [Tags] Verify_Error_While_Initiating_BMC_Dump_During_Dumping_State 309 310 Redfish Power Off stack_mode=skip 311 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} 312 313 # Check error while initiating BMC dump while dump in progress. 314 ${payload}= Create Dictionary DiagnosticDataType=Manager 315 Redfish.Post 316 ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData 317 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}] 318 319 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. 320 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${task_id} 321 322 323Verify BMC Dump Create Errors While Another BMC Dump In Progress 324 [Documentation] Verify BMC dump creation error until older BMC dump completion. 325 [Tags] Verify_BMC_Dump_Create_Errors_While_Another_BMC_Dump_In_Progress 326 327 Redfish Power Off stack_mode=skip 328 329 # Initiate a BMC dump that returns without completion. 330 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} 331 332 # Now continue to initiate multiple dump request which is not expected to be accepted 333 # till earlier BMC dump task is completed. A limit is set to avoid risk of infinite loop. 334 ${payload}= Create Dictionary DiagnosticDataType=Manager 335 WHILE True limit=1000 336 ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id} 337 IF '${task_dict['TaskState']}' == 'Completed' BREAK 338 ${resp}= Redfish.Post 339 ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData 340 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}, ${HTTP_ACCEPTED}] 341 342 # Sample response of above POST request: 343 # 503 344 # Strict-Transport-Security max-age=31536000; includeSubdomains; preload 345 # X-Frame-Options DENY 346 # Pragma no-cache 347 # Cache-Control no-Store,no-Cache 348 # X-XSS-Protection 1; mode=block 349 # X-Content-Type-Options nosniff 350 # Content-Security-Policy default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 351 # 'self'; script-src 'self'; connect-src 'self' wss:; form-action 'none'; frame-ancestors 'none'; 352 # object-src 'none'; base-uri 'none' 353 # Content-Type application/json 354 # Date Thu, 16 Mar 2023 06:41:06 GMT 355 # Content-Length 573 356 # { 357 # "error": { 358 # "@Message.ExtendedInfo": [ 359 # { 360 # "@odata.type": "#Message.v1_1_1.Message", 361 # "Message": "The request could not be performed because the resource is in standby.", 362 # "MessageArgs": [], 363 # "MessageId": "Base.1.8.1.ResourceInStandby", 364 # "MessageSeverity": "Critical", 365 # "Resolution": "Ensure that the resource is in the correct power state and resubmit the request." 366 # } 367 # ], 368 # "code": "Base.1.8.1.ResourceInStandby", 369 # "message": "The request could not be performed because the resource is in standby." 370 # } 371 # } 372 373 # At this point the request HTTPS response should be HTTP_SERVICE_UNAVAILABLE. However, if the 374 # response contains the HTTPS response code HTTP_ACCEPTED, it means there is a dump initiation 375 # request which is completed. We verify this with below check. 376 ${resp}= Convert To String ${resp} 377 ${contains}= Run Keyword And Return Status Should Contain ${resp.split()[0]} ${HTTP_ACCEPTED} 378 IF ${contains} 379 ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id} 380 Should Be True '${task_dict['TaskState']}' == 'Completed' 381 END 382 END 383 384 # The next BMC dump initiation request should be accepted as earlier dump is completed. 385 # Wait for few seconds before initiating the dump. 386 Sleep 2s 387 ${resp}= Redfish.Post 388 ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData 389 ... body=${payload} valid_status_codes=[${HTTP_ACCEPTED}] 390 391 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. 392 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${resp.dict['Id']} 393 394 395Verify Core Dump After Terminating Dump Manager Service 396 [Documentation] Verify initiate core dumps and kill Phosphor-dump-manager. 397 [Tags] Verify_Core_Dump_After_Terminating_Dump_Manager_Service 398 399 Redfish Power Off stack_mode=skip 400 401 # Remove all available dumps in BMC. 402 Redfish Delete All BMC Dumps 403 404 # Find the pid of the phosphor-dump-manage process and kill it. 405 ${cmd_buf}= Catenate kill -s SEGV $(pgrep phosphor-dump-manager) 406 ${cmd_output} ${stderr} ${rc}= BMC Execute Command ${cmd_buf} 407 Should Be Equal As Integers ${rc} ${0} 408 409 # Verify that BMC dump is available. 410 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 411 412 # Verifying that there is only one dump. 413 ${dump_entries}= Get BMC Dump Entries 414 ${length}= Get length ${dump_entries} 415 Should Be Equal As Integers ${length} ${1} 416 417 418Verify Error Log And Dump For Internal Failure 419 [Documentation] Verify error log and dump for internal failure. 420 [Tags] Verify_Error_Log_And_Dump_For_Internal_Failure 421 422 Redfish Purge Event Log 423 Redfish Delete All BMC Dumps 424 425 # Create an internal failure error log. 426 BMC Execute Command ${CMD_INTERNAL_FAILURE} 427 428 # With internal failure, an error log file is generated. Check if 429 # BMC has only one error log for this internal failure. 430 ${resp}= Redfish.Get /redfish/v1/Systems/system/LogServices/CELog/Entries 431 Should Be True ${resp.dict["Members@odata.count"]} == ${1} 432 433 # Wait for the BMC dump to become available and verify its presence. 434 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available 435 ${dump_entries}= Get BMC Dump Entries 436 ${length}= Get length ${dump_entries} 437 Should Be Equal As Integers ${length} ${1} 438 439 440Verify Core Watchdog Initiated BMC Dump 441 [Documentation] Verify core watchdog timeout initiated BMC dump. 442 [Tags] Verify_Core_Watchdog_Initiated_BMC_Dump 443 444 Redfish Delete All BMC Dumps 445 Redfish Power Off stack_mode=skip 446 447 # Trigger watchdog timeout. 448 Redfish Initiate Auto Reboot 2000 449 450 # Wait for BMC dump to get generated after injecting watchdog timeout. 451 Wait Until Keyword Succeeds 4 min 20 sec Is BMC Dump Available 452 453 # Verify that only one BMC dump is available. 454 ${dump_entry_list}= Get BMC Dump Entries 455 ${length}= Get length ${dump_entry_list} 456 Should Be Equal As Integers ${length} ${1} 457 458 459*** Keywords *** 460 461Get BMC Dump Entries 462 [Documentation] Return BMC dump ids list. 463 464 ${dump_uris}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries 465 ${dump_ids}= Create List 466 467 FOR ${dump_uri} IN @{dump_uris} 468 ${dump_id}= Fetch From Right ${dump_uri} / 469 Append To List ${dump_ids} ${dump_id} 470 END 471 472 [Return] ${dump_ids} 473 474 475Is BMC Dump Available 476 [Documentation] Verify if BMC dump is available. 477 478 ${dump_entries}= Get BMC Dump Entries 479 480 # Verifying that BMC dump is available. 481 ${length}= Get length ${dump_entries} 482 Should Be True 0 < ${length} 483 484 485Get Disk Usage For Dumps 486 [Documentation] Return disk usage in kilobyte for BMC dumps. 487 488 ${usage_output} ${stderr} ${rc}= BMC Execute Command du -s ${BMC_DUMP_COLLECTOR_PATH} 489 490 # Example of output from above BMC cli command. 491 # $ du -s /var/lib/phosphor-debug-collector/dumps 492 # 516 /var/lib/phosphor-debug-collector/dumps 493 494 ${usage_output}= Fetch From Left ${usage_output} / 495 ${usage_output}= Convert To Integer ${usage_output} 496 497 [return] ${usage_output} 498 499 500Test Teardown Execution 501 [Documentation] Do test teardown operation. 502 503 FFDC On Test Case Fail 504 Close All Connections 505