1*** Settings *** 2Documentation Verify Auto Restart policy for set of mission critical 3... services needed for functioning on BMC. 4 5Resource ../../lib/resource.robot 6Resource ../../lib/connection_client.robot 7Resource ../../lib/openbmc_ffdc.robot 8Resource ../../lib/utils.robot 9Library ../../data/platform_variables.py 10 11Suite Setup Open Connection And Log In 12Suite Teardown Close All Connections 13Test Teardown FFDC On Test Case Fail 14 15Test Tags Server_Restart_Policy 16 17*** Variables *** 18${LOG_SERVICE} xyz.openbmc_project.Logging.service 19 20*** Test Cases *** 21 22Verify OpenBMC Services Auto Restart Policy 23 [Documentation] Kill active services and expect auto restart. 24 [Tags] Verify_OpenBMC_Services_Auto_Restart_Policy 25 # The services listed below restart policy should be "always" 26 # Command output: 27 # systemctl -p Restart show xyz.openbmc_project.Logging.service | cat 28 # Restart=always 29 @{services}= 30 ... Create List xyz.openbmc_project.Logging.service 31 ... xyz.openbmc_project.ObjectMapper.service 32 ... xyz.openbmc_project.State.BMC.service 33 ... xyz.openbmc_project.State.Chassis@0.service 34 ... xyz.openbmc_project.State.Host@0.service 35 FOR ${SERVICE} IN @{services} 36 Check Service Autorestart ${SERVICE} 37 END 38 39 40Kill Services And Expect Service Restart 41 [Documentation] Kill the service and it must restart. 42 [Tags] Kill_Services_And_Expect_Service_Restart 43 44 # Get the MainPID and service state. 45 ${MainPID}= Get Service Attribute MainPID ${LOG_SERVICE} 46 Should Not Be Equal ${0} ${MainPID} 47 ... msg=Logging service not restarted. 48 49 ${ActiveState}= Get Service Attribute ActiveState ${LOG_SERVICE} 50 Should Be Equal active ${ActiveState} 51 ... msg=Logging Service not in active state. 52 53 BMC Execute Command kill -9 ${MainPID} 54 Sleep 10s reason=Wait for service to restart. 55 56 ${MainPID}= Get Service Attribute MainPID ${LOG_SERVICE} 57 Should Not Be Equal ${0} ${MainPID} 58 ... msg=Logging service not restarted. 59 60 ${ActiveState}= Get Service Attribute ActiveState ${LOG_SERVICE} 61 Should Be Equal active ${ActiveState} 62 ... msg=Logging service not in active state. 63 64Kill The List Of Services And Expect Killed Service Gets Restarted 65 [Documentation] Kill the given services and expect again services get restarted automatically. 66 [Tags] Kill_The_List_Of_Services_And_Expect_Killed_Service_Gets_Restarted 67 68 # Create a list of services in respective server model python file 69 # like romulus.py, witherspoon.py on openbmc-test-automation/data directory etc. 70 # Example of creating a list of services in their respective server model python file 71 # SERVICES = { 72 # "BMC_SERVICES": ['xyz.openbmc_project.Logging.service', 'xyz.openbmc_project.ObjectMapper.service', 73 # 'xyz.openbmc_project.State.BMC.service', 'xyz.openbmc_project.State.Chassis.service', 74 # 'xyz.openbmc_project.State.Host.service'] 75 76 @{auto_restart_policy_always_services}= Create List 77 @{incorrect_auto_restart_policy_services}= Create List 78 @{service_not_started}= Create List 79 80 # Creating an list of services which needs to be validated. 81 82 ${services}= Get Service Restart Policy Services ${OPENBMC_MODEL} 83 ${service_list}= Get From Dictionary ${services} BMC_SERVICES 84 ${length_services}= Get Length ${service_list} 85 86 # From service list it will check service auto-restart policy 87 # If incorrect those services will be appended to incorrect_auto_restart_policy_services list 88 # Proper restart policy services will be appended to auto_restart_policy_always_services list. 89 90 FOR ${service} IN @{service_list} 91 ${service_status}= Run Keyword And Return Status Check Service Autorestart ${service} 92 IF ${service_status} == False 93 Append To List ${incorrect_auto_restart_policy_services} ${service} 94 ELSE 95 Append To List ${auto_restart_policy_always_services} ${service} 96 END 97 END 98 99 ${length_incorrect_autorestart_policy}= Get Length ${incorrect_auto_restart_policy_services} 100 IF ${length_incorrect_autorestart_policy} != 0 and ${length_incorrect_autorestart_policy} == ${length_services} 101 Log ${incorrect_auto_restart_policy_services} 102 Fail msg=All the given services have incorrect auto-restart policy. 103 ELSE IF ${length_incorrect_autorestart_policy} != 0 and ${length_incorrect_autorestart_policy} != ${length_services} 104 Log ${incorrect_auto_restart_policy_services} 105 Run Keyword And Continue On Failure Fail msg=Listed services are having incorrect auto-restart policy. 106 END 107 108 # This will get process id and check the service active state before killing the services. 109 # If service process id was 0 or service was not in active state then those services will get 110 # appended to service_not_started list. 111 # Only services with process ID and in active state get killed and checked whether 112 # they automatically restart and put into active state. 113 114 FOR ${service} IN @{auto_restart_policy_always_services} 115 ${Old_MainPID}= Get Service Attribute MainPID ${service} 116 ${ActiveState}= Get Service Attribute ActiveState ${service} 117 ${main_pid_status}= Run Keyword And Return Status Should Not Be Equal ${0} ${Old_MainPID} 118 ${active_state_status}= Run Keyword And Return Status Should Be Equal active ${ActiveState} 119 IF ${main_pid_status} == False or ${active_state_status} == False 120 Append To List ${service_not_started} ${service} 121 CONTINUE 122 END 123 124 BMC Execute Command kill -9 ${Old_MainPID} 125 Sleep 10s reason=Wait for service to restart. 126 127 ${New_MainPID}= Get Service Attribute MainPID ${service} 128 Run Keyword And Continue On Failure Should Not Be Equal ${0} ${New_MainPID} 129 ... msg=${service} service not restarted. 130 Run Keyword And Continue On Failure Should Not Be Equal ${Old_MainPID} ${New_MainPID} 131 ... msg=Old process ID is mapped to ${service} service after service restart.. 132 133 ${ActiveState}= Get Service Attribute ActiveState ${service} 134 Run Keyword And Continue On Failure Should Be Equal active ${ActiveState} 135 ... msg=${service} service not in active state. 136 END 137 138 ${length_service_not_started}= Get Length ${service_not_started} 139 ${incorrect_services}= Evaluate ${length_incorrect_autorestart_policy} + ${length_service_not_started} 140 141 IF ${incorrect_services} == ${length_services} and ${length_service_not_started} != 0 142 Log ${service_not_started} 143 Fail msg=All the services were either not started or not in active state by default. 144 ELSE IF ${incorrect_services} != ${length_services} and ${length_service_not_started} != 0 145 Log ${service_not_started} 146 Fail msg=Few listed services were either not started or not in active state by default. 147 END 148 149*** Keywords *** 150 151Check Service Autorestart 152 [Documentation] Check if given policy is "always". 153 [Arguments] ${servicename} 154 # servicename Qualified service name 155 ${restart_policy}= Get Service Attribute Restart ${servicename} 156 Should Be Equal always ${restart_policy} 157 ... msg=Incorrect policy for ${servicename} 158