1# BMC Reset with Host Booted 2 3Author: Andrew Geissler (geissonator) 4 5Other contributors: 6 7Created: June 23rd, 2020 8 9## Problem Description 10BMCs can reboot for a lot of different reasons. User request, firmware update, 11and a variety of different error scenarios. When the BMC is rebooted while the 12host is up and running, there needs to be a process by which the two synchronize 13with each other and the BMC gets itself into a state that matches with the host. 14 15## Background and References 16A good portion of this is explained in the phosphor-state-manager [README][1]. 17 18This design doc is written to formalize the design and add some more 19details on dealing with both IPMI and PLDM communication to the host as well as 20desired behavior when unable to talk with the host. 21 22The high level flow is that OpenBMC software first checks for pgood, and if set, 23it then checks the host to see if it is up and alive. If the power is on and the 24host is running, then files are created in the filesystem to indicate this: 25- /run/openbmc/chassis@0-on 26- /run/openbmc/host@0-on 27 28It should be noted that although full support is not in place for multi-chassis 29and multi-host systems, the framework is there to build on. 30`op-reset-chassis-running@.service` is a templated service, checking pgood in 31its instances power domain. It creates a file in the filesystem, 32/run/openbmc/chassis@%i-on, to indicate power is on for that instance. Similar 33implementation is done for the host via `phosphor-reset-host-check@.service` and 34the file /run/openbmc/host@%i-on. 35 36If chassis power is on and the host is up, then 37`obmc-chassis-poweron@.target` and `obmc-host-start@.target` are started. 38 39The /run/ files are used by OpenBMC services to determine if they need to run 40or not when the chassis and host targets are started. For example, if the 41chassis is already powered on and the host is running, there is no need to 42actually turn power on, or start the host. The behavior wanted is that these 43services "start" but do nothing. That is commonly done within a systemd service 44file via the following: 45- `ConditionPathExists=!/run/openbmc/chassis@%i-on` 46- `ConditionPathExists=!/run/openbmc/host@%i-on` 47 48This allows the targets to start and for the BMC to get in synch with the 49state of the chassis and host without any special software checks. 50 51Different systems have different requirements on what the behavior should be 52when the chassis power is on, but the host is unreachable. This design needs to 53allow this customization. For example, some systems would prefer to just leave 54the system in whatever state it is in and let the user correct things. Some 55systems want to recover automatically (i.e. reboot the host) for the user. 56Some systems have a hybrid approach where depending on where the host was in 57its boot, the BMC may leave it or recover. 58 59## Requirements 60- Support both IPMI and PLDM as mechanisms to determine if the host is running. 61 - Allow either or both to be enabled 62- Support custom behavior when chassis power is on but the BMC is unable to 63 communicate with the host 64- Both IPMI and PLDM stacks will give the host a set amount of time to 65 respond. Lack of response within this time limit will result in the BMC 66 potentially taking recovery actions. 67 - This time limit must be configurable at build time 68- IPMI and PLDM will implement a phosphor-dbus-interface interface, 69 `xyz.openbmc_project.Condition.HostFirmware`, which will have a 70 `CurrentFirmwareCondition` property which other applications can read to 71 determine if the host is running. 72 73### IPMI Detailed Requirements 74- IPMI will continue to utilize the SMS_ATN command to indicate to the host that 75 a "Get Message Flags Command" is requested. Upon the host issuing that 76 command, it will be considered up and running 77 78### PLDM Detailed Requirements 79- PLDM will utilize a GetTID command to the host to determine if it is running 80- Where applicable, PLDM will provide a mechanism to distinguish between 81 different host firmware stacks 82 - For example, on IBM systems there is a difference between the 83 hostboot (host initialization) firmware and Hypervisor firmware stacks. 84 Both are host firmware and talking PLDM but the BMC recovery paths will 85 differ based on which is running. The `CurrentFirmwareCondition` property 86 should not return "Running" unless the Hypervisor firmware is running. 87 88## Proposed Design 89High Level Flow: 90- Check pgood 91- Call mapper for all implementations of 92 `xyz.openbmc_project.Condition.HostFirmware` PDI interface 93- Read `CurrentFirmwareCondition` property of all interface. If any call returns 94 that a host is running then create file and start host target. 95- Otherwise, check host via any custom mechanisms 96- Execute automated recovery of host if desired 97 98IPMI and PLDM software will be started as applicable. A combination of systemd 99services and applications within phosphor-state-manager will coordinate the 100checking of pgood, and if set, request the IPMI and PLDM applications to 101discover if the host is running. Based on the response from these queries 102the software in phosphor-state-manager will take the appropriate action of 103creating the /run files and starting the chassis and host targets or entering 104into recovery of the host. 105 106The systemd targets responsible for this and any common services will be hosted 107within phosphor-state-manager. Any system or company specific services can 108be installed in the common targets: 109- obmc-chassis-powerreset@.target.require 110- obmc-host-reset@.target.requires 111 112### Automated Recovery when host does not respond 113 114A separate service and application will be created within phosphor-state-manager 115to execute the following logic in situations where chassis power is on 116but the host has failed to respond to any of the different mechanisms to 117communicate with it: 118- If chassis power on (/run/openbmc/chassis@%i-on) 119- And host is off (!ConditionPathExists=!/run/openbmc/host@%i-on) 120- And restored BootProgress is not None 121- Then (assume host was booting before BMC reboot) 122 - Log error indicating situation 123 - Move host to Quiesce and allow automated recovery to kick in 124 125### Note on custom mechanism for IBM systems 126IBM systems will utilize a processor CFAM register. The specific register is 127**Mailbox scratch register 12**. 128 129If the chassis power is on but the BMC is unable to communicate with the 130host via IPMI or PLDM, then the BMC will read this processor CFAM register. 131 132The Host code will write `0xA5000001` to this register to indicate when it has 133reached a state in which it can boot an operating system without needing the 134BMC. If the BMC sees this value written in the CFAM register, then it will leave 135the host as-is, even if it is unable to communicate with the host over IPMI or 136PLDM. It will log an error indicating it was unable to communicate with the host 137but it will also show the host state as `Running`. 138 139If the register is not `0xA5000001`, then the BMC will follow whatever recovery 140mechanisms are defined for when the host has a failure (most likely a reboot 141of the host). 142 143It is the responsibility of the host firmware to set this register as 144applicable during the boot of the system. Host firmware will clear this register 145in shutdown scenarios. To handle different host crash scenarios, the register 146will also be cleared by the BMC firmware on power off's, system checkstops, and 147during Memory Preserving reboots. 148 149 150## Alternatives Considered 151One thought was to avoid IPMI/PLDM all together and only use a "lowest common 152denominator" hardware register of some sort. The problem with that is you start 153creating your own protocol, and before you know it you have something like IPMI 154or PLDM anyway, except you created something custom. So 99% of the time the 155IPMI or PLDM path will be fine, and as a backup option, system owners can 156put their own custom host-detection applications in. 157 158## Impacts 159None 160 161## Testing 162The normal path of IPMI and PLDM will be simple to test. Boot your system and 163reboot the BMC. Verify the BMC chassis and host states are correct and verify 164the host continued to run without issue throughout the BMC reset. 165 166The more complicated tests will be error paths: 167- Reboot the BMC while the host is booting, but before it's in a state where 168 it can continue to run without the BMC. Verify the BMC detects this error 169 scenario once it comes back from its reboot and takes the proper recovery 170 actions for the host. 171- Reboot the BMC when the host is up and running but disable the IPMI/PLDM stack 172 on the host so it does not respond to the BMC when it comes back from its 173 reboot. Ensure the BMC follows the defined recovery for the system in this 174 situation. 175 176[1]: https://github.com/openbmc/phosphor-state-manager#bmc-reset-with-host-andor-chassis-on 177