1# Fail Boot on Hardware Errors 2 3Author: Andrew Geissler (geissonator) 4 5Primary assignee: Andrew Geissler (geissonator) 6 7Other contributors: 8 9Created: Feb 20, 2020 10 11## Problem Description 12Some groups, for example a manufacturing team, have a requirement for the BMC 13firmware to halt a system if an error log is created which calls out a piece of 14hardware. The reason behind this is to ensure a system is not shipped to a 15customer if it has any type of hardware issue. It also ensures when an error is 16found, it is identified quickly and all activity stops until the issue is fixed. 17If the system has a hardware issue once shipped from manufacturing, then the BMC 18firmware behavior should be to report the error, but allow the system to 19continue to boot and operate. 20 21OpenBMC firmware needs a mechanism to support this use case. 22 23## Background and References 24Within IBM, this function has been enabled/disabled by what is called 25manufacturing flags. They were bits the user could set in registry variables 26which the firmware would then query. These registry variables were only 27settable by someone with admin authority to the system. These flags were not 28used outside of manufacturing and test. 29 30Extensions within phosphor-logging may process logs that do not always come 31through the standard phosphor-logging interfaces (for example logs sent 32down by the host). In these cases the system must still halt if those logs 33contain hardware callouts. 34 35[This][1] email thread was sent on this topic to the list. 36 37## Requirements 38- Provide a mechanism to cause the OpenBMC firmware to halt a system if a 39 phosphor-logging log is created with a inventory callout 40 - The mechanism to enable/disable this feature does not need to be an 41 external API (i.e. Redfish). It can simply be a busctl command one runs 42 in an ssh to the BMC 43 - The halt must be obvious to the user when it occurs 44 - The log which causes the halt must be identifiable 45 - The halt must only stop the chassis/host instance that encountered the error 46 - The halt must stop the host (run obmc-host-stop@X.target) associated with 47 the error and attempt to leave system in the fail state (i.e. chassis power 48 remains on if it is on) 49 - The chassis/host instance pair will not be allowed to power on until 50 the log that caused the halt is resolved or deleted 51 - A BMC reset will clear this power on prevention 52- Ensure the mechanism used to halt firmware on inventory callouts can also be 53 utilized by phosphor-logging extensions to halt firmware for other causes 54 - These causes will be defined within the extensions documentation 55- Quiesce the associated host during this failure 56 57**Special Note:** Initially the associated host and chassis will be hard coded to 58chassis0 and host0. More work throughout the BMC stack is required to handle 59multiple chassis and hosts. This design allows that type of feature to be 60enabled at a later time. 61 62## Proposed Design 63Create a [phosphor-settingsd][2] setting, 64`xyz.openbmc_project.Logging.Settings`. Within this create a boolean property 65called QuiesceOnHwError. This property will be hosted under the 66xyz.openbmc_project.Settings service. 67 68Define a new D-Bus interface which will indicate an error has been created which 69will prevent the boot of a chassis/host instance: 70`xyz.openbmc_project.Logging.ErrorBlocksTransition` 71 72This interface will be hosted under a instance based D-Bus object 73`/xyz/openbmc_project/logging/blockX` where X is the instance of the 74chassis/host pair being blocked. 75 76When an error is created via a phosphor-logging interface, the software will 77check to see if the error has a callout, and if so it will check the new 78`xyz.openbmc_project.Logging.Settings.QuiesceOnHwError`. If this is true then 79phosphor-logging will create a `/xyz/openbmc_project/logging/blockX` D-Bus 80object with a `xyz.openbmc_project.Logging.ErrorBlocksTransition` interface 81under it. A mapper [association][3] between the log and this new D-Bus 82object will be created. The corresponding host instance will be put 83in quiesce by phosphor-logging. 84 85The blocked state can be exited by rebooting the BMC or clearing the log 86responsible for the blocking. Other system specific policies could be placed 87in the appropriate targets (for example if a chassis power off should clear 88the block) 89 90See the phosphor-logging [callout][4] design for more information on callouts. 91 92The appropriate `obmc-host-stop@.target` instance will also be called when 93`obmc-bmc-quiesce.target` is started. This ensures the host is stopped as soon as 94the error is discovered. 95 96obmcutil will be enhanced to look for these block interfaces and notify the 97user via the `obmcutil state` command if a block is enabled and what log 98is associated with it. 99 100The goal is to build upon this concept when future design work is done to allow 101developers to associate certain error logs with causing a halt to the system 102until a log is handled. 103 104## Host Errors 105 106In certain scenarios, it is desirable to also halt the boot, and prevent it 107from rebooting, when the host sends down certain errors to the BMC. 108 109These errors may be of SEL format, or may be OEM specific, such as the 110[PEL format][5] used by IBM. 111 112The interfaces provided within phosphor-logging to handle the hardware callout 113scenarios can be repurposed for this use case. 114 115## Alternatives Considered 116Currently this feature is a part of the base phosphor-logging design. If no 117one other then IBM sees value, we could roll this into the PEL-specific 118portion of phosphor-logging. 119 120A systemd target could be created to do the host stop and quiesce (and any 121other system specific things people need) but at this point there doesn't 122seem to be a ton of value in it. Could always be added later if needed. 123 124## Impacts 125This will require some additional checking on reported logs but should have 126minimal overhead. 127 128There will be no changes to system behavior unless a user turns on this new 129setting. 130 131## Testing 132Unit tests will be run to ensure logic to detect errors with logs and verify 133both possible values of the new setting. 134 135Test cases will need to look for this new blocking D-Bus object and handle 136appropriately. 137 138 139[1]: https://lists.ozlabs.org/pipermail/openbmc/2020-February/020575.html 140[2]: https://github.com/openbmc/phosphor-settingsd 141[3]: https://github.com/openbmc/docs/blob/master/architecture/object-mapper.md#associations 142[4]: https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/yaml/xyz/openbmc_project/Common/Callout/README.md 143[5]: https://github.com/openbmc/phosphor-logging/blob/master/extensions/openpower-pels/README.md 144