docs/designs/bmc-service-failure-debug-and-recovery.md

1 # BMC Service Failure Debug and Recovery
22 recovery of a failed BMC.
40 recovery in the face of well-defined error conditions, but the need to mitigate
69 In the face of continued operation due to internal recovery a service restart is
156 #### Recovery Mechanisms
158 The ability for external consumers to control the recovery behaviour of BMC
163 | Severity | BMC Recovery Mechanism  | Used for                                                    …
165 | 1        | Graceful reboot request | Normal circumstances or recovery from platform data provider…
166 | 2        | Forceful reboot request | Recovery from unresponsive platform data transport providers…
178 therefore limited to recovery of unresponsive in-band interfaces.
197 needs an interface to the BMC that is dedicated to the role of BMC recovery,
202 #### Behavioural Requirements for Recovery Mechanism 2
207    recovery message from the host
211 1. The host has some indication that the recovery process has been activated
216 1. The host make use of a timeout to escalate to recovery mechanism 3 as it's
217    possible the BMC will be unresponsive to recovery mechanism 2
219 #### Analysis of BMC Recovery Mechanisms for Power10 Platforms
221 The implementation of recovery mechanism 1 is already accounted for in the
225 To address recovery mechanism 3, the Power10 platform designs wire up a GPIO
227 the BMC has become unresponsive to its escalating recovery requests, it can
231 to implement recovery mechanism 2 in the platform designs.
235 The implementation of recovery mechanism 2 is limited to using existing
259 implementing recovery mechanism 2:
287 interactions with `init`, as the reason for invoking the recovery mechanism is
328 1. If the `Ready` bit in STR is clear, escalate to recovery mechanism 3.
331 2. If the `IBF` bit in STR is set, escalate to recovery mechanism 3. Otherwise,
343 7. If `Ready` becomes set before expiry, disarm the escalation timer. Recovery
346 8. Escalate to recovery mechanism 3 if the escalation timer expires at any point
350 behave correctly without one. Recovery is only necessary if other paths have
351 failed, so STR can be read by the host when it decides recovery is required, and
364 the recovery protocol.