1OpenBMC & Systemd 2=================== 3OpenBMC uses [systemd](https://www.freedesktop.org/wiki/Software/systemd/) to 4manage all processes. It has its own set of target and service units to 5control which processes are started. There is a lot of documentation on systemd 6and to do OpenBMC state work, you're going to have to read some of it. Here's 7the highlights: 8 9[Unit](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#) - 10Units are the basic framework of all systemd work. 11[Service](https://www.freedesktop.org/software/systemd/man/systemd.service.html) - 12Services are a type of unit, that define the processes to be run and execute. 13[Target](https://www.freedesktop.org/software/systemd/man/systemd.target.html) - 14Targets are another type of unit, they have two purposes: 15 161. Define synchronization points among services. 172. Define the services that get run for a given target. 18 19On an OpenBMC system, you can go to /lib/systemd/system/ and see all of the 20systemd units on the system. You can easily cat these files to start looking at 21the relationships among them. Service files can also be found in 22/etc/systemd/system and /run/systemd/system as well. 23 24[systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html) is 25the main tool you use to interact with systemd and its units. 26 27---------- 28## Initial Power 29When an OpenBMC system first has power applied, it starts the "default.target" 30unless an alternate target is specified on the kernel command line. In 31Phosphor OpenBMC, there is a link from `default.target` to `multi-user.target`. 32 33You'll find all the phosphor services associated with `multi-user.target`. 34 35## Server Power On 36When OpenBMC is used within a server, the [obmc-host-start@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-start%40.target) 37is what drives the boot of the system. 38 39To start it you would run `systemctl start obmc-host-start@0.target`. 40 41If you dig into its .requires relationship, you'll see the following in the file 42system 43 44``` 45ls -1 /lib/systemd/system/obmc-host-start@0.target.requires/ 46obmc-host-startmin@0.target 47phosphor-reset-host-reboot-attempts@0.service 48``` 49The [obmc-host-startmin@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-startmin%40.target) 50represents the bare minimum of services and targets required to start the host. 51This target is also utilized in host reboot scenarios. This distinction of a 52host-start and a host-startmin target allows the user to put services in the 53`obmc-host-start@.target` that should only be run on an initial host boot (and 54not run on host reboots). For example, in the output above you can see the user 55only wants to run the `phosphor-reset-host-reboot-attempts@0.service` on a fresh 56host boot attempt. 57 58Next if we look at the `obmc-host-startmin@0.target`, we see this: 59``` 60ls -1 /lib/systemd/system/obmc-host-startmin@0.target.requires/ 61obmc-chassis-poweron@0.target 62start_host@0.service 63``` 64 65You can see within `obmc-host-startmin@0.target` that we have another target in 66there, `obmc-chassis-poweron@0.target`, along with a service aptly named 67`start_host@0.service`. 68 69The `obmc-chassis-poweron@0.target` has corresponding services associated with 70it: 71``` 72ls -1 /lib/systemd/system/obmc-chassis-poweron@0.target.requires/ 73op-power-start@0.service 74op-wait-power-on@0.service 75``` 76 77If you run `systemctl start obmc-host-start@0.target` then systemd will start 78execution of all of the above associated target and services. 79 80The services have dependencies within them that control the execution of each 81service (for example, the op-power-start.service will run prior to the 82op-wait-power-on.service). These dependencies are set using targets and the 83Wants,Before,After keywords. 84 85## Server Power Off (Soft) 86The soft server power off function is encapsulated in the 87`obmc-host-shutdown@.target`. This target is soft in that it notifies the host 88of the power off request and gives it a certain amount of time to shut itself 89down. 90 91## Server Power Off (Hard) 92The hard server power off is encapsulated in the 93`obmc-chassis-hard-poweroff@.target`. This target will force the stopping 94of the soft power off service if running, and immediately cut power to the 95system. 96 97## Server Reboot 98The reboot of the server is encapsulated in the `obmc-host-reboot@.target`. 99This target will utilize the soft power off target and then, once that 100completes, start the host power on target. 101 102## Server Quiesce 103The `obmc-host-quiesce@.target` is utilized in host error scenarios. When the 104`obmc-host-quiesce@0.target` is entered, it puts the host state D-Bus 105[object][https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/State/Host.interface.yaml] 106in a `Quiesced` state. 107 108## Server Emergency Power Off due to Error 109The `obmc-chassis-emergency-poweroff@.target` is a wrapper target around the 110`obmc-chassis-hard-poweroff@.target` and `obmc-host-quiesce@.target`. It is 111utilized by applications that have detected critical thermal or power errors 112which require an immediate shutdown of the system. It will turn the chassis off 113and put the host into Quiesce (if the host is running). Certain non-critical 114services in the shutdown path can conflict with this target to ensure only the 115most critical services are run in this path. 116 117Automated error recovery (i.e. host reboot) will not be done if this target is 118started. User intervention is required to exit from it. The user could request 119a power on if they wished or a host stop / power off if they wanted to get out 120of quiesce. 121 122## Systemd Control in OpenBMC 123There are a collection of services within OpenBMC that interact with systemd and 124its unit files, providing somewhat of an abstraction layer from the user of the 125OpenBMC system and systemd. See the [state](https://github.com/openbmc/phosphor-dbus-interfaces/tree/master/xyz/openbmc_project/State) 126interfaces for more information on this function. 127 128For example, if you wanted to execute the server power on function, you would do 129the following: 130 131> busctl set-property xyz.openbmc_project.State.Host /xyz/openbmc_project/state/host0 132xyz.openbmc_project.State.Host RequestedHostTransition s 133xyz.openbmc_project.State.Host.Transition.On 134 135Underneath the covers, this is calling systemd with the server power on target. 136 137## Systemd Services or Monitoring Applications 138A common question when creating new OpenBMC applications which need to 139execute some logic in the context of systemd targets is whether they should 140be triggered by systemd services or by monitoring for the appropriate 141D-Bus signal indicating the start/stop of the target they are interested in. 142 143The basic guidelines for when to create a systemd service are the following: 144- If your application logic depends on other systemd based services then 145 make it a systemd service and utilize the Wants/After/Before service 146 capabilities. 147- If other applications depend on your application logic then it should be a 148 systemd service. 149- If your application failing during the target start could impact targets or 150 services that run after it, then it should be a systemd service. This ensures 151 dependent targets are not started if your application fails. 152 153## Error Handling of Systemd 154With great numbers of targets and services, come great chances for failures. 155To make OpenBMC a robust and productive system, it needs to be sure to have an 156error handling policy for when services and their targets fail. 157 158When a failure occurs, the OpenBMC software needs to notify the users of the 159system and provide mechanisms for either the system to automatically retry the 160failed operation (i.e. reboot the system) or to stay in a quiesced state so that 161error data can be collected and the failure can be investigated. 162 163There are two main failure scenarios when it comes to OpenBMC and systemd usage: 164 1651. A service within a target fails 166- If the service is a "oneshot" type, and the service is required 167(not wanted) by the target then the target will fail if the service 168fails 169 - Define a behavior for when the target fails using the 170 "OnFailure" option (i.e. go to a new failure target if any required 171 service fails) 172- If the service is not a "oneshot", then it can not fail the target 173(the target only knows that it started successfully) 174 - Define a behavior for when the service fails (OnFailure) 175 option. 176 - The service can not have "RemainAfterExit=yes" otherwise, the OnFailure 177 action does not occur until the service is stopped (instead of when it 178 fails) 179 - *See more information below on [RemainAfterExit](#RemainAfterExit) 180 1812. A failure outside of a normal systemd target/service (host watchdog expires, 182host checkstop detected) 183- The service which detects this failure is responsible for logging the 184appropriate error, and instructing systemd to go to the appropriate target 185 186Within OpenBMC, there is a host quiesce target. This is the target that other 187host related targets should go to when they hit a failure. Other software within 188OpenBMC can then monitor for the entry into this quiesce target and will handle 189the halt vs. automatic reboot functionality. 190 191Targets which are not host related, will need special thought in regards to 192their error handling. For example, the target responsible for applying chassis 193power, `obmc-chassis-poweron@0.target`, will have a 194`OnFailure=obmc-chassis-poweroff@%i.target` error path. That is, if the 195chassis power on target fails then power off the chassis. 196 197The above info sets up some general **guidelines** for our host related 198targets and services: 199 200- All targets should have a `OnFailure=obmc-quiesce-host@.target` 201- All services which are required for a target to achieve its function should 202be RequiredBy that target (not WantedBy) 203- All services should first try to be "Type=oneshot" so that we can just rely on 204the target failure path 205- If a service can not be "Type=oneshot", then it needs to have a 206`OnFailure=obmc-quiesce-host@.target` and ideally set "RemainAfterExit=no" 207(but see caveats on this below) 208- If a service can not be any of these then it's up to the service application 209to call systemd with the `obmc-quiesce-host@.target` on failures 210 211### RemainAfterExit 212This is set to "yes" for most OpenBMC services to handle the situation where 213someone starts the same target twice. If the associated service with that 214target is not running (i.e. RemainAfterExit=no), then the service will be 215executed again. Think about someone accidentally running the 216`obmc-chassis-poweron@.target` twice. If you execute it when the operating system 217is up and running, and the service which toggles the pgood pin is re-executed, 218you're going to crash your system. Given this info, the goal should always be 219to write "oneshot" services that have RemainAfterExit set to yes. 220 221## Target and Service Dependency Details 222There are some tools available out there to visualize systemd service and 223target dependencies (systemd-analyze) but due to the complexity of our design, 224they do not generate anything very useful. 225 226For now, document the current dependencies on a witherspoon system for 227reference. 228 229``` 230R = Requires 231W = Wants 232A = After 233B = Before 234S = Start (runs a command to start another target or service) 235(S) = Synchronization Target 236``` 237 238### Soft Power Off 239``` 240obmc-host-shutdown.target 241 R: xyz.openbmc_project.Ipmi.Internal.SoftPowerOff.service 242 W: obmc-host-stopping.target (S) 243 B: obmc-host-stopping.target (S) 244 R: obmc-chassis-poweroff.target 245 R: obmc-host-stop.target 246 R: op-occ-disable.service 247 B: obmc-host-stop-pre.target 248 R: op-power-stop.service 249 W: obmc-power-stop.target (S) 250 B: obmc-power-stop.target (S) 251 W: obmc-power-stop-pre.target (S) 252 A: obmc-power-stop-pre.target (S) 253 W: mapper-wait@-org-openbmc-control-power.service 254 A: mapper-wait@-org-openbmc-control-power.service 255 R: op-wait-power-off.service 256 B: obmc-power-off.target (S) 257 W: obmc-power-stop.target (S) 258 B: obmc-power-stop.target (S) 259 W: obmc-power-stop-pre.target (S) 260 A: obmc-power-stop-pre.target (S) 261 W: mapper-wait@-org-openbmc-control-power.service 262 A: mapper-wait@-org-openbmc-control-power.service 263 R: op-powered-off.service 264 A: op-wait-power-off.service 265 R: op-wait-power-off.service 266 S: obmc-chassis-powered-off.target 267 W: pcie-poweroff.service 268 B: op-power-stop.service 269 A: obmc-power-stop-pre@.target 270``` 271 272#### Synchronization Target Dependencies 273``` 274obmc-power-stop.target 275 W: obmc-power-stop-pre.target 276 A: obmc-power-stop-pre.target 277 278obmc-power-stop-pre.target 279 W: obmc-host-stopped.target 280 A: obmc-host-stopped.target 281 282obmc-host-stopped.target 283 W: obmc-host-stopping.target 284 A: obmc-host-stopping.target 285 B: obmc-power-stop-pre.target 286 287obmc-host-stopping.target 288 W: obmc-host-stop-pre.target 289 A: obmc-host-stop-pre.target 290 B: obmc-host-stopped.target 291 292obmc-host-stop-pre.target 293 B: obmc-host-stopping.target 294``` 295