1# OpenBMC & Systemd 2 3OpenBMC uses [systemd](https://www.freedesktop.org/wiki/Software/systemd/) to 4manage all processes. It has its own set of target and service units to control 5which processes are started. There is a lot of documentation on systemd and to 6do OpenBMC state work, you're going to have to read some of it. Here's the 7highlights: 8 9[Unit](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#) - 10Units are the basic framework of all systemd work. 11[Service](https://www.freedesktop.org/software/systemd/man/systemd.service.html) - 12Services are a type of unit, that define the processes to be run and execute. 13[Target](https://www.freedesktop.org/software/systemd/man/systemd.target.html) - 14Targets are another type of unit, they have two purposes: 15 161. Define synchronization points among services. 172. Define the services that get run for a given target. 18 19On an OpenBMC system, you can go to /lib/systemd/system/ and see all of the 20systemd units on the system. You can easily cat these files to start looking at 21the relationships among them. Service files can also be found in 22/etc/systemd/system and /run/systemd/system as well. 23 24[systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html) is 25the main tool you use to interact with systemd and its units. 26 27--- 28 29## Initial Power 30 31When an OpenBMC system first has power applied, it starts the "default.target" 32unless an alternate target is specified on the kernel command line. In Phosphor 33OpenBMC, there is a link from `default.target` to `multi-user.target`. 34 35You'll find all the phosphor services associated with `multi-user.target`. 36 37## Server Power On 38 39When OpenBMC is used within a server, the 40[obmc-host-start@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-start%40.target) 41is what drives the boot of the system. 42 43To start it you would run `systemctl start obmc-host-start@0.target`. 44 45If you dig into its .requires relationship, you'll see the following in the file 46system 47 48```shell 49ls -1 /lib/systemd/system/obmc-host-start@0.target.requires/ 50obmc-host-startmin@0.target 51phosphor-reset-host-reboot-attempts@0.service 52``` 53 54The 55[obmc-host-startmin@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-startmin%40.target) 56represents the bare minimum of services and targets required to start the host. 57This target is also utilized in host reboot scenarios. This distinction of a 58host-start and a host-startmin target allows the user to put services in the 59`obmc-host-start@.target` that should only be run on an initial host boot (and 60not run on host reboots). For example, in the output above you can see the user 61only wants to run the `phosphor-reset-host-reboot-attempts@0.service` on a fresh 62host boot attempt. 63 64Next if we look at the `obmc-host-startmin@0.target`, we see this: 65 66```shell 67ls -1 /lib/systemd/system/obmc-host-startmin@0.target.requires/ 68obmc-chassis-poweron@0.target 69start_host@0.service 70``` 71 72You can see within `obmc-host-startmin@0.target` that we have another target in 73there, `obmc-chassis-poweron@0.target`, along with a service aptly named 74`start_host@0.service`. 75 76The `obmc-chassis-poweron@0.target` has corresponding services associated with 77it: 78 79```shell 80ls -1 /lib/systemd/system/obmc-chassis-poweron@0.target.requires/ 81op-power-start@0.service 82op-wait-power-on@0.service 83``` 84 85If you run `systemctl start obmc-host-start@0.target` then systemd will start 86execution of all of the above associated target and services. 87 88The services have dependencies within them that control the execution of each 89service (for example, the op-power-start.service will run prior to the 90op-wait-power-on.service). These dependencies are set using targets and the 91Wants,Before,After keywords. 92 93## Server Power Off (Soft) 94 95The soft server power off function is encapsulated in the 96`obmc-host-shutdown@.target`. This target is soft in that it notifies the host 97of the power off request and gives it a certain amount of time to shut itself 98down. 99 100## Server Power Off (Hard) 101 102The hard server power off is encapsulated in the 103`obmc-chassis-hard-poweroff@.target`. This target will force the stopping of the 104soft power off service if running, and immediately cut power to the system. 105 106## Server Reboot 107 108The reboot of the server is encapsulated in the `obmc-host-reboot@.target`. This 109target will utilize the soft power off target and then, once that completes, 110start the host power on target. 111 112## Server Quiesce 113 114The `obmc-host-quiesce@.target` is utilized in host error scenarios. When the 115`obmc-host-quiesce@0.target` is entered, it puts the host state D-Bus 116[object](https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/yaml/xyz/openbmc_project/State/Host.interface.yaml) 117in a `Quiesced` state. 118 119## Server Emergency Power Off due to Error 120 121The `obmc-chassis-emergency-poweroff@.target` is a wrapper target around the 122`obmc-chassis-hard-poweroff@.target` and `obmc-host-quiesce@.target`. It is 123utilized by applications that have detected critical thermal or power errors 124which require an immediate shutdown of the system. It will turn the chassis off 125and put the host into Quiesce (if the host is running). Certain non-critical 126services in the shutdown path can conflict with this target to ensure only the 127most critical services are run in this path. 128 129Automated error recovery (i.e. host reboot) will not be done if this target is 130started. User intervention is required to exit from it. The user could request a 131power on if they wished or a host stop / power off if they wanted to get out of 132quiesce. 133 134## Systemd Control in OpenBMC 135 136There are a collection of services within OpenBMC that interact with systemd and 137its unit files, providing somewhat of an abstraction layer from the user of the 138OpenBMC system and systemd. See the 139[state](https://github.com/openbmc/phosphor-dbus-interfaces/tree/master/yaml/xyz/openbmc_project/State) 140interfaces for more information on this function. 141 142For example, if you wanted to execute the server power on function, you would do 143the following: 144 145> busctl set-property xyz.openbmc_project.State.Host 146> /xyz/openbmc_project/state/host0 xyz.openbmc_project.State.Host 147> RequestedHostTransition s xyz.openbmc_project.State.Host.Transition.On 148 149Underneath the covers, this is calling systemd with the server power on target. 150 151## Systemd Services or Monitoring Applications 152 153A common question when creating new OpenBMC applications which need to execute 154some logic in the context of systemd targets is whether they should be triggered 155by systemd services or by monitoring for the appropriate D-Bus signal indicating 156the start/stop of the target they are interested in. 157 158The basic guidelines for when to create a systemd service are the following: 159 160- If your application logic depends on other systemd based services then make it 161 a systemd service and utilize the Wants/After/Before service capabilities. 162- If other applications depend on your application logic then it should be a 163 systemd service. 164- If your application failing during the target start could impact targets or 165 services that run after it, then it should be a systemd service. This ensures 166 dependent targets are not started if your application fails. 167 168## Error Handling of Systemd 169 170With great numbers of targets and services, come great chances for failures. To 171make OpenBMC a robust and productive system, it needs to be sure to have an 172error handling policy for when services and their targets fail. 173 174When a failure occurs, the OpenBMC software needs to notify the users of the 175system and provide mechanisms for either the system to automatically retry the 176failed operation (i.e. reboot the system) or to stay in a quiesced state so that 177error data can be collected and the failure can be investigated. 178 179There are two main failure scenarios when it comes to OpenBMC and systemd usage: 180 1811. A service within a target fails 182 - If the service is a "oneshot" type, and the service is required (not 183 wanted) by the target then the target will fail if the service fails - 184 Define a behavior for when the target fails using the "OnFailure" option 185 (i.e. go to a new failure target if any required service fails) 186 - If the service is not a "oneshot", then it can not fail the target (the 187 target only knows that it started successfully) - Define a behavior for 188 when the service fails (OnFailure) option. - The service can not have 189 "RemainAfterExit=yes" otherwise, the OnFailure action does not occur until 190 the service is stopped (instead of when it fails) - \*See more information 191 below on [RemainAfterExit](#remainafterexit) 192 1932. A failure outside of a normal systemd target/service (host watchdog expires, 194 host checkstop detected) 195 - The service which detects this failure is responsible for logging the 196 appropriate error, and instructing systemd to go to the appropriate target 197 198Within OpenBMC, there is a host quiesce target. This is the target that other 199host related targets should go to when they hit a failure. Other software within 200OpenBMC can then monitor for the entry into this quiesce target and will handle 201the halt vs. automatic reboot functionality. 202 203Targets which are not host related, will need special thought in regards to 204their error handling. For example, the target responsible for applying chassis 205power, `obmc-chassis-poweron@0.target`, will have a 206`OnFailure=obmc-chassis-poweroff@%i.target` error path. That is, if the chassis 207power on target fails then power off the chassis. 208 209The above info sets up some general **guidelines** for our host related targets 210and services: 211 212- All targets should have a `OnFailure=obmc-quiesce-host@.target` 213- All services which are required for a target to achieve its function should be 214 RequiredBy that target (not WantedBy) 215- All services should first try to be "Type=oneshot" so that we can just rely on 216 the target failure path 217- If a service can not be "Type=oneshot", then it needs to have a 218 `OnFailure=obmc-quiesce-host@.target` and ideally set "RemainAfterExit=no" 219 (but see caveats on this below) 220- If a service can not be any of these then it's up to the service application 221 to call systemd with the `obmc-quiesce-host@.target` on failures 222 223### RemainAfterExit 224 225This is set to "yes" for most OpenBMC services to handle the situation where 226someone starts the same target twice. If the associated service with that target 227is not running (i.e. RemainAfterExit=no), then the service will be executed 228again. Think about someone accidentally running the 229`obmc-chassis-poweron@.target` twice. If you execute it when the operating 230system is up and running, and the service which toggles the pgood pin is 231re-executed, you're going to crash your system. Given this info, the goal should 232always be to write "oneshot" services that have RemainAfterExit set to yes. 233 234## Target and Service Dependency Details 235 236There are some tools available out there to visualize systemd service and target 237dependencies (systemd-analyze) but due to the complexity of our design, they do 238not generate anything very useful. 239 240For now, document the current dependencies on a witherspoon system for 241reference. 242 243```text 244R = Requires 245W = Wants 246A = After 247B = Before 248S = Start (runs a command to start another target or service) 249(S) = Synchronization Target 250``` 251 252### Soft Power Off 253 254```text 255obmc-host-shutdown.target 256 R: xyz.openbmc_project.Ipmi.Internal.SoftPowerOff.service 257 W: obmc-host-stopping.target (S) 258 B: obmc-host-stopping.target (S) 259 R: obmc-chassis-poweroff.target 260 R: obmc-host-stop.target 261 R: op-occ-disable.service 262 B: obmc-host-stop-pre.target 263 R: op-power-stop.service 264 W: obmc-power-stop.target (S) 265 B: obmc-power-stop.target (S) 266 W: obmc-power-stop-pre.target (S) 267 A: obmc-power-stop-pre.target (S) 268 W: mapper-wait@-org-openbmc-control-power.service 269 A: mapper-wait@-org-openbmc-control-power.service 270 R: op-wait-power-off.service 271 B: obmc-power-off.target (S) 272 W: obmc-power-stop.target (S) 273 B: obmc-power-stop.target (S) 274 W: obmc-power-stop-pre.target (S) 275 A: obmc-power-stop-pre.target (S) 276 W: mapper-wait@-org-openbmc-control-power.service 277 A: mapper-wait@-org-openbmc-control-power.service 278 R: op-powered-off.service 279 A: op-wait-power-off.service 280 R: op-wait-power-off.service 281 S: obmc-chassis-powered-off.target 282 W: pcie-poweroff.service 283 B: op-power-stop.service 284 A: obmc-power-stop-pre@.target 285``` 286 287#### Synchronization Target Dependencies 288 289```text 290obmc-power-stop.target 291 W: obmc-power-stop-pre.target 292 A: obmc-power-stop-pre.target 293 294obmc-power-stop-pre.target 295 W: obmc-host-stopped.target 296 A: obmc-host-stopped.target 297 298obmc-host-stopped.target 299 W: obmc-host-stopping.target 300 A: obmc-host-stopping.target 301 B: obmc-power-stop-pre.target 302 303obmc-host-stopping.target 304 W: obmc-host-stop-pre.target 305 A: obmc-host-stop-pre.target 306 B: obmc-host-stopped.target 307 308obmc-host-stop-pre.target 309 B: obmc-host-stopping.target 310``` 311