1OpenBMC & Systemd 2=================== 3OpenBMC uses [systemd](https://www.freedesktop.org/wiki/Software/systemd/) to 4manage all processes. It has its own set of target and service units to 5control which processes are started. There is a lot of documentation on systemd 6and to do OpenBMC state work, you're going to have to read some of it. Here's 7the highlights: 8 9[Unit](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#) - 10Units are the basic framework of all systemd work. 11[Service](https://www.freedesktop.org/software/systemd/man/systemd.service.html) - 12Services are a type of unit, that define the processes to be run and execute. 13[Target](https://www.freedesktop.org/software/systemd/man/systemd.target.html) - 14Targets are another type of unit, they have two purposes: 15 161. Define synchronization points among services. 172. Define the services that get run for a given target. 18 19On an OpenBMC system, you can go to /lib/systemd/system/ and see all of the 20systemd units on the system. You can easily cat these files to start looking at 21the relationships among them. Service files can also be found in 22/etc/systemd/system and /run/systemd/system as well. 23 24[systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html) is 25the main tool you use to interact with systemd and its units. 26 27---------- 28## Initial Power 29When an OpenBMC system first has power applied, it starts the "default.target" 30unless an alternate target is specified on the kernel command line. In 31Phosphor OpenBMC, there is a link from default.target to multi-user.target. 32 33You'll find all the phosphor services associated with multi-user.target. 34 35## Server Power On 36When OpenBMC is used within a server, the [obmc-host-start@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-start%40.target) 37is what drives the boot of the system. 38 39If you dig into its .requires relationship, you'll see the following in the file 40system 41 42``` 43ls -1 /lib/systemd/system/obmc-host-start@0.target.requires 44obmc-chassis-poweron@0.target 45start_host@0.service 46``` 47 48You can see we have another target in here, obmc-chassis-poweron@0.target, 49along with a service that will all be started by systemd when you do a 50"systemctl start obmc-host-start@0.target". 51 52The target has corresponding services associated with it: 53``` 54ls -1 /lib/systemd/system/obmc-chassis-poweron\@0.target.requires/ 55op-power-start@0.service 56op-wait-power-on@0.service 57``` 58So basically, if you run `systemctl start obmc-host-start@0.target` then 59systemd will start execution of all associated services. 60 61The services have dependencies within them that control the execution of each 62service (for example, the op-power-start.service will run prior to the 63op-wait-power-on.service). These dependencies are set using targets and the 64Wants,Before,After keywords. 65 66## Server Power Off (Soft) 67The soft server power off function is encapsulated in the 68obmc-host-shutdown@.target. This target is soft in that it notifies the host 69of the power off request and gives it a certain amount of time to shut itself 70down. 71 72## Server Power Off (Hard) 73The hard server power off is encapsulated in the 74obmc-chassis-hard-poweroff@.target. This target will force the stopping 75of the soft power off service if running, and immediately cut power to the 76system. 77 78## Server Reboot 79The reboot of the server is encapsulated in the obmc-host-reboot@.target. 80This target will utilize the soft power off target and then, once that 81completes, start the host power on target. 82 83## Systemd Control in OpenBMC 84There are a collection of services within OpenBMC that interact with systemd and 85its unit files, providing somewhat of an abstraction layer from the user of the 86OpenBMC system and systemd. See the [state](https://github.com/openbmc/phosphor-dbus-interfaces/tree/master/xyz/openbmc_project/State) 87interfaces for more information on this function. 88 89For example, if you wanted to execute the server power on function, you would do 90the following: 91 92> busctl set-property xyz.openbmc_project.State.Host /xyz/openbmc_project/state/host0 93xyz.openbmc_project.State.Host RequestedHostTransition s 94xyz.openbmc_project.State.Host.Transition.On 95 96Underneath the covers, this is calling systemd with the server power on target. 97 98## Error Handling of Systemd 99With great numbers of targets and services, come great chances for failures. 100To make OpenBMC a robust and productive system, it needs to be sure to have an 101error handling policy for when services and their targets fail. 102 103When a failure occurs, the OpenBMC software needs to notify the users of the 104system and provide mechanisms for either the system to automatically retry the 105failed operation (i.e. reboot the system) or to stay in a quiesced state so that 106error data can be collected and the failure can be investigated. 107 108There are two main failure scenarios when it comes to OpenBMC and systemd usage: 109 1101. A service within a target fails 111- If the service is a "oneshot" type, and the service is required 112(not wanted) by the target then the target will fail if the service 113fails 114 - Define a behavior for when the target fails using the 115 "OnFailure" option (i.e. go to a new failure target if any required 116 service fails) 117- If the service is not a "oneshot", then it can not fail the target 118(the target only knows that it started successfully) 119 - Define a behavior for when the service fails (OnFailure) 120 option. 121 - The service can not have "RemainAfterExit=yes" otherwise, the OnFailure 122 action does not occur until the service is stopped (instead of when it 123 fails) 124 - *See more information below on [RemainAfterExit](#RemainAfterExit) 125 1262. A failure outside of a normal systemd target/service (host watchdog expires, 127host checkstop detected) 128- The service which detects this failure is responsible for logging the 129appropriate error, and instructing systemd to go to the appropriate target 130 131Within OpenBMC, there is a host quiesce target. This is the target that other 132host related targets should go to when they hit a failure. Other software within 133OpenBMC can then monitor for the entry into this quiesce target and will handle 134the halt vs. automatic reboot functionality. 135 136Targets which are not host related, will need special thought in regards to 137their error handling. For example, the target responsible for applying chassis 138power, obmc-chassis-poweron@0.target, will have a 139"OnFailure=obmc-chassis-poweroff@%i.target" error path. That is, if the 140chassis power on target fails then power off the chassis. 141 142The above info sets up some general **guidelines** for our host related 143targets and services: 144 145- All targets should have an "OnFailure=obmc-quiesce-host@.target" 146- All services which are required for a target to achieve its function should 147be RequiredBy that target (not WantedBy) 148- All services should first try to be "Type=oneshot" so that we can just rely on 149the target failure path 150- If a service can not be "Type=oneshot", then it needs to have a 151"OnFailure=obmc-quiesce-host@.target" and ideally set "RemainAfterExit=no" 152(but see caveats on this below) 153- If a service can not be any of these then it's up to the service application 154to call systemd with the obmc-quiesce-host@.target on failures 155 156### RemainAfterExit 157This is set to "yes" for most OpenBMC services to handle the situation where 158someone starts the same target twice. If the associated service with that 159target is not running (i.e. RemainAfterExit=no), then the service will be 160executed again. Think about someone accidentally running the 161obmc-chassis-start@.target twice. If you execute it when the operating system 162is up and running, and the service which toggles the pgood pin is re-executed, 163you're going to crash your system. Given this info, the goal should always be 164to write "oneshot" services that have RemainAfterExit set to yes. 165 166## Target and Service Dependency Details 167There are some tools available out there to visualize systemd service and 168target dependencies (systemd-analyze) but due to the complexity of our design, 169they do not generate anything very useful. 170 171For now, document the current dependencies on a witherspoon system for 172reference. 173 174``` 175R = Requires 176W = Wants 177A = After 178B = Before 179S = Start (runs a command to start another target or service) 180(S) = Synchronization Target 181``` 182 183### Soft Power Off 184``` 185obmc-host-shutdown.target 186 R: xyz.openbmc_project.Ipmi.Internal.SoftPowerOff.service 187 W: obmc-host-stopping.target (S) 188 B: obmc-host-stopping.target (S) 189 R: obmc-chassis-poweroff.target 190 R: obmc-host-stop.target 191 R: op-occ-disable.service 192 B: obmc-host-stop-pre.target 193 R: op-power-stop.service 194 W: obmc-power-stop.target (S) 195 B: obmc-power-stop.target (S) 196 W: obmc-power-stop-pre.target (S) 197 A: obmc-power-stop-pre.target (S) 198 W: mapper-wait@-org-openbmc-control-power.service 199 A: mapper-wait@-org-openbmc-control-power.service 200 R: op-wait-power-off.service 201 B: obmc-power-off.target (S) 202 W: obmc-power-stop.target (S) 203 B: obmc-power-stop.target (S) 204 W: obmc-power-stop-pre.target (S) 205 A: obmc-power-stop-pre.target (S) 206 W: mapper-wait@-org-openbmc-control-power.service 207 A: mapper-wait@-org-openbmc-control-power.service 208 R: op-powered-off.service 209 A: op-wait-power-off.service 210 R: op-wait-power-off.service 211 S: obmc-chassis-powered-off.target 212 W: pcie-poweroff.service 213 B: op-power-stop.service 214 A: obmc-power-stop-pre@.target 215``` 216 217#### Synchronization Target Dependencies 218``` 219obmc-power-stop.target 220 W: obmc-power-stop-pre.target 221 A: obmc-power-stop-pre.target 222 223obmc-power-stop-pre.target 224 W: obmc-host-stopped.target 225 A: obmc-host-stopped.target 226 227obmc-host-stopped.target 228 W: obmc-host-stopping.target 229 A: obmc-host-stopping.target 230 B: obmc-power-stop-pre.target 231 232obmc-host-stopping.target 233 W: obmc-host-stop-pre.target 234 A: obmc-host-stop-pre.target 235 B: obmc-host-stopped.target 236 237obmc-host-stop-pre.target 238 B: obmc-host-stopping.target 239``` 240