1OpenBMC & Systemd 2=================== 3OpenBMC uses [systemd](https://www.freedesktop.org/wiki/Software/systemd/) to 4manage all processes. It has its own set of target and service units to 5control which processes are started. There is a lot of documentation on systemd 6and to do OpenBMC state work, you're going to have to read some of it. Here's 7the highlights: 8 9[Unit](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#) - 10Units are the basic framework of all systemd work. 11[Service](https://www.freedesktop.org/software/systemd/man/systemd.service.html) - 12Services are a type of unit, that define the processes to be run and execute. 13[Target](https://www.freedesktop.org/software/systemd/man/systemd.target.html) - 14Targets are another type of unit, they have two purposes: 15 161. Define synchronization points among services. 172. Define the services that get run for a given target. 18 19On an OpenBMC system, you can go to /lib/systemd/system/ and see all of the 20systemd units on the system. You can easily cat these files to start looking at 21the relationships among them. Service files can also be found in 22/etc/systemd/system and /run/systemd/system as well. 23 24[systemctl](https://www.freedesktop.org/software/systemd/man/systemctl.html) is 25the main tool you use to interact with systemd and its units. 26 27---------- 28## Initial Power 29When an OpenBMC system first has power applied, it starts the "default.target" 30unless an alternate target is specified on the kernel command line. In 31Phosphor OpenBMC, there is a link from default.target to multi-user.target. 32 33You'll find all the phosphor services associated with multi-user.target. 34 35## Server Power On 36When OpenBMC is used within a server, the [obmc-host-start@.target](https://github.com/openbmc/phosphor-state-manager/blob/master/target_files/obmc-host-start%40.target) 37is what drives the boot of the system. 38 39If you dig into its .requires relationship, you'll see the following in the file 40system 41 42``` 43ls -1 /lib/systemd/system/obmc-host-start@0.target.requires 44obmc-chassis-poweron@0.target 45start_host@0.service 46``` 47 48You can see we have another target in here, obmc-chassis-poweron@0.target, 49along with a service that will all be started by systemd when you do a 50"systemctl start obmc-host-start@0.target". 51 52The target has corresponding services associated with it: 53``` 54ls -1 /lib/systemd/system/obmc-chassis-poweron\@0.target.requires/ 55op-power-start@0.service 56op-wait-power-on@0.service 57``` 58So basically, if you run `systemctl start obmc-host-start@0.target` then 59systemd will start execution of all associated services. 60 61The services have dependencies within them that control the execution of each 62service (for example, the op-power-start.service will run prior to the 63op-wait-power-on.service). These dependencies are set using targets and the 64Wants,Before,After keywords. 65 66## Server Power Off (Soft) 67The soft server power off function is encapsulated in the 68obmc-host-shutdown@.target. This target is soft in that it notifies the host 69of the power off request and gives it a certain amount of time to shut itself 70down. 71 72## Server Power Off (Hard) 73The hard server power off is encapsulated in the 74obmc-chassis-hard-poweroff@.target. This target will force the stopping 75of the soft power off service if running, and immediately cut power to the 76system. 77 78## Server Reboot 79The reboot of the server is encapsulated in the obmc-host-reboot@.target. 80This target will utilize the soft power off target and then, once that 81completes, start the host power on target. 82 83## Systemd Control in OpenBMC 84There are a collection of services within OpenBMC that interact with systemd and 85its unit files, providing somewhat of an abstraction layer from the user of the 86OpenBMC system and systemd. See the [state](https://github.com/openbmc/phosphor-dbus-interfaces/tree/master/xyz/openbmc_project/State) 87interfaces for more information on this function. 88 89For example, if you wanted to execute the server power on function, you would do 90the following: 91 92> busctl set-property xyz.openbmc_project.State.Host /xyz/openbmc_project/state/host0 93xyz.openbmc_project.State.Host RequestedHostTransition s 94xyz.openbmc_project.State.Host.Transition.On 95 96Underneath the covers, this is calling systemd with the server power on target. 97 98## Systemd Services or Monitoring Applications 99A common question when creating new OpenBMC applications which need to 100execute some logic in the context of systemd targets is whether they should 101be triggered by systemd services or by monitoring for the appropriate 102D-Bus signal indicating the start/stop of the target they are interested in. 103 104The basic guidelines for when to create a systemd service are the following: 105- If your application logic depends on other systemd based services then 106 make it a systemd service and utilize the Wants/After/Before service 107 capabilities. 108- If other applications depend on your application logic then it should be a 109 systemd service. 110- If your application failing during the target start could impact targets or 111 services that run after it, then it should be a systemd service. This ensures 112 dependent targets are not started if your application fails. 113 114## Error Handling of Systemd 115With great numbers of targets and services, come great chances for failures. 116To make OpenBMC a robust and productive system, it needs to be sure to have an 117error handling policy for when services and their targets fail. 118 119When a failure occurs, the OpenBMC software needs to notify the users of the 120system and provide mechanisms for either the system to automatically retry the 121failed operation (i.e. reboot the system) or to stay in a quiesced state so that 122error data can be collected and the failure can be investigated. 123 124There are two main failure scenarios when it comes to OpenBMC and systemd usage: 125 1261. A service within a target fails 127- If the service is a "oneshot" type, and the service is required 128(not wanted) by the target then the target will fail if the service 129fails 130 - Define a behavior for when the target fails using the 131 "OnFailure" option (i.e. go to a new failure target if any required 132 service fails) 133- If the service is not a "oneshot", then it can not fail the target 134(the target only knows that it started successfully) 135 - Define a behavior for when the service fails (OnFailure) 136 option. 137 - The service can not have "RemainAfterExit=yes" otherwise, the OnFailure 138 action does not occur until the service is stopped (instead of when it 139 fails) 140 - *See more information below on [RemainAfterExit](#RemainAfterExit) 141 1422. A failure outside of a normal systemd target/service (host watchdog expires, 143host checkstop detected) 144- The service which detects this failure is responsible for logging the 145appropriate error, and instructing systemd to go to the appropriate target 146 147Within OpenBMC, there is a host quiesce target. This is the target that other 148host related targets should go to when they hit a failure. Other software within 149OpenBMC can then monitor for the entry into this quiesce target and will handle 150the halt vs. automatic reboot functionality. 151 152Targets which are not host related, will need special thought in regards to 153their error handling. For example, the target responsible for applying chassis 154power, obmc-chassis-poweron@0.target, will have a 155"OnFailure=obmc-chassis-poweroff@%i.target" error path. That is, if the 156chassis power on target fails then power off the chassis. 157 158The above info sets up some general **guidelines** for our host related 159targets and services: 160 161- All targets should have an "OnFailure=obmc-quiesce-host@.target" 162- All services which are required for a target to achieve its function should 163be RequiredBy that target (not WantedBy) 164- All services should first try to be "Type=oneshot" so that we can just rely on 165the target failure path 166- If a service can not be "Type=oneshot", then it needs to have a 167"OnFailure=obmc-quiesce-host@.target" and ideally set "RemainAfterExit=no" 168(but see caveats on this below) 169- If a service can not be any of these then it's up to the service application 170to call systemd with the obmc-quiesce-host@.target on failures 171 172### RemainAfterExit 173This is set to "yes" for most OpenBMC services to handle the situation where 174someone starts the same target twice. If the associated service with that 175target is not running (i.e. RemainAfterExit=no), then the service will be 176executed again. Think about someone accidentally running the 177obmc-chassis-poweron@.target twice. If you execute it when the operating system 178is up and running, and the service which toggles the pgood pin is re-executed, 179you're going to crash your system. Given this info, the goal should always be 180to write "oneshot" services that have RemainAfterExit set to yes. 181 182## Target and Service Dependency Details 183There are some tools available out there to visualize systemd service and 184target dependencies (systemd-analyze) but due to the complexity of our design, 185they do not generate anything very useful. 186 187For now, document the current dependencies on a witherspoon system for 188reference. 189 190``` 191R = Requires 192W = Wants 193A = After 194B = Before 195S = Start (runs a command to start another target or service) 196(S) = Synchronization Target 197``` 198 199### Soft Power Off 200``` 201obmc-host-shutdown.target 202 R: xyz.openbmc_project.Ipmi.Internal.SoftPowerOff.service 203 W: obmc-host-stopping.target (S) 204 B: obmc-host-stopping.target (S) 205 R: obmc-chassis-poweroff.target 206 R: obmc-host-stop.target 207 R: op-occ-disable.service 208 B: obmc-host-stop-pre.target 209 R: op-power-stop.service 210 W: obmc-power-stop.target (S) 211 B: obmc-power-stop.target (S) 212 W: obmc-power-stop-pre.target (S) 213 A: obmc-power-stop-pre.target (S) 214 W: mapper-wait@-org-openbmc-control-power.service 215 A: mapper-wait@-org-openbmc-control-power.service 216 R: op-wait-power-off.service 217 B: obmc-power-off.target (S) 218 W: obmc-power-stop.target (S) 219 B: obmc-power-stop.target (S) 220 W: obmc-power-stop-pre.target (S) 221 A: obmc-power-stop-pre.target (S) 222 W: mapper-wait@-org-openbmc-control-power.service 223 A: mapper-wait@-org-openbmc-control-power.service 224 R: op-powered-off.service 225 A: op-wait-power-off.service 226 R: op-wait-power-off.service 227 S: obmc-chassis-powered-off.target 228 W: pcie-poweroff.service 229 B: op-power-stop.service 230 A: obmc-power-stop-pre@.target 231``` 232 233#### Synchronization Target Dependencies 234``` 235obmc-power-stop.target 236 W: obmc-power-stop-pre.target 237 A: obmc-power-stop-pre.target 238 239obmc-power-stop-pre.target 240 W: obmc-host-stopped.target 241 A: obmc-host-stopped.target 242 243obmc-host-stopped.target 244 W: obmc-host-stopping.target 245 A: obmc-host-stopping.target 246 B: obmc-power-stop-pre.target 247 248obmc-host-stopping.target 249 W: obmc-host-stop-pre.target 250 A: obmc-host-stop-pre.target 251 B: obmc-host-stopped.target 252 253obmc-host-stop-pre.target 254 B: obmc-host-stopping.target 255``` 256