1# phosphor-pid-control 2 3## Objective 4 5Develop a tray level fan control system that will use exhaust temperature and 6other machine temperature information to control fan speeds in order to keep 7machines within acceptable operating conditions. 8 9Effectively porting the Chromium EC thermal code to run on the BMC and use the 10OpenBMC dbus namespace and IPMI commands. 11 12## Background 13 14Recent server systems come with a general secondary processing system attached 15for the purpose of monitoring and control, generally referred to as a BMC[^2]. 16There is a large effort to develop an open source framework for writing 17applications and control systems that will run on the BMC, known as 18OpenBMC[^3]<sup>,</sup>[^4]. Within Google the effort has been internalized 19(while also providing upstream pushes) as gBMC[^5]. The primary goal of OpenBMC 20is to provide support for remote and local system management through a REST 21interface, and also through IPMI[^6] tied together via the system dbus. OpenBMC 22provides many applications and daemons that we can leverage and improve. 23 24The BMC is wired such that it has direct access and control over many 25motherboard components, including fans and temperature sensors[^7]. Therefore, 26it is an ideal location to run a thermal control loop, similar to the EC. 27However, to upstream it will need to follow (as best as possible) the OpenBMC 28specifications for communicating and executing[^8]. 29 30IPMI allows for OEM commands to provide custom information flow or system 31control with a BMC. OEM commands are already lined up for certain other accesses 32routed through the BMC, and can be upstreamed for others to use. 33 34## Overview 35 36The BMC will run a daemon that controls the fans by pre-defined zones. The 37application will use thermal control, such that each defined zone is kept within 38a range and adjusted based on thermal information provided from locally readable 39sensors as well as host-provided information over an IPMI OEM command. 40 41A system (or tray) will be broken out into one or more zones, specified via 42configuration files or dbus. Each zone will contain at least one fan and at 43least one temperature sensor and some device margins. The sensor data can be 44provided via sysfs, dbus, or through IPMI. In either case, default margins 45should be provided in case of failure or other unknown situation. 46 47The system will run a control loop for each zone with the attempt to maintain 48the temperature within that zone within the margin for the devices specified. 49 50## Detailed Design 51 52The software will run as a multi-threaded daemon that runs a control loop for 53each zone, and has a master thread which listens for dbus messages. Each zone 54will require at least one fan that it exclusively controls, however, zones can 55share temperature sensors. 56 57 58 59In this figure the communications channels between swampd and ipmid and 60phosphor-hwmon are laid out. 61 62### OpenBMC Upstream 63 64To be upstreamed to OpenBMC for use on open-power systems, we need to follow the 65OpenBMC code style specification[^9] and leverage the dbus framework for reading 66sensors and fan control[^10]. 67 68There is already a daemon, which given a configuration file for a hwmon device, 69will add it to the dbus objects namespace which handles queries for values such 70a temperature or fan speed and allows another process to control the fan 71speed[^11]. It is the goal to utilize this other daemon through dbus to read the 72onboard sensors and control the fans. 73 74Because of the present implementation of the dbus interfaces to require 75controlling the fans only via specifying the RPM target, whereas the driver 76we're using for Quanta-Q71l (the first system) only allows writing PWM. This can 77be controlled either directly or via dbus. 78 79### Zone Specification 80 81A configuration file will need to exist for each board, likely in YAML[^12]. 82Similar information will also be necessary for gsys, such that it knows what 83sensors to read and send to the BMC. Presently it does something similar with 84EC, so it shouldn't be unreasonable to do something similar. 85 86Each zone must have at least one fan that it exclusively controls. Each zone 87must have at least one temperature sensor, but they may be shared. 88 89The external devices specified in the zone must have default information as a 90fallback, while their current temperatures will be provided by gsys. Some 91devices adapt quickly and others slowly, and this distinction will need to be a 92factor and described in the configuration. 93 94The internal thermometers specified will be read via sysfs. 95 96#### A proposed configuration file: 97 98``` 99{ZONEID}: 100 {PIDID}: 101 type: "fan" | "margin" 102 ipmi: 103 {IPMI_ID} 104 name: "fan1" 105 readPath: "/xyz/openbmc_project/sensors/fan_tach/fan1" 106 writePath: "/sys/class/hwmon/hwmon0/pwm0" 107 pidinfo: 108 samplerate: 0.1 // sample time in seconds 109 proportionalCoeff: 0.01 // coefficient for proportional 110 integralCoeff: 0.001 // coefficient for integral 111 integral_limit: 112 min: 0 113 max: 100 114 output_limit: 115 min: 0 116 max: 100 117 slewNegative: 0 118 slewPositive: 0 119 {PIDID}: 120 type: "margin" 121 ipmi: 122 {IPMI_ID} 123 name: "sluggish0" 124 readPath: "/xyz/openbmc_project/sensors/external/sluggish0" 125 writePath: "" 126 pidinfo: 127 samplerate: 1 // sample time in seconds 128 proportionalCoeff: 94.0 129 integralCoeff: 2.0 130 integral_limit: 131 min: 3000 132 max: 10000 133 output_limit: 134 min: 3000 135 max: 10000 136 slewNegative: 0 137 slewPositive: 0 138``` 139 140### Chassis Delta 141 142Due to data center requirements, the delta between the outgoing air temperature 143and the environmental air temperature must be no greater than 15C. 144 145### IPMI Access to Phosphor-pid-control 146 147[OEM-IPMI Definitions](ipmid.md) 148 149#### Set Sensor Value 150 151Tools needs to update the thermal controller with information not necessarily 152available to the BMC. This will comprise of a list of temperature (or margin?) 153sensors that are updated by the set sensor command. Because they don't represent 154real sensors in the system, the set sensor handler can simply broadcast the 155update as a properties update on dbus when it receives the command over IPMI. 156 157#### Set Fan PWM 158 159A tool can override a specific fan's PWM when we implement the set sensor IPMI 160command pathway. 161 162#### Get Fan Tach 163 164A tool can read fan_tach through the normal IPMI interface presently exported 165for sensors. 166 167### Sensor Update Loop 168 169The plan is to listen for fan_tach updates for each fan in a background thread. 170This will receive an update from phosphor-hwmon each time it updates any sensor 171it cares about. 172 173By default phosphor-hwmon reads each sensor in turn and then sleeps for 1 174second. We'll be updating phosphor-hwmon to sleep for a shorter period -- how 175short though is still TBD. We'll also be updating phosphor-hwmon to support pwm 176as a target. 177 178### Thermal Control Loops 179 180Each zone will require a control loop that monitors the associated thermals and 181controls the fan(s). The EC PID loop is designed to hit the fans 10 times per 182second to drive them to the desired value and read the sensors once per second. 183We'll be receiving sensor updates with such regularly, however, at present it 184takes ~0.13s to read all 8 fans. Which can't be read constantly without bringing 185the system to its knees -- in that all CPU cycles would be spent reading the 186fans. TBD on how frequently we'll be reading the fan sensors and the impact this 187will have. 188 189### Main Thread 190 191The main thread will manage the other threads, and process the initial 192configuration files. It will also register a dbus handler for the OEM message. 193 194### Enabling Logging & Tuning 195 196By default, swampd won't log information. To enable logging pass "-l" on the 197command line with a parameter that is the folder into which to write the logs. 198 199The log files will be named {folderpath}/zone_{zoneid}.log 200 201To enable tuning, pass "-t" on the command line. 202 203See [Logging & Tuning](tuning.md) for more information. 204 205## Project Information 206 207This project is designed to be a daemon running within the OpenBMC environment. 208It will use a well-defined configuration file to control the temperature of the 209tray components to keep them within operating conditions. It will require 210coordinate with host-side tooling and OpenBMC. Providing a host-side service 211upstream to talk to the BMC is beyond the scope of this project. 212 213## Security Considerations 214 215A rogue client on the host could send invalid thermal information causing 216physical damage to the system. There will be an effort to sanity check all input 217from a host-tool to alleviate this concern. 218 219## Privacy Considerations 220 221This device holds no user data, however, you could profile the types of jobs 222executed on the server by watching its temperatures. 223 224## Testing Plan 225 226Testing individual code logic will be handled through unit-tests, however some 227pieces of code rely on abstractions such that we can swap out dbus with 228something encapsulated such that testing can be done without strictly running on 229a real system. 230 231Testing the system on real hardware will be performed to verify: 232 2331. The fallback values are used when the host isn't reporting. 2341. The system behaves as expected given the information it reads. 235 236Unit-tests will provide that we know it validates information from the host 237properly as well as handles difficult to reproduce edge cases. 238 239The testing of this project on real hardware can likely fold into the general 240gBMC testing planned. 241 242## Code Notes 243 244Swampd's primary function is to drive the fans of a system given various inputs. 245 246### Layout 247 248The code is broken out into modules as follows: 249 250* `dbus` - Any read or write interface that uses dbus primarily. 251* `experiments` - Small execution paths that allow for fan examination 252 including how quickly fans respond to changes. 253* `ipmi` - Manual control for any zone is handled by receiving an IPMI 254 message. This holds the ipmid provider for receiving those messages and 255 sending them onto swampd. 256* `notimpl` - These are read-only and write-only interface implementations 257 that can be dropped into a pluggable sensor to make it complete. 258* `pid` - This contains all the PID associated code, including the zone 259 definition, controller definition, and the PID computational code. 260* `scripts` - This contains the scripts that convert YAML into C++. 261* `sensors` - This contains a couple of sensor types including the pluggable 262 sensor's definition. It also holds the sensor manager. 263* `sysfs` - This contains code that reads from or writes to sysfs. 264* `threads` - Most of swampd's threads run in this method where there's just a 265 dbus bus that we manage. 266 267## Example System Configurations 268 269### Two Margin Sensors Into Three Fans (Non-Step PID) 270 271``` 272A single zone system where multiple margin thermal sensors are fed into one PID 273that generates the output RPM for a set of fans controlled by one PID. 274 275margin sensors as input to thermal pid 276 277fleeting0+---->+-------+ +-------+ Thermal PID sampled 278 | min()+--->+ PID | slower rate. 279fleeting1+---->+-------+ +---+---+ 280 | 281 | 282 | RPM setpoint 283 Current RPM v 284 +--+-----+ 285 The Fan PID fan0+---> | New PWM +-->fan0 286 samples at a | | | 287 faster rate fan1+---> PID +---------->--->fan1 288 speeding up the | | | 289 fans. fan2+---> | +-->fan2 290 ^ +--------+ + 291 | | 292 +-------------------------------+ 293 RPM updated by PWM. 294``` 295 296## Notes 297 298[^2]: BMC - Board Management Controller 299[^3]: with url https://github.com/openbmc/openbmc 300[^4]: with url https://github.com/facebook/openbmc 301[^5]: with url http://go/gbmc 302[^6]: with url 303 http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html 304[^7]: Excluding temperature sensors on PCIe cards and other add-ons. 305[^8]: They prefer c++. 306[^9]: With url 307 https://github.com/openbmc/docs/blob/master/cpp-style-and-conventions.md 308[^10]: with url https://github.com/openbmc/phosphor-dbus-interfaces 309[^11]: with url https://github.com/openbmc/phosphor-hwmon 310[^12]: YAML appears to be the configuration language of choice for OpenBMC. 311
