1# phosphor-pid-control 2 3[TOC] 4 5## Objective 6 7Develop a tray level fan control system that will use exhaust temperature and 8other machine temperature information to control fan speeds in order to keep 9machines within acceptable operating conditions. 10 11Effectively porting the Chromium EC thermal code to run on the BMC and use the 12OpenBMC dbus namespace and IPMI commands. 13 14## Background 15 16Recent server systems come with a general secondary processing system attached 17for the purpose of monitoring and control, generally referred to as a BMC[^2]. 18There is a large effort to develop an open source framework for writing 19applications and control systems that will run on the BMC, known as 20OpenBMC[^3]<sup>,</sup>[^4]. Within Google the effort has been internalized 21(while also providing upstream pushes) as gBMC[^5]. The primary goal of OpenBMC 22is to provide support for remote and local system management through a REST 23interface, and also through IPMI[^6] tied together via the system dbus. OpenBMC 24provides many applications and daemons that we can leverage and improve. 25 26The BMC is wired such that it has direct access and control over many 27motherboard components, including fans and temperature sensors[^7]. Therefore, 28it is an ideal location to run a thermal control loop, similar to the EC. 29However, to upstream it will need to follow (as best as possible) the OpenBMC 30specifications for communicating and executing[^8]. 31 32IPMI allows for OEM commands to provide custom information flow or system 33control with a BMC. OEM commands are already lined up for certain other accesses 34routed through the BMC, and can be upstreamed for others to use. 35 36## Overview 37 38The BMC will run a daemon that controls the fans by pre-defined zones. The 39application will use thermal control, such that each defined zone is kept 40within a range and adjusted based on thermal information provided from locally 41readable sensors as well as host-provided information over an IPMI OEM 42command. 43 44A system (or tray) will be broken out into one or more zones, specified via 45configuration files or dbus. Each zone will contain at least one fan and at 46least one temperature sensor and some device margins. The sensor data can 47be provided via sysfs, dbus, or through IPMI. In either case, default margins 48should be provided in case of failure or other unknown situation. 49 50The system will run a control loop for each zone with the attempt to maintain 51the temperature within that zone within the margin for the devices specified. 52 53## Detailed Design 54 55The software will run as a multi-threaded daemon that runs a control loop for 56each zone, and has a master thread which listens for dbus messages. Each zone 57will require at least one fan that it exclusively controls, however, zones can 58 share temperature sensors. 59 60 61 62In this figure the communications channels between swampd and ipmid and 63phosphor-hwmon are laid out. 64 65### OpenBMC Upstream 66 67To be upstreamed to OpenBMC for use on open-power systems, we need to follow the 68OpenBMC code style specification[^9] and leverage the dbus framework for reading 69sensors and fan control[^10]. 70 71There is already a daemon, which given a configuration file for a hwmon device, 72will add it to the dbus objects namespace which handles queries for values such 73a temperature or fan speed and allows another process to control the fan 74speed[^11]. It is the goal to utilize this other daemon through dbus to read the 75onboard sensors and control the fans. 76 77Because of the present implementation of the dbus interfaces to require 78controlling the fans only via specifying the RPM target, whereas the driver 79we're using for Quanta-Q71l (the first system) only allows writing PWM. This 80can be controlled either directly or via dbus. 81 82### Zone Specification 83 84A configuration file will need to exist for each board, likely in YAML[^12]. 85Similar information will also be necessary for gsys, such that it knows what 86sensors to read and send to the BMC. Presently it does something similar with 87EC, so it shouldn't be unreasonable to do something similar. 88 89Each zone must have at least one fan that it exclusively controls. Each zone 90must have at least one temperature sensor, but they may be shared. 91 92The external devices specified in the zone must have default information as a 93fallback, while their current temperatures will be provided by gsys. Some 94devices adapt quickly and others slowly, and this distinction will need to be a 95factor and described in the configuration. 96 97The internal thermometers specified will be read via sysfs. 98 99#### A proposed configuration file: 100 101``` 102{ZONEID}: 103 {PIDID}: 104 type: "fan" | "margin" 105 ipmi: 106 {IPMI_ID} 107 name: "fan1" 108 readpath: "/xyz/openbmc_project/sensors/fan_tach/fan1" 109 writepath: "/sys/class/hwmon/hwmon0/pwm0" 110 pidinfo: 111 samplerate: 0.1 // sample time in seconds 112 p_coeff: 0.01 // coefficient for proportional 113 i_coeff: 0.001 // coefficient for integral 114 integral_limit: 115 min: 0 116 max: 100 117 output_limit: 118 min: 0 119 max: 100 120 slew_negative: 0 121 slew_positive: 0 122 {PIDID}: 123 type: "margin" 124 ipmi: 125 {IPMI_ID} 126 name: "sluggish0" 127 readpath: "/xyz/openbmc_project/sensors/external/sluggish0" 128 writepath: "" 129 pidinfo: 130 samplerate: 1 // sample time in seconds 131 p_coeff: 94.0 132 i_coeff: 2.0 133 integral_limit: 134 min: 3000 135 max: 10000 136 output_limit: 137 min: 3000 138 max: 10000 139 slew_negative: 0 140 slew_positive: 0 141``` 142 143### Chassis Delta 144 145Due to data center requirements, the delta between the outgoing air temperature 146and the environmental air temperature must be no greater than 15C. 147 148### IPMI Command Specification 149 150Gsys needs the ability to send to the BMC, the margin information on the devices 151that it knows how to read that the BMC cannot. There is no command in IPMI that 152currently supports this use-case, therefore it will be added as an OEM command. 153 154The state of the BMC readable temperature sensors can be read through normal 155IPMI commands and is already supported. 156 157#### OEM Set Control 158 159Gsys needs to be able to set the control of the thermal system to either 160automatic or manual. When manual, the daemon will effectively wait to be told to 161be put back in automatic mode. It is expected in this manual mode that something 162will be controlling the fans via the other commands. 163 164Manual mode is controlled by zone through the following OEM command: 165 166##### Request 167 168Byte | Purpose | Value 169---- | ------------ | ----------------------------------------------------- 170`00` | `netfn` | `0x2e` 171`01` | `command` | `0x04 (also using manual command)` 172`02` | `oem1` | `0xcf` 173`03` | `oem2` | `0xc2` 174`04` | `padding` | `0x00` 175`05` | `SubCommand` | `Get or Set. Get == 0, Set == 1` 176`06` | `ZoneId` | 177`07` | `Mode` | `If Set, Value 1 == Manual Mode, 0 == Automatic Mode` 178 179##### Response 180 181Byte | Purpose | Value 182---- | --------- | ----------------------------------------------------- 183`02` | `oem1` | `0xcf` 184`03` | `oem2` | `0xc2` 185`04` | `padding` | `0x00` 186`07` | `Mode` | `If Set, Value 1 == Manual Mode, 0 == Automatic Mode` 187 188#### OEM Get Failsafe Mode 189 190Gbmctool needs to be able to read back whether a zone is in failsafe mode. This 191setting is read-only because it's dynamically determined within Swampd per zone. 192 193Byte | Purpose | Value 194---- | ------------ | ---------------------------------- 195`00` | `netfn` | `0x2e` 196`01` | `command` | `0x04 (also using manual command)` 197`02` | `oem1` | `0xcf` 198`03` | `oem2` | `0xc2` 199`04` | `padding` | `0x00` 200`05` | `SubCommand` | `Get == 2` 201`06` | `ZoneId` | 202 203##### Response 204 205Byte | Purpose | Value 206---- | ---------- | ----------------------------------------------- 207`02` | `oem1` | `0xcf` 208`03` | `oem2` | `0xc2` 209`04` | `padding` | `0x00` 210`07` | `failsafe` | `1 == in Failsafe Mode, 0 not in failsafe mode` 211 212#### Set Sensor Value 213 214Gsys needs to update the thermal controller with information not necessarily 215available to the BMC. This will comprise of a list of temperature (or margin?) 216sensors that are updated by the set sensor command. Because they don't represent 217real sensors in the system, the set sensor handler can simply broadcast the 218update as a properties update on dbus when it receives the command over IPMI. 219 220#### Set Fan PWM 221 222Gsys can override a specific fan's PWM when we implement the set sensor IPMI 223command pathway. 224 225#### Get Fan Tach 226 227Gsys can read fan_tach through the normal IPMI interface presently exported for 228sensors. 229 230### Sensor Update Loop 231 232The plan is to listen for fan_tach updates for each fan in a background thread. 233This will receive an update from phosphor-hwmon each time it updates any sensor 234it cares about. 235 236By default phosphor-hwmon reads each sensor in turn and then sleeps for 1 237second. We'll be updating phosphor-hwmon to sleep for a shorter period -- how 238short though is still TBD. We'll also be updating phosphor-hwmon to support pwm 239as a target. 240 241### Thermal Control Loops 242 243Each zone will require a control loop that monitors the associated thermals and 244controls the fan(s). The EC PID loop is designed to hit the fans 10 times per 245second to drive them to the desired value and read the sensors once per second. 246We'll be receiving sensor updates with such regularly, however, at present it 247takes ~0.13s to read all 8 fans. Which can't be read constantly without bringing 248the system to its knees -- in that all CPU cycles would be spent reading the 249fans. TBD on how frequently we'll be reading the fan sensors and the impact this 250will have. 251 252### Main Thread 253 254The main thread will manage the other threads, and process the initial 255configuration files. It will also register a dbus handler for the OEM message. 256 257### Enabling Logging 258 259By default, swampd isn't compiled to log information. To compile it for tuning, 260you'll need to add: 261 262``` 263EXTRA_OEMAKE_append_YOUR_MACHINE = " CXXFLAGS='${CXXFLAGS} -D__TUNING_LOGGING__'" 264``` 265 266To the recipe. 267 268## Project Information 269 270This project is designed to be a daemon running within the OpenBMC environment. 271It will use a well-defined configuration file to control the temperature of the 272tray components to keep them within operating conditions. It will require 273coordinate with gsys and OpenBMC. Providing a host-side service upstream to talk 274to the BMC is beyond the scope of this project. 275 276## Security Considerations 277 278A rogue client on the host could send invalid thermal information causing 279physical damage to the system. There will be an effort to sanity check all input 280from gsys to alleviate this concern. 281 282## Privacy Considerations 283 284This device holds no user data, however, you could profile the types of jobs 285executed on the server by watching its temperatures. 286 287## Testing Plan 288 289Testing individual code logic will be handled through unit-tests, however some 290pieces of code rely on abstractions such that we can swap out dbus with 291something encapsulated such that testing can be done without strictly running on 292a real system. 293 294Testing the system on real hardware will be performed to verify: 295 2961. The fallback values are used when gsys isn't reporting. 2971. The system behaves as expected given the information it reads. 298 299Unit-tests will provide that we know it validates information from gsys properly 300as well as handles difficult to reproduce edge cases. 301 302The testing of this project on real hardware can likely fold into the general 303gBMC testing planned. 304 305## Notes 306 307[^2]: BMC - Board Management Controller 308[^3]: with url https://github.com/openbmc/openbmc 309[^4]: with url https://github.com/facebook/openbmc 310[^5]: with url http://go/gbmc 311[^6]: with url 312 http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html 313[^7]: Excluding temperature sensors on PCIe cards and other add-ons. 314[^8]: They prefer c++. 315[^9]: With url 316 https://github.com/openbmc/docs/blob/master/cpp-style-and-conventions.md 317[^10]: with url https://github.com/openbmc/phosphor-dbus-interfaces 318[^11]: with url https://github.com/openbmc/phosphor-hwmon 319[^12]: YAML appears to be the configuration language of choice for OpenBMC. 320
