1 # phosphor-pid-control 2 3 [TOC] 4 5 ## Objective 6 7 Develop a tray level fan control system that will use exhaust temperature and 8 other machine temperature information to control fan speeds in order to keep 9 machines within acceptable operating conditions. 10 11 Effectively porting the Chromium EC thermal code to run on the BMC and use the 12 OpenBMC dbus namespace and IPMI commands. 13 14 ## Background 15 16 Recent server systems come with a general secondary processing system attached 17 for the purpose of monitoring and control, generally referred to as a BMC[^2]. 18 There is a large effort to develop an open source framework for writing 19 applications and control systems that will run on the BMC, known as 20 OpenBMC[^3]<sup>,</sup>[^4]. Within Google the effort has been internalized 21 (while also providing upstream pushes) as gBMC[^5]. The primary goal of OpenBMC 22 is to provide support for remote and local system management through a REST 23 interface, and also through IPMI[^6] tied together via the system dbus. OpenBMC 24 provides many applications and daemons that we can leverage and improve. 25 26 The BMC is wired such that it has direct access and control over many 27 motherboard components, including fans and temperature sensors[^7]. Therefore, 28 it is an ideal location to run a thermal control loop, similar to the EC. 29 However, to upstream it will need to follow (as best as possible) the OpenBMC 30 specifications for communicating and executing[^8]. 31 32 IPMI allows for OEM commands to provide custom information flow or system 33 control with a BMC. OEM commands are already lined up for certain other accesses 34 routed through the BMC, and can be upstreamed for others to use. 35 36 ## Overview 37 38 The BMC will run a daemon that controls the fans by pre-defined zones. The 39 application will use thermal control, such that each defined zone is kept 40 within a range and adjusted based on thermal information provided from locally 41 readable sensors as well as host-provided information over an IPMI OEM 42 command. 43 44 A system (or tray) will be broken out into one or more zones, specified via 45 configuration files or dbus. Each zone will contain at least one fan and at 46 least one temperature sensor and some device margins. The sensor data can 47 be provided via sysfs, dbus, or through IPMI. In either case, default margins 48 should be provided in case of failure or other unknown situation. 49 50 The system will run a control loop for each zone with the attempt to maintain 51 the temperature within that zone within the margin for the devices specified. 52 53 ## Detailed Design 54 55 The software will run as a multi-threaded daemon that runs a control loop for 56 each zone, and has a master thread which listens for dbus messages. Each zone 57 will require at least one fan that it exclusively controls, however, zones can 58 share temperature sensors. 59 60  61 62 In this figure the communications channels between swampd and ipmid and 63 phosphor-hwmon are laid out. 64 65 ### OpenBMC Upstream 66 67 To be upstreamed to OpenBMC for use on open-power systems, we need to follow the 68 OpenBMC code style specification[^9] and leverage the dbus framework for reading 69 sensors and fan control[^10]. 70 71 There is already a daemon, which given a configuration file for a hwmon device, 72 will add it to the dbus objects namespace which handles queries for values such 73 a temperature or fan speed and allows another process to control the fan 74 speed[^11]. It is the goal to utilize this other daemon through dbus to read the 75 onboard sensors and control the fans. 76 77 Because of the present implementation of the dbus interfaces to require 78 controlling the fans only via specifying the RPM target, whereas the driver 79 we're using for Quanta-Q71l (the first system) only allows writing PWM. This 80 can be controlled either directly or via dbus. 81 82 ### Zone Specification 83 84 A configuration file will need to exist for each board, likely in YAML[^12]. 85 Similar information will also be necessary for gsys, such that it knows what 86 sensors to read and send to the BMC. Presently it does something similar with 87 EC, so it shouldn't be unreasonable to do something similar. 88 89 Each zone must have at least one fan that it exclusively controls. Each zone 90 must have at least one temperature sensor, but they may be shared. 91 92 The external devices specified in the zone must have default information as a 93 fallback, while their current temperatures will be provided by gsys. Some 94 devices adapt quickly and others slowly, and this distinction will need to be a 95 factor and described in the configuration. 96 97 The internal thermometers specified will be read via sysfs. 98 99 #### A proposed configuration file: 100 101 ``` 102 {ZONEID}: 103 {PIDID}: 104 type: "fan" | "margin" 105 ipmi: 106 {IPMI_ID} 107 name: "fan1" 108 readpath: "/xyz/openbmc_project/sensors/fan_tach/fan1" 109 writepath: "/sys/class/hwmon/hwmon0/pwm0" 110 pidinfo: 111 samplerate: 0.1 // sample time in seconds 112 p_coeff: 0.01 // coefficient for proportional 113 i_coeff: 0.001 // coefficient for integral 114 integral_limit: 115 min: 0 116 max: 100 117 output_limit: 118 min: 0 119 max: 100 120 slew_negative: 0 121 slew_positive: 0 122 {PIDID}: 123 type: "margin" 124 ipmi: 125 {IPMI_ID} 126 name: "sluggish0" 127 readpath: "/xyz/openbmc_project/sensors/external/sluggish0" 128 writepath: "" 129 pidinfo: 130 samplerate: 1 // sample time in seconds 131 p_coeff: 94.0 132 i_coeff: 2.0 133 integral_limit: 134 min: 3000 135 max: 10000 136 output_limit: 137 min: 3000 138 max: 10000 139 slew_negative: 0 140 slew_positive: 0 141 ``` 142 143 ### Chassis Delta 144 145 Due to data center requirements, the delta between the outgoing air temperature 146 and the environmental air temperature must be no greater than 15C. 147 148 ### IPMI Command Specification 149 150 Gsys needs the ability to send to the BMC, the margin information on the devices 151 that it knows how to read that the BMC cannot. There is no command in IPMI that 152 currently supports this use-case, therefore it will be added as an OEM command. 153 154 The state of the BMC readable temperature sensors can be read through normal 155 IPMI commands and is already supported. 156 157 #### OEM Set Control 158 159 Gsys needs to be able to set the control of the thermal system to either 160 automatic or manual. When manual, the daemon will effectively wait to be told to 161 be put back in automatic mode. It is expected in this manual mode that something 162 will be controlling the fans via the other commands. 163 164 Manual mode is controlled by zone through the following OEM command: 165 166 ##### Request 167 168 Byte | Purpose | Value 169 ---- | ------------ | ----------------------------------------------------- 170 `00` | `netfn` | `0x2e` 171 `01` | `command` | `0x04 (also using manual command)` 172 `02` | `oem1` | `0xcf` 173 `03` | `oem2` | `0xc2` 174 `04` | `padding` | `0x00` 175 `05` | `SubCommand` | `Get or Set. Get == 0, Set == 1` 176 `06` | `ZoneId` | 177 `07` | `Mode` | `If Set, Value 1 == Manual Mode, 0 == Automatic Mode` 178 179 ##### Response 180 181 Byte | Purpose | Value 182 ---- | --------- | ----------------------------------------------------- 183 `02` | `oem1` | `0xcf` 184 `03` | `oem2` | `0xc2` 185 `04` | `padding` | `0x00` 186 `07` | `Mode` | `If Set, Value 1 == Manual Mode, 0 == Automatic Mode` 187 188 #### OEM Get Failsafe Mode 189 190 Gbmctool needs to be able to read back whether a zone is in failsafe mode. This 191 setting is read-only because it's dynamically determined within Swampd per zone. 192 193 Byte | Purpose | Value 194 ---- | ------------ | ---------------------------------- 195 `00` | `netfn` | `0x2e` 196 `01` | `command` | `0x04 (also using manual command)` 197 `02` | `oem1` | `0xcf` 198 `03` | `oem2` | `0xc2` 199 `04` | `padding` | `0x00` 200 `05` | `SubCommand` | `Get == 2` 201 `06` | `ZoneId` | 202 203 ##### Response 204 205 Byte | Purpose | Value 206 ---- | ---------- | ----------------------------------------------- 207 `02` | `oem1` | `0xcf` 208 `03` | `oem2` | `0xc2` 209 `04` | `padding` | `0x00` 210 `07` | `failsafe` | `1 == in Failsafe Mode, 0 not in failsafe mode` 211 212 #### Set Sensor Value 213 214 Gsys needs to update the thermal controller with information not necessarily 215 available to the BMC. This will comprise of a list of temperature (or margin?) 216 sensors that are updated by the set sensor command. Because they don't represent 217 real sensors in the system, the set sensor handler can simply broadcast the 218 update as a properties update on dbus when it receives the command over IPMI. 219 220 #### Set Fan PWM 221 222 Gsys can override a specific fan's PWM when we implement the set sensor IPMI 223 command pathway. 224 225 #### Get Fan Tach 226 227 Gsys can read fan_tach through the normal IPMI interface presently exported for 228 sensors. 229 230 ### Sensor Update Loop 231 232 The plan is to listen for fan_tach updates for each fan in a background thread. 233 This will receive an update from phosphor-hwmon each time it updates any sensor 234 it cares about. 235 236 By default phosphor-hwmon reads each sensor in turn and then sleeps for 1 237 second. We'll be updating phosphor-hwmon to sleep for a shorter period -- how 238 short though is still TBD. We'll also be updating phosphor-hwmon to support pwm 239 as a target. 240 241 ### Thermal Control Loops 242 243 Each zone will require a control loop that monitors the associated thermals and 244 controls the fan(s). The EC PID loop is designed to hit the fans 10 times per 245 second to drive them to the desired value and read the sensors once per second. 246 We'll be receiving sensor updates with such regularly, however, at present it 247 takes ~0.13s to read all 8 fans. Which can't be read constantly without bringing 248 the system to its knees -- in that all CPU cycles would be spent reading the 249 fans. TBD on how frequently we'll be reading the fan sensors and the impact this 250 will have. 251 252 ### Main Thread 253 254 The main thread will manage the other threads, and process the initial 255 configuration files. It will also register a dbus handler for the OEM message. 256 257 ### Enabling Logging 258 259 By default, swampd isn't compiled to log information. To compile it for tuning, 260 you'll need to add: 261 262 ``` 263 EXTRA_OEMAKE_append_YOUR_MACHINE = " CXXFLAGS='${CXXFLAGS} -D__TUNING_LOGGING__'" 264 ``` 265 266 To the recipe. 267 268 ## Project Information 269 270 This project is designed to be a daemon running within the OpenBMC environment. 271 It will use a well-defined configuration file to control the temperature of the 272 tray components to keep them within operating conditions. It will require 273 coordinate with gsys and OpenBMC. Providing a host-side service upstream to talk 274 to the BMC is beyond the scope of this project. 275 276 ## Security Considerations 277 278 A rogue client on the host could send invalid thermal information causing 279 physical damage to the system. There will be an effort to sanity check all input 280 from gsys to alleviate this concern. 281 282 ## Privacy Considerations 283 284 This device holds no user data, however, you could profile the types of jobs 285 executed on the server by watching its temperatures. 286 287 ## Testing Plan 288 289 Testing individual code logic will be handled through unit-tests, however some 290 pieces of code rely on abstractions such that we can swap out dbus with 291 something encapsulated such that testing can be done without strictly running on 292 a real system. 293 294 Testing the system on real hardware will be performed to verify: 295 296 1. The fallback values are used when gsys isn't reporting. 297 1. The system behaves as expected given the information it reads. 298 299 Unit-tests will provide that we know it validates information from gsys properly 300 as well as handles difficult to reproduce edge cases. 301 302 The testing of this project on real hardware can likely fold into the general 303 gBMC testing planned. 304 305 ## Notes 306 307 [^2]: BMC - Board Management Controller 308 [^3]: with url https://github.com/openbmc/openbmc 309 [^4]: with url https://github.com/facebook/openbmc 310 [^5]: with url http://go/gbmc 311 [^6]: with url 312 http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html 313 [^7]: Excluding temperature sensors on PCIe cards and other add-ons. 314 [^8]: They prefer c++. 315 [^9]: With url 316 https://github.com/openbmc/docs/blob/master/cpp-style-and-conventions.md 317 [^10]: with url https://github.com/openbmc/phosphor-dbus-interfaces 318 [^11]: with url https://github.com/openbmc/phosphor-hwmon 319 [^12]: YAML appears to be the configuration language of choice for OpenBMC. 320
