1a788899eSBrad Bishop# phosphor-pid-control 2cc74ae19SPatrick Venture 3d0aeed9bSPatrick VentureThis is a daemon running within the OpenBMC environment. It uses a well-defined 4d0aeed9bSPatrick Ventureconfiguration file to control the temperature of the tray components to keep 5d0aeed9bSPatrick Venturethem within operating conditions. It may require coordination with host-side 6d0aeed9bSPatrick Venturetooling and OpenBMC. 7cc74ae19SPatrick Venture 867efa244SPatrick Venture## Overview 9cc74ae19SPatrick Venture 10cc74ae19SPatrick VentureThe BMC will run a daemon that controls the fans by pre-defined zones. The 11c2aa9cf1SPatrick Ventureapplication will use thermal control, such that each defined zone is kept within 12c2aa9cf1SPatrick Venturea range and adjusted based on thermal information provided from locally readable 13c2aa9cf1SPatrick Venturesensors as well as host-provided information over an IPMI OEM command. 14cc74ae19SPatrick Venture 15cc74ae19SPatrick VentureA system (or tray) will be broken out into one or more zones, specified via 16cc74ae19SPatrick Ventureconfiguration files or dbus. Each zone will contain at least one fan and at 17c2aa9cf1SPatrick Ventureleast one temperature sensor and some device margins. The sensor data can be 18c2aa9cf1SPatrick Ventureprovided via sysfs, dbus, or through IPMI. In either case, default margins 19cc74ae19SPatrick Ventureshould be provided in case of failure or other unknown situation. 20cc74ae19SPatrick Venture 21cc74ae19SPatrick VentureThe system will run a control loop for each zone with the attempt to maintain 22cc74ae19SPatrick Venturethe temperature within that zone within the margin for the devices specified. 23cc74ae19SPatrick Venture 241dad21b9SPatrick Venture## Configuring 251dad21b9SPatrick Venture 261dad21b9SPatrick Venture[How to configure phosphor-pid-control](configure.md) 271dad21b9SPatrick Venture 2867efa244SPatrick Venture## Detailed Design 29cc74ae19SPatrick Venture 30cc74ae19SPatrick VentureThe software will run as a multi-threaded daemon that runs a control loop for 31cc74ae19SPatrick Ventureeach zone, and has a master thread which listens for dbus messages. Each zone 32cc74ae19SPatrick Venturewill require at least one fan that it exclusively controls, however, zones can 33cc74ae19SPatrick Ventureshare temperature sensors. 34cc74ae19SPatrick Venture 35cc74ae19SPatrick Venture 36cc74ae19SPatrick Venture 37cc74ae19SPatrick VentureIn this figure the communications channels between swampd and ipmid and 38cc74ae19SPatrick Venturephosphor-hwmon are laid out. 39cc74ae19SPatrick Venture 4067efa244SPatrick Venture### Zone Specification 41cc74ae19SPatrick Venture 42d0aeed9bSPatrick VentureA configuration file will need to exist for each board. 43cc74ae19SPatrick Venture 44cc74ae19SPatrick VentureEach zone must have at least one fan that it exclusively controls. Each zone 45cc74ae19SPatrick Venturemust have at least one temperature sensor, but they may be shared. 46cc74ae19SPatrick Venture 47d0aeed9bSPatrick VentureThe internal thermometers specified can be read via sysfs or dbus. 48cc74ae19SPatrick Venture 4967efa244SPatrick Venture### Chassis Delta 50cc74ae19SPatrick Venture 51cc74ae19SPatrick VentureDue to data center requirements, the delta between the outgoing air temperature 52cc74ae19SPatrick Ventureand the environmental air temperature must be no greater than 15C. 53cc74ae19SPatrick Venture 542192a6daSPatrick Venture### IPMI Access to Phosphor-pid-control 55cc74ae19SPatrick Venture 5602839e37SMichael Shen[OEM-IPMI Definitions](ipmi.md) 57cc74ae19SPatrick Venture 5867efa244SPatrick Venture#### Set Sensor Value 59cc74ae19SPatrick Venture 602192a6daSPatrick VentureTools needs to update the thermal controller with information not necessarily 61cc74ae19SPatrick Ventureavailable to the BMC. This will comprise of a list of temperature (or margin?) 62cc74ae19SPatrick Venturesensors that are updated by the set sensor command. Because they don't represent 63cc74ae19SPatrick Venturereal sensors in the system, the set sensor handler can simply broadcast the 64cc74ae19SPatrick Ventureupdate as a properties update on dbus when it receives the command over IPMI. 65cc74ae19SPatrick Venture 6667efa244SPatrick Venture#### Set Fan PWM 67cc74ae19SPatrick Venture 682192a6daSPatrick VentureA tool can override a specific fan's PWM when we implement the set sensor IPMI 69cc74ae19SPatrick Venturecommand pathway. 70cc74ae19SPatrick Venture 7167efa244SPatrick Venture#### Get Fan Tach 72cc74ae19SPatrick Venture 732192a6daSPatrick VentureA tool can read fan_tach through the normal IPMI interface presently exported 742192a6daSPatrick Venturefor sensors. 75cc74ae19SPatrick Venture 7667efa244SPatrick Venture### Sensor Update Loop 77cc74ae19SPatrick Venture 78cc74ae19SPatrick VentureThe plan is to listen for fan_tach updates for each fan in a background thread. 79cc74ae19SPatrick VentureThis will receive an update from phosphor-hwmon each time it updates any sensor 80cc74ae19SPatrick Ventureit cares about. 81cc74ae19SPatrick Venture 82cc74ae19SPatrick VentureBy default phosphor-hwmon reads each sensor in turn and then sleeps for 1 83cc74ae19SPatrick Venturesecond. We'll be updating phosphor-hwmon to sleep for a shorter period -- how 84cc74ae19SPatrick Ventureshort though is still TBD. We'll also be updating phosphor-hwmon to support pwm 85cc74ae19SPatrick Ventureas a target. 86cc74ae19SPatrick Venture 8767efa244SPatrick Venture### Thermal Control Loops 88cc74ae19SPatrick Venture 89cc74ae19SPatrick VentureEach zone will require a control loop that monitors the associated thermals and 90cc74ae19SPatrick Venturecontrols the fan(s). The EC PID loop is designed to hit the fans 10 times per 91cc74ae19SPatrick Venturesecond to drive them to the desired value and read the sensors once per second. 92cc74ae19SPatrick VentureWe'll be receiving sensor updates with such regularly, however, at present it 93cc74ae19SPatrick Venturetakes ~0.13s to read all 8 fans. Which can't be read constantly without bringing 94cc74ae19SPatrick Venturethe system to its knees -- in that all CPU cycles would be spent reading the 95cc74ae19SPatrick Venturefans. TBD on how frequently we'll be reading the fan sensors and the impact this 96cc74ae19SPatrick Venturewill have. 97cc74ae19SPatrick Venture 9867efa244SPatrick Venture### Main Thread 99cc74ae19SPatrick Venture 100cc74ae19SPatrick VentureThe main thread will manage the other threads, and process the initial 101cc74ae19SPatrick Ventureconfiguration files. It will also register a dbus handler for the OEM message. 102cc74ae19SPatrick Venture 103de79ee05SPatrick Venture### Enabling Logging & Tuning 104cc74ae19SPatrick Venture 105de79ee05SPatrick VentureBy default, swampd won't log information. To enable logging pass "-l" on the 106de79ee05SPatrick Venturecommand line with a parameter that is the folder into which to write the logs. 107de79ee05SPatrick Venture 108d0aeed9bSPatrick VentureThe log files will be named `{folderpath}/zone_{zoneid}.log`. 109de79ee05SPatrick Venture 110de79ee05SPatrick VentureTo enable tuning, pass "-t" on the command line. 111cc74ae19SPatrick Venture 112926411e1SKun YiSee [Logging & Tuning](tuning.md) for more information. 113926411e1SKun Yi 114d0aeed9bSPatrick Venture## Code Layout 1153652fce5SPatrick Venture 1163652fce5SPatrick VentureThe code is broken out into modules as follows: 1173652fce5SPatrick Venture 118217a827dSPatrick Williams- `dbus` - Any read or write interface that uses dbus primarily. 119217a827dSPatrick Williams- `experiments` - Small execution paths that allow for fan examination including 120217a827dSPatrick Williams how quickly fans respond to changes. 121217a827dSPatrick Williams- `ipmi` - Manual control for any zone is handled by receiving an IPMI message. 122217a827dSPatrick Williams This holds the ipmid provider for receiving those messages and sending them 123217a827dSPatrick Williams onto swampd. 124217a827dSPatrick Williams- `notimpl` - These are read-only and write-only interface implementations that 125217a827dSPatrick Williams can be dropped into a pluggable sensor to make it complete. 126217a827dSPatrick Williams- `pid` - This contains all the PID associated code, including the zone 1273652fce5SPatrick Venture definition, controller definition, and the PID computational code. 128217a827dSPatrick Williams- `scripts` - This contains the scripts that convert YAML into C++. 129217a827dSPatrick Williams- `sensors` - This contains a couple of sensor types including the pluggable 1303652fce5SPatrick Venture sensor's definition. It also holds the sensor manager. 131217a827dSPatrick Williams- `sysfs` - This contains code that reads from or writes to sysfs. 132217a827dSPatrick Williams- `threads` - Most of swampd's threads run in this method where there's just a 1333652fce5SPatrick Venture dbus bus that we manage. 1343652fce5SPatrick Venture 13535621ebeSPatrick Venture## Example System Configurations 1363652fce5SPatrick Venture 13735621ebeSPatrick Venture### Two Margin Sensors Into Three Fans (Non-Step PID) 1383652fce5SPatrick Venture 13935621ebeSPatrick VentureA single zone system where multiple margin thermal sensors are fed into one PID 14035621ebeSPatrick Venturethat generates the output RPM for a set of fans controlled by one PID. 1413652fce5SPatrick Venture 14235621ebeSPatrick Venturemargin sensors as input to thermal pid 1433652fce5SPatrick Venture 144*608b9391SGeorge Liu```text 14535621ebeSPatrick Venturefleeting0+---->+-------+ +-------+ Thermal PID sampled 14635621ebeSPatrick Venture | min()+--->+ PID | slower rate. 14735621ebeSPatrick Venturefleeting1+---->+-------+ +---+---+ 14835621ebeSPatrick Venture | 14935621ebeSPatrick Venture | 1507280e27eSPatrick Venture | RPM setpoint 15135621ebeSPatrick Venture Current RPM v 15235621ebeSPatrick Venture +--+-----+ 15335621ebeSPatrick Venture The Fan PID fan0+---> | New PWM +-->fan0 15435621ebeSPatrick Venture samples at a | | | 15535621ebeSPatrick Venture faster rate fan1+---> PID +---------->--->fan1 15635621ebeSPatrick Venture speeding up the | | | 15735621ebeSPatrick Venture fans. fan2+---> | +-->fan2 15835621ebeSPatrick Venture ^ +--------+ + 15935621ebeSPatrick Venture | | 16035621ebeSPatrick Venture +-------------------------------+ 16135621ebeSPatrick Venture RPM updated by PWM. 162f0433678SPatrick Venture``` 163