1*d070b7d7SDeepak Kodihalli# PLDM stack on OpenBMC 2*d070b7d7SDeepak Kodihalli 3*d070b7d7SDeepak KodihalliAuthor: Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> <dkodihal> 4*d070b7d7SDeepak KodihalliPrimary assignee: Deepak Kodihalli 5*d070b7d7SDeepak KodihalliCreated: 2019-01-22 6*d070b7d7SDeepak Kodihalli 7*d070b7d7SDeepak Kodihalli## Problem Description 8*d070b7d7SDeepak KodihalliOn OpenBMC, in-band IPMI is currently the primary industry-standard means of 9*d070b7d7SDeepak Kodihallicommunication between the BMC and the Host firmware. We've started hitting some 10*d070b7d7SDeepak Kodihalliinherent limitations of IPMI on OpenPOWER servers: a limited number of sensors, 11*d070b7d7SDeepak Kodihalliand a lack of a generic control mechanism (sensors are a generic monitoring 12*d070b7d7SDeepak Kodihallimechanism) are the major ones. There is a need to improve upon the communication 13*d070b7d7SDeepak Kodihalliprotocol, but at the same time inventing a custom protocol is undesirable. 14*d070b7d7SDeepak Kodihalli 15*d070b7d7SDeepak KodihalliThis design aims to employ Platform Level Data Model (PLDM), a standard 16*d070b7d7SDeepak Kodihalliapplication layer communication protocol defined by the DMTF. PLDM draws inputs 17*d070b7d7SDeepak Kodihallifrom IPMI, but it overcomes most of the latter's limitations. PLDM is also 18*d070b7d7SDeepak Kodihallidesigned to run on standard transport protocols, for e.g. MCTP (also designed by 19*d070b7d7SDeepak Kodihallithe DMTF). MCTP provides for a common transport layer over several physical 20*d070b7d7SDeepak Kodihallichannels, by defining hardware bindings. The solution of PLDM over MCTP also 21*d070b7d7SDeepak Kodihallihelps overcome some of the limitations of the hardware channels that IPMI uses. 22*d070b7d7SDeepak Kodihalli 23*d070b7d7SDeepak KodihalliPLDM's purpose is to enable all sorts of "inside the box communication": BMC - 24*d070b7d7SDeepak KodihalliHost, BMC - BMC, BMC - Network Controller and BMC - Other (for e.g. sensor) 25*d070b7d7SDeepak Kodihallidevices. 26*d070b7d7SDeepak Kodihalli 27*d070b7d7SDeepak Kodihalli## Background and References 28*d070b7d7SDeepak KodihalliPLDM is designed to be an effective interface and data model that provides 29*d070b7d7SDeepak Kodihalliefficient access to low-level platform inventory, monitoring, control, event, 30*d070b7d7SDeepak Kodihalliand data/parameters transfer functions. For example, temperature, voltage, or 31*d070b7d7SDeepak Kodihallifan sensors can have a PLDM representation that can be used to monitor and 32*d070b7d7SDeepak Kodihallicontrol the platform using a set of PLDM messages. PLDM defines data 33*d070b7d7SDeepak Kodihallirepresentations and commands that abstract the platform management hardware. 34*d070b7d7SDeepak Kodihalli 35*d070b7d7SDeepak KodihalliPLDM groups commands under broader functions, and defines 36*d070b7d7SDeepak Kodihalliseparate specifications for each of these functions (also called PLDM "Types"). 37*d070b7d7SDeepak KodihalliThe currently defined Types (and corresponding specs) are : PLDM base (with 38*d070b7d7SDeepak Kodihalliassociated IDs and states specs), BIOS, FRU, Platform monitoring and control, 39*d070b7d7SDeepak KodihalliFirmware Update and SMBIOS. All these specifications are available at: 40*d070b7d7SDeepak Kodihalli 41*d070b7d7SDeepak Kodihallihttps://www.dmtf.org/standards/pmci 42*d070b7d7SDeepak Kodihalli 43*d070b7d7SDeepak KodihalliSome of the reasons PLDM sounds promising (some of these are advantages over 44*d070b7d7SDeepak KodihalliIPMI): 45*d070b7d7SDeepak Kodihalli 46*d070b7d7SDeepak Kodihalli- Common in-band communication protocol. 47*d070b7d7SDeepak Kodihalli 48*d070b7d7SDeepak Kodihalli- Already existing PLDM Type specifications that cover the most common 49*d070b7d7SDeepak Kodihalli communication requirements. Up to 64 PLDM Types can be defined (the last one 50*d070b7d7SDeepak Kodihalli is OEM). At the moment, 6 are defined. Each PLDM type can house up to 256 PLDM 51*d070b7d7SDeepak Kodihalli commands. 52*d070b7d7SDeepak Kodihalli 53*d070b7d7SDeepak Kodihalli- PLDM sensors are 2 bytes in length. 54*d070b7d7SDeepak Kodihalli 55*d070b7d7SDeepak Kodihalli- PLDM introduces the concept of effecters - a control mechanism. Both sensors 56*d070b7d7SDeepak Kodihalli and effecters are associated to entities (similar to IPMI, entities can be 57*d070b7d7SDeepak Kodihalli physical or logical), where sensors are a mechanism for monitoring and 58*d070b7d7SDeepak Kodihalli effecters are a mechanism for control. Effecters can be numeric or state 59*d070b7d7SDeepak Kodihalli based. PLDM defines commonly used entities and their IDs, but there 8K slots 60*d070b7d7SDeepak Kodihalli available to define OEM entities. 61*d070b7d7SDeepak Kodihalli 62*d070b7d7SDeepak Kodihalli- A very active PLDM related working group in the DMTF. 63*d070b7d7SDeepak Kodihalli 64*d070b7d7SDeepak KodihalliThe plan is to run PLDM over MCTP. MCTP is defined in a spec of its own, and a 65*d070b7d7SDeepak Kodihalliproposal on the MCTP design is in discussion already. There's going to be an 66*d070b7d7SDeepak Kodihalliintermediate PLDM over MCTP binding layer, which lets us send PLDM messages over 67*d070b7d7SDeepak KodihalliMCTP. This is defined in a spec of its own, and the design for this binding will 68*d070b7d7SDeepak Kodihallibe proposed separately. 69*d070b7d7SDeepak Kodihalli 70*d070b7d7SDeepak Kodihalli## Requirements 71*d070b7d7SDeepak KodihalliHow different BMC applications make use of PLDM messages is outside the scope 72*d070b7d7SDeepak Kodihalliof this requirements doc. The requirements listed here are related to the PLDM 73*d070b7d7SDeepak Kodihalliprotocol stack and the request/response model: 74*d070b7d7SDeepak Kodihalli 75*d070b7d7SDeepak Kodihalli- Marshalling and unmarshalling of PLDM messages, defined in various PLDM Type 76*d070b7d7SDeepak Kodihalli specs, must be implemented. This can of course be staged based on the need of 77*d070b7d7SDeepak Kodihalli specific Types and functions. Since this is just encoding and decoding PLDM 78*d070b7d7SDeepak Kodihalli messages, this can be a library that could shared between the BMC, and other 79*d070b7d7SDeepak Kodihalli firmware stacks. The specifics of each PLDM Type (such as FRU table 80*d070b7d7SDeepak Kodihalli structures, sensor PDR structures, etc) are implemented by this lib. 81*d070b7d7SDeepak Kodihalli 82*d070b7d7SDeepak Kodihalli- Mapping PLDM concepts to native OpenBMC concepts must be implemented. For 83*d070b7d7SDeepak Kodihalli e.g.: mapping PLDM sensors to phosphor-hwmon hosted D-Bus objects, mapping 84*d070b7d7SDeepak Kodihalli PLDM FRU data to D-Bus objects hosted by phosphor-inventory-manager, etc. The 85*d070b7d7SDeepak Kodihalli mapping shouldn't be restrictive to D-Bus alone (meaning it shouldn't be 86*d070b7d7SDeepak Kodihalli necessary to put objects on the Bus just to serve PLDM requests, a problem 87*d070b7d7SDeepak Kodihalli that exists with phosphor-host-ipmid today). Essentially these are platform 88*d070b7d7SDeepak Kodihalli specific PLDM message handlers. 89*d070b7d7SDeepak Kodihalli 90*d070b7d7SDeepak Kodihalli- The BMC should be able to act as a PLDM responder as well as a PLDM requester. 91*d070b7d7SDeepak Kodihalli As a PLDM requester, the BMC can monitor/control other devices. As a PLDM 92*d070b7d7SDeepak Kodihalli responder, the BMC can react to PLDM messages directed to it via requesters in 93*d070b7d7SDeepak Kodihalli the platform. 94*d070b7d7SDeepak Kodihalli 95*d070b7d7SDeepak Kodihalli- As a PLDM requester, the BMC must be able to discover other PLDM enabled 96*d070b7d7SDeepak Kodihalli components in the platform. 97*d070b7d7SDeepak Kodihalli 98*d070b7d7SDeepak Kodihalli- As a PLDM requester, the BMC must be able to send simultaneous messages to 99*d070b7d7SDeepak Kodihalli different responders. 100*d070b7d7SDeepak Kodihalli 101*d070b7d7SDeepak Kodihalli- As a PLDM requester, the BMC must be able to handle out of order responses. 102*d070b7d7SDeepak Kodihalli 103*d070b7d7SDeepak Kodihalli- As a PLDM responder, the BMC may simultaneously respond to messages from 104*d070b7d7SDeepak Kodihalli different requesters, but the spec doesn't mandate this. In other words the 105*d070b7d7SDeepak Kodihalli responder could be single-threaded. 106*d070b7d7SDeepak Kodihalli 107*d070b7d7SDeepak Kodihalli- It should be possible to plug-in OEM PLDM types/functions into the PLDM stack. 108*d070b7d7SDeepak Kodihalli 109*d070b7d7SDeepak Kodihalli## Proposed Design 110*d070b7d7SDeepak KodihalliThis document covers the architectural, interface, and design details. It 111*d070b7d7SDeepak Kodihalliprovides recommendations for implementations, but implementation details are 112*d070b7d7SDeepak Kodihallioutside the scope of this document. 113*d070b7d7SDeepak Kodihalli 114*d070b7d7SDeepak KodihalliThe design aims at having a single PLDM daemon serve both the requester and 115*d070b7d7SDeepak Kodihalliresponder functions, and having transport specific endpoints to communicate 116*d070b7d7SDeepak Kodihallion different channels. 117*d070b7d7SDeepak Kodihalli 118*d070b7d7SDeepak KodihalliThe design enables concurrency aspects of the requester and responder functions, 119*d070b7d7SDeepak Kodihallibut the goal is to employ asynchronous IO and event loops, instead of multiple 120*d070b7d7SDeepak Kodihallithreads, wherever possible. 121*d070b7d7SDeepak Kodihalli 122*d070b7d7SDeepak KodihalliThe following are high level structural elements of the design: 123*d070b7d7SDeepak Kodihalli 124*d070b7d7SDeepak Kodihalli### PLDM encode/decode libraries 125*d070b7d7SDeepak Kodihalli 126*d070b7d7SDeepak KodihalliThis library would take a PLDM message, decode it and extract the different 127*d070b7d7SDeepak Kodihallifields of the message. Conversely, given a PLDM Type, command code, and the 128*d070b7d7SDeepak Kodihallicommand's data fields, it would make a PLDM message. The thought is to design 129*d070b7d7SDeepak Kodihallithis as a common library, that can be used by the BMC and other firmware stacks, 130*d070b7d7SDeepak Kodihallibecause it's the encode/decode and protocol piece (and not the handling of a 131*d070b7d7SDeepak Kodihallimessage). 132*d070b7d7SDeepak Kodihalli 133*d070b7d7SDeepak Kodihalli### PLDM provider libraries 134*d070b7d7SDeepak Kodihalli 135*d070b7d7SDeepak KodihalliThese libraries would implement the platform specific handling of incoming PLDM 136*d070b7d7SDeepak Kodihallirequests (basically helping with the PLDM responder implementation, see next 137*d070b7d7SDeepak Kodihallibullet point), so for instance they would query D-Bus objects (or even something 138*d070b7d7SDeepak Kodihallilike a JSON file) to fetch platform specific information to respond to the PLDM 139*d070b7d7SDeepak Kodihallimessage. They would link with the encode/decode lib. 140*d070b7d7SDeepak Kodihalli 141*d070b7d7SDeepak KodihalliIt should be possible to plug-in a provider library, that lets someone add 142*d070b7d7SDeepak Kodihallifunctionality for new PLDM (standard as well as OEM) Types. The libraries would 143*d070b7d7SDeepak Kodihalliimplement a "register" API to plug-in handlers for specific PLDM messages. 144*d070b7d7SDeepak KodihalliSomething like: 145*d070b7d7SDeepak Kodihalli 146*d070b7d7SDeepak Kodihallitemplate <typename Handler, typename... args> 147*d070b7d7SDeepak Kodihalliauto register(uint8_t type, uint8_t command, Handler handler); 148*d070b7d7SDeepak Kodihalli 149*d070b7d7SDeepak KodihalliThis allows for providing a strongly-typed C++ handler registration scheme. It 150*d070b7d7SDeepak Kodihalliwould also be possible to validate the parameters passed to the handler at 151*d070b7d7SDeepak Kodihallicompile time. 152*d070b7d7SDeepak Kodihalli 153*d070b7d7SDeepak Kodihalli### Request/Response Model 154*d070b7d7SDeepak Kodihalli 155*d070b7d7SDeepak KodihalliThe PLDM daemon links with the encode/decode and provider libs. The daemon 156*d070b7d7SDeepak Kodihalliwould have to implement the following functions: 157*d070b7d7SDeepak Kodihalli 158*d070b7d7SDeepak Kodihalli#### Receiver/Responder 159*d070b7d7SDeepak KodihalliThe receiver wakes up on getting notified of incoming PLDM messages (via D-Bus 160*d070b7d7SDeepak Kodihallisignal or callback from the transport layer) from a remote PLDM device. If the 161*d070b7d7SDeepak Kodihallimessage type is "Request" it would route them to a PLDM provider library. Via 162*d070b7d7SDeepak Kodihallithe library, asynchronous D-Bus calls (using sdbusplus-asio) would be made, so 163*d070b7d7SDeepak Kodihallithat the receiver can register a handler for the D-Bus response, instead of 164*d070b7d7SDeepak Kodihallihaving to wait for the D-Bus response. This way it can go back to listening for 165*d070b7d7SDeepak Kodihalliincoming PLDM messages. 166*d070b7d7SDeepak Kodihalli 167*d070b7d7SDeepak KodihalliIn the D-Bus response handler, the receiver will send out the PLDM response 168*d070b7d7SDeepak Kodihallimessage via the transport's send message API. If the transport's send message 169*d070b7d7SDeepak KodihalliAPI blocks for a considerably long duration, then it would have to be run in a 170*d070b7d7SDeepak Kodihallithread of it's own. 171*d070b7d7SDeepak Kodihalli 172*d070b7d7SDeepak KodihalliIf the incoming PLDM message is of type "Response", then the receiver emits a 173*d070b7d7SDeepak KodihalliD-Bus signal pointing to the response message. Any time the message is too 174*d070b7d7SDeepak Kodihallilarge to fit in a D-Bus payload, the message is written to a file, and a 175*d070b7d7SDeepak Kodihalliread-only file descriptor pointing to that file is contained in the D-Bus 176*d070b7d7SDeepak Kodihallisignal. 177*d070b7d7SDeepak Kodihalli 178*d070b7d7SDeepak Kodihalli#### Requester 179*d070b7d7SDeepak KodihalliDesigning the BMC as a PLDM requester is interesting. We haven't had this with 180*d070b7d7SDeepak KodihalliIPMI, because the BMC was typically an IPMI server. PLDM requester functions 181*d070b7d7SDeepak Kodihalliwill be spread across multiple OpenBMC applications (instead of a single big 182*d070b7d7SDeepak Kodihallirequester app) - based on the responder they're talking to and the high level 183*d070b7d7SDeepak Kodihallifunction they implement. For example, there could be an app that lets the BMC 184*d070b7d7SDeepak Kodihalliupgrade firmware for other devices using PLDM - this would be a generic app 185*d070b7d7SDeepak Kodihalliin the sense that the same set of commands might have to be run irrespective 186*d070b7d7SDeepak Kodihalliof the device on the other side. There could also be an app that does fan 187*d070b7d7SDeepak Kodihallicontrol on a remote device, based on sensors from that device and algorithms 188*d070b7d7SDeepak Kodihallispecific to that device. 189*d070b7d7SDeepak Kodihalli 190*d070b7d7SDeepak KodihalliThe PLDM daemon would have to implement D-Bus interfaces to form the requester 191*d070b7d7SDeepak Kodihallifunctions: a method to send a PLDM message over the underlying transport (again, 192*d070b7d7SDeepak Kodihallithis will have two versions: one that accepts a byte stream, and the other that 193*d070b7d7SDeepak Kodihalliaccepts an fd, for large messages) and a signal to indicate a PLDM response from 194*d070b7d7SDeepak Kodihallithe remote PLDM device. The signal would comprise of the transport headers, PLDM 195*d070b7d7SDeepak Kodihalliheaders, and the PLDM payload. 196*d070b7d7SDeepak Kodihalli 197*d070b7d7SDeepak KodihalliThe typical flow for a requester app would be to send the PLDM message via the 198*d070b7d7SDeepak KodihalliD-Bus API (the PLDM daemon would have to assign an instance id), and add a 199*d070b7d7SDeepak Kodihallihandler for the D-Bus signal containing the response. As this flow is 200*d070b7d7SDeepak Kodihalliasynchronous, the requester app can execute other scheduled work, if any, in its 201*d070b7d7SDeepak Kodihallievent loop, while it waits for the D-Bus signal containing the response. The 202*d070b7d7SDeepak KodihalliD-Bus API to send a PLDM message to the remote PLDM device would call the 203*d070b7d7SDeepak Kodihalliunderlying transport's send API. If that API blocks for too long, the call may 204*d070b7d7SDeepak Kodihallihave to run in a thread of it's own. The D-Bus signal containing a response 205*d070b7d7SDeepak Kodihallimessage is emitted by the receiver (see above). 206*d070b7d7SDeepak Kodihalli 207*d070b7d7SDeepak Kodihalli### Multiple transport channels 208*d070b7d7SDeepak KodihalliThe PLDM daemon might have to talk to remote PLDM devices via different 209*d070b7d7SDeepak Kodihallichannels. While a level of abstraction might be provided by MCTP, the PLDM 210*d070b7d7SDeepak Kodihallidaemon would have to implement a D-Bus interface to target a specific 211*d070b7d7SDeepak Kodihallitransport channel, so that requester apps on the BMC can send messages over 212*d070b7d7SDeepak Kodihallithat transport. Also, it should be possible to plug-in platform specific D-Bus 213*d070b7d7SDeepak Kodihalliobjects that implement an interface to target a platform specific transport. 214*d070b7d7SDeepak Kodihalli 215*d070b7d7SDeepak Kodihalli## Alternatives Considered 216*d070b7d7SDeepak KodihalliContinue using IPMI, but start making more use of OEM extensions to 217*d070b7d7SDeepak Kodihallisuit the requirements of new platforms. However, given that the IPMI 218*d070b7d7SDeepak Kodihallistandard is no longer under active development, we would likely end up 219*d070b7d7SDeepak Kodihalliwith a large amount of platform-specific customisations. This also does 220*d070b7d7SDeepak Kodihallinot solve the hardware channel issues in a standard manner. 221*d070b7d7SDeepak KodihalliOn OpenPOWER hardware at least, we've started to hit some of the limitations of 222*d070b7d7SDeepak KodihalliIPMI (for example, we have need for >255 sensors). 223*d070b7d7SDeepak Kodihalli 224*d070b7d7SDeepak Kodihalli## Impacts 225*d070b7d7SDeepak KodihalliDevelopment would be required to implement the PLDM protocol, the 226*d070b7d7SDeepak Kodihallirequest/response model, and platform specific handling. Low level design is 227*d070b7d7SDeepak Kodihallirequired to implement the protocol specifics of each of the PLDM Types. Such low 228*d070b7d7SDeepak Kodihallilevel design is not included in this proposal. 229*d070b7d7SDeepak Kodihalli 230*d070b7d7SDeepak KodihalliDesign and development needs to involve the firmware stacks of management 231*d070b7d7SDeepak Kodihallicontrollers and management devices of a platform management subsystem. 232*d070b7d7SDeepak Kodihalli 233*d070b7d7SDeepak Kodihalli## Testing 234*d070b7d7SDeepak KodihalliTesting can be done without having to depend on the underlying transport layer. 235*d070b7d7SDeepak Kodihalli 236*d070b7d7SDeepak KodihalliThe responder function can be tested by mocking a requester and the transport 237*d070b7d7SDeepak Kodihallilayer: this would essentially test the protocol handling and platform specific 238*d070b7d7SDeepak Kodihallihandling. The requester function can be tested by mocking a responder: this 239*d070b7d7SDeepak Kodihalliwould test the instance id handling and the send/receive functions. 240*d070b7d7SDeepak Kodihalli 241*d070b7d7SDeepak KodihalliAPIs from the shared libraries can be tested via fuzzing. 242