1*0344dc18SVelumani Thangavel# Multi-host IPMI design 2*0344dc18SVelumani Thangavel 3*0344dc18SVelumani ThangavelAuthor: 4*0344dc18SVelumani Thangavel Velumani T(velu), [velumanit@hcl](mailto:velumanit@hcl.com) 5*0344dc18SVelumani Thangavel Kumar T(kumar_t), [thangavel.k@hcl.com](mailto:thangavel.k@hcl.com) 6*0344dc18SVelumani Thangavel 7*0344dc18SVelumani ThangavelPrimary assignee: 8*0344dc18SVelumani Thangavel 9*0344dc18SVelumani ThangavelOther contributors: 10*0344dc18SVelumani Thangavel 11*0344dc18SVelumani ThangavelCreated: 12*0344dc18SVelumani Thangavel June 26, 2020 13*0344dc18SVelumani Thangavel 14*0344dc18SVelumani Thangavel## Problem Description 15*0344dc18SVelumani ThangavelThe current version of OpenBMC does not support multi-host implementation in 16*0344dc18SVelumani ThangavelIPMI commands handling. We have a multi-host system and proposing the design 17*0344dc18SVelumani Thangavelto support multi-host. 18*0344dc18SVelumani Thangavel 19*0344dc18SVelumani ThangavelAs detailed below the hosts are connected in the IPMB interface, all host 20*0344dc18SVelumani Thangavelrelated communication is based on IPMB. The OpenBMC uses ipmbbridged to manage 21*0344dc18SVelumani ThangavelIPMB buses and the IPMB messages are routed to ipmid. 22*0344dc18SVelumani Thangavel 23*0344dc18SVelumani ThangavelIssue 1: ipmbridged does not send the channel number (ie HostId) 24*0344dc18SVelumani ThangavelIssue 2: ipmid does not have the information on which IPMB channel the request 25*0344dc18SVelumani Thangavelhas come from. The ipmid handlers should have the host details to fetch the 26*0344dc18SVelumani Thangavelhost specific responses. 27*0344dc18SVelumani Thangavel 28*0344dc18SVelumani Thangavel## Background and References 29*0344dc18SVelumani ThangavelIPMI and IPMB System architecture: 30*0344dc18SVelumani Thangavel``` 31*0344dc18SVelumani Thangavel +------------------------------------+ 32*0344dc18SVelumani Thangavel | BMC | 33*0344dc18SVelumani Thangavel | +-----------+ +------------+ | +--------+ 34*0344dc18SVelumani Thangavel | | | | | | IPMB1| | 35*0344dc18SVelumani Thangavel | | | | |-|------| Host-1 | 36*0344dc18SVelumani Thangavel | | | | | | | | 37*0344dc18SVelumani Thangavel | | | | | | +--------+ 38*0344dc18SVelumani Thangavel | | | | | | 39*0344dc18SVelumani Thangavel | | | | | | 40*0344dc18SVelumani Thangavel | | | D-Bus | | | +--------+ 41*0344dc18SVelumani Thangavel | | ipmid |-------| ipmbbridged| | IPMB2| | 42*0344dc18SVelumani Thangavel | | | | |-|------| Host-2 | 43*0344dc18SVelumani Thangavel | | | | | | | | 44*0344dc18SVelumani Thangavel | | | | | | +--------+ 45*0344dc18SVelumani Thangavel | | | | | | 46*0344dc18SVelumani Thangavel | | | | | | 47*0344dc18SVelumani Thangavel | | | | | | +--------+ 48*0344dc18SVelumani Thangavel | | | | | | IPMBn| | 49*0344dc18SVelumani Thangavel | | | | |-|------| Host-N | 50*0344dc18SVelumani Thangavel | | | | | | | | 51*0344dc18SVelumani Thangavel | +-----------+ +------------+ | +--------+ 52*0344dc18SVelumani Thangavel +------------------------------------+ 53*0344dc18SVelumani Thangavel``` 54*0344dc18SVelumani ThangavelHosts are connected with IPMB interface, the hosts can be 1 to N. The IPMB 55*0344dc18SVelumani Thangavelrequest coming from the hosts are routed to ipmid by the ipmbbridged. 56*0344dc18SVelumani ThangavelThe IPMB requests are routed from ipmid or any service by D-Bus interface and 57*0344dc18SVelumani ThangavelThe outgoing IPMB requests are routed by ipmbbridged to IPMB interface. 58*0344dc18SVelumani Thangavel 59*0344dc18SVelumani Thangavel## Requirements 60*0344dc18SVelumani Thangavel 61*0344dc18SVelumani ThangavelThe current version of OpenBMC does not support multi-host implementation in 62*0344dc18SVelumani ThangavelIPMI commands handling. We have a multi-host system and proposing the design 63*0344dc18SVelumani Thangavelto support multi-host. 64*0344dc18SVelumani Thangavel 65*0344dc18SVelumani Thangavel## Proposed Design 66*0344dc18SVelumani Thangavel 67*0344dc18SVelumani ThangavelTo address issue1 and issue2, we propose the following design changes in 68*0344dc18SVelumani Thangavelipmbbridged and ipmid. 69*0344dc18SVelumani ThangavelTo address out-of-band IPMI command from the network,the proposal is captured 70*0344dc18SVelumani Thangavelin section "Changes in netipmid". 71*0344dc18SVelumani Thangavel 72*0344dc18SVelumani ThangavelIssue1: Changes in ipmbbridged: 73*0344dc18SVelumani Thangavel- 74*0344dc18SVelumani Thangavelipmbbridged to send the channel details from where the request is received. 75*0344dc18SVelumani Thangavel 76*0344dc18SVelumani Thangavel**Change: Sending Host detail as additional parameter** 77*0344dc18SVelumani Thangavel 78*0344dc18SVelumani ThangavelWhile routing the IPMB requests coming from the host channel, We will be 79*0344dc18SVelumani Thangaveladding new entry in the json config file for the host ID '"devIndex": 0' 80*0344dc18SVelumani Thangavelipmbbridged will send '"devIndex": 0' as optional parameter(options) in D-Bus 81*0344dc18SVelumani Thangavelinterface to ipmid.This can be used to get the information on which IPMB bus 82*0344dc18SVelumani Thangavelthe message comes from. 83*0344dc18SVelumani Thangavel 84*0344dc18SVelumani ThangavelThe json file looks like below. Each devIndex can have one "me" and "ipmb" 85*0344dc18SVelumani Thangavelchannel.To ensure existing platforms does not get affected, if the "devIndex" 86*0344dc18SVelumani Thangavelkey is not present in the file ipmbbridged uses default "devIndex" as 0. 87*0344dc18SVelumani Thangavel 88*0344dc18SVelumani Thangavel{ "type": "me", 89*0344dc18SVelumani Thangavel"slave-path": "/dev/ipmb-1", 90*0344dc18SVelumani Thangavel"bmc-addr": 32, 91*0344dc18SVelumani Thangavel"remote-addr": 64, 92*0344dc18SVelumani Thangavel"devIndex": 0 93*0344dc18SVelumani Thangavel}, 94*0344dc18SVelumani Thangavel{ "type": "ipmb", 95*0344dc18SVelumani Thangavel"slave-path": "/dev/ipmb-2", 96*0344dc18SVelumani Thangavel"bmc-addr": 32, 97*0344dc18SVelumani Thangavel"remote-addr": 64, 98*0344dc18SVelumani Thangavel"devIndex": 0 99*0344dc18SVelumani Thangavel}, 100*0344dc18SVelumani Thangavel{ "type": "me", 101*0344dc18SVelumani Thangavel"slave-path": "/dev/ipmb-3", 102*0344dc18SVelumani Thangavel"bmc-addr": 32, 103*0344dc18SVelumani Thangavel"remote-addr": 64, 104*0344dc18SVelumani Thangavel"devIndex": 1 105*0344dc18SVelumani Thangavel}, 106*0344dc18SVelumani Thangavel{ "type": "ipmb", 107*0344dc18SVelumani Thangavel"slave-path": "/dev/ipmb-4", 108*0344dc18SVelumani Thangavel"bmc-addr": 32, 109*0344dc18SVelumani Thangavel"remote-addr": 64, 110*0344dc18SVelumani Thangavel"devIndex": 1 111*0344dc18SVelumani Thangavel}, 112*0344dc18SVelumani Thangavel 113*0344dc18SVelumani ThangavelIssue2: Changes in ipmid: 114*0344dc18SVelumani Thangavel- 115*0344dc18SVelumani ThangavelReceive the optional parameter sent by the ipmbbridged as host details, while 116*0344dc18SVelumani Thangavelreceiving the parameter in the executionEntry method call the same will be 117*0344dc18SVelumani Thangavelpassed to the command handlers in both common and oem handlers.The command 118*0344dc18SVelumani Thangavelhandlers can make use of the host information to fetch host specific data. 119*0344dc18SVelumani Thangavel 120*0344dc18SVelumani ThangavelFor example, host1 send a request to get boot order from BMC, BMC maintains 121*0344dc18SVelumani Thangaveldata separately for each host. When this command comes to ipmid the commands 122*0344dc18SVelumani Thangavelhandler gets the host in which the command received. The handler will fetch 123*0344dc18SVelumani Thangavelhost1 boot order details and respond from the command handler. This is 124*0344dc18SVelumani Thangavelapplicable for both common and oem handlers. 125*0344dc18SVelumani Thangavel 126*0344dc18SVelumani ThangavelChanges in netipmid: 127*0344dc18SVelumani Thangavel- 128*0344dc18SVelumani ThangavelThe "options" parameter can be used for sending the host information from 129*0344dc18SVelumani Thangavelnetipmid. The changes proposed for ipmbbridged can be used in netipmid as well. 130*0344dc18SVelumani ThangavelThe netipmid sends the "devIndex" on which channel the request comes from. 131*0344dc18SVelumani ThangavelThere will not be any further changes required in ipmid. 132*0344dc18SVelumani ThangavelThe netipmid can have multiple approaches to handle multi-host.Some of the 133*0344dc18SVelumani Thangavelapproaches are listed down and but not limited to this list. 134*0344dc18SVelumani Thangavel1. Virtual Ethernet interfaces - One virtual interface per host. 135*0344dc18SVelumani Thangavel2. Different port numbers - Can have different port numbers for each host. 136*0344dc18SVelumani Thangavel3. VLAN Ids- VLAN IDs can be used to support multi host. 137*0344dc18SVelumani ThangavelThe netipmid shall have a config file where in the interfaces can be configured 138*0344dc18SVelumani Thangavelfor each host. Also one or more interfaces can be configured to each of the 139*0344dc18SVelumani Thangavelhost. The interfaces can be virtual or physical. 140*0344dc18SVelumani ThangavelBelow is the sample configuration file 141*0344dc18SVelumani Thangavel 142*0344dc18SVelumani Thangavel{"Host":1, 143*0344dc18SVelumani Thangavel"Interface-1":"eth0", 144*0344dc18SVelumani Thangavel"Interface-2":"eth1", 145*0344dc18SVelumani Thangavel"Interface-3":"veth4", 146*0344dc18SVelumani Thangavel"Interface-4":"veth5" 147*0344dc18SVelumani Thangavel}, 148*0344dc18SVelumani Thangavel{"Host":2, 149*0344dc18SVelumani Thangavel"Interface-1":"eth2", 150*0344dc18SVelumani Thangavel"Interface-2":"eth3", 151*0344dc18SVelumani Thangavel"Interface-3":"veth1", 152*0344dc18SVelumani Thangavel"Interface-4":"veth2" 153*0344dc18SVelumani Thangavel}, 154*0344dc18SVelumani Thangavel 155*0344dc18SVelumani ThangavelExample implementation of approach1:Virtual ethernet interface. 156*0344dc18SVelumani Thangavel 157*0344dc18SVelumani Thangavel``` 158*0344dc18SVelumani Thangavel+--------------------------------------------+ 159*0344dc18SVelumani Thangavel| BMC | 160*0344dc18SVelumani Thangavel| +--------+ +-----------+ +------+ | +--------+ 161*0344dc18SVelumani Thangavel| | | D-Bus | | | | | | | 162*0344dc18SVelumani Thangavel| | |-------| netipmid1 |---|veth1 |---------| Host-1 | 163*0344dc18SVelumani Thangavel| | | | | | | | | | 164*0344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 165*0344dc18SVelumani Thangavel| | | | 166*0344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 167*0344dc18SVelumani Thangavel| | ipmid | D-Bus | | | | | | | 168*0344dc18SVelumani Thangavel| | |-------| netipmid2 |---|veth2 |---------| Host-2 | 169*0344dc18SVelumani Thangavel| | | | | | | | | | 170*0344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 171*0344dc18SVelumani Thangavel| | | | 172*0344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 173*0344dc18SVelumani Thangavel| | | D-Bus | | | | | | | 174*0344dc18SVelumani Thangavel| | |-------| netipmidN |---|vethN |---------| Host-N | 175*0344dc18SVelumani Thangavel| | | | | | | | | | 176*0344dc18SVelumani Thangavel| +--------+ +-----------+ +------+ | +--------+ 177*0344dc18SVelumani Thangavel+--------------------------------------------+ 178*0344dc18SVelumani Thangavel``` 179*0344dc18SVelumani ThangavelIn the above diagram one instance of netipmid runs per host. Each instance 180*0344dc18SVelumani Thangavelis tied to one virtual ethernet interface, The virutal interface ID can be 181*0344dc18SVelumani Thangavelused to make a devIndex. This represents the HostId. 182*0344dc18SVelumani Thangavel 183*0344dc18SVelumani Thangavel## Alternatives Considered 184*0344dc18SVelumani Thangavel 185*0344dc18SVelumani ThangavelApproach1:ipmbbridged to send host-id in the payload 186*0344dc18SVelumani Thangavel- 187*0344dc18SVelumani ThangavelThe ipmbbridged shall be modified to send the host id in data payload. This 188*0344dc18SVelumani Thangavellooks to be a simple change but impacts the existing platforms which are already 189*0344dc18SVelumani Thangavelusing it.This may not be a right approach. 190*0344dc18SVelumani Thangavel 191*0344dc18SVelumani ThangavelApproach2:Create multiple ipmid to handle multihost.One ipmid process per host. 192*0344dc18SVelumani Thangavel- 193*0344dc18SVelumani ThangavelThis is a multi service appoach,one instance of ipmid service shall be 194*0344dc18SVelumani Thangavelspawned to respond each host.The changes looks simple and no major design 195*0344dc18SVelumani Thangavelchange from the existing design. But many common handlers will be running as 196*0344dc18SVelumani Thangavelduplicate in multiple instances. 197*0344dc18SVelumani Thangavel 198*0344dc18SVelumani ThangavelApproach3:Using a different IPMI channel for handling multiple host. 199*0344dc18SVelumani Thangavel- 200*0344dc18SVelumani ThangavelUsing a different IPMI channel for handling multiple hosts, in the ipmbbridged 201*0344dc18SVelumani Thangavelthe channel id can be used to identify host. In this approach we will be having 202*0344dc18SVelumani Thangavelmultiple instances of ipmbbridged and each instance will be registered with the 203*0344dc18SVelumani Thangavela channel number.Maximum channel numbers are limited to 8 as per the 204*0344dc18SVelumani Thangavelspecification.This limits the maximum hosts to be supported. 205*0344dc18SVelumani Thangavel 206*0344dc18SVelumani Thangavel## Impacts 207*0344dc18SVelumani Thangavel 208*0344dc18SVelumani ThangavelThere may not be an impact in ipmid command handler functions.This design will 209*0344dc18SVelumani Thangavelnot affect the current functionality. 210*0344dc18SVelumani Thangavel 211*0344dc18SVelumani Thangavel## Testing 212*0344dc18SVelumani Thangavel 213*0344dc18SVelumani ThangavelThe proposed design can be tested in a platform in which the multiple hosts 214*0344dc18SVelumani Thangavelare connected.The commands requests are received from all the hosts and 215*0344dc18SVelumani Thangavelresponses are host specific data.When the request coming from the host as IPMB 216*0344dc18SVelumani Thangavelcommand, ipmbbridged appends devIndex and ipmid receives the respective 217*0344dc18SVelumani ThangaveldevIndex.ipmid responds based on the received devIndex(Host Id) and 218*0344dc18SVelumani Thangavelresponse reaches all the way to host.The data can be validated in host. 219