10344dc18SVelumani Thangavel# Multi-host IPMI design 20344dc18SVelumani Thangavel 3*f4febd00SPatrick WilliamsAuthor: Velumani T(velu), [velumanit@hcl](mailto:velumanit@hcl.com) Kumar 4*f4febd00SPatrick WilliamsT(kumar_t), [thangavel.k@hcl.com](mailto:thangavel.k@hcl.com) 50344dc18SVelumani Thangavel 60344dc18SVelumani ThangavelOther contributors: 70344dc18SVelumani Thangavel 8*f4febd00SPatrick WilliamsCreated: June 26, 2020 90344dc18SVelumani Thangavel 100344dc18SVelumani Thangavel## Problem Description 11*f4febd00SPatrick Williams 120344dc18SVelumani ThangavelThe current version of OpenBMC does not support multi-host implementation in 13*f4febd00SPatrick WilliamsIPMI commands handling. We have a multi-host system and proposing the design to 14*f4febd00SPatrick Williamssupport multi-host. 150344dc18SVelumani Thangavel 160344dc18SVelumani ThangavelAs detailed below the hosts are connected in the IPMB interface, all host 170344dc18SVelumani Thangavelrelated communication is based on IPMB. The OpenBMC uses ipmbbridged to manage 180344dc18SVelumani ThangavelIPMB buses and the IPMB messages are routed to ipmid. 190344dc18SVelumani Thangavel 20*f4febd00SPatrick WilliamsIssue 1: ipmbridged does not send the channel number (ie HostId) Issue 2: ipmid 21*f4febd00SPatrick Williamsdoes not have the information on which IPMB channel the request has come from. 22*f4febd00SPatrick WilliamsThe ipmid handlers should have the host details to fetch the host specific 23*f4febd00SPatrick Williamsresponses. 240344dc18SVelumani Thangavel 250344dc18SVelumani Thangavel## Background and References 26*f4febd00SPatrick Williams 270344dc18SVelumani ThangavelIPMI and IPMB System architecture: 28*f4febd00SPatrick Williams 290344dc18SVelumani Thangavel``` 300344dc18SVelumani Thangavel +------------------------------------+ 310344dc18SVelumani Thangavel | BMC | 320344dc18SVelumani Thangavel | +-----------+ +------------+ | +--------+ 330344dc18SVelumani Thangavel | | | | | | IPMB1| | 340344dc18SVelumani Thangavel | | | | |-|------| Host-1 | 350344dc18SVelumani Thangavel | | | | | | | | 360344dc18SVelumani Thangavel | | | | | | +--------+ 370344dc18SVelumani Thangavel | | | | | | 380344dc18SVelumani Thangavel | | | | | | 390344dc18SVelumani Thangavel | | | D-Bus | | | +--------+ 400344dc18SVelumani Thangavel | | ipmid |-------| ipmbbridged| | IPMB2| | 410344dc18SVelumani Thangavel | | | | |-|------| Host-2 | 420344dc18SVelumani Thangavel | | | | | | | | 430344dc18SVelumani Thangavel | | | | | | +--------+ 440344dc18SVelumani Thangavel | | | | | | 450344dc18SVelumani Thangavel | | | | | | 460344dc18SVelumani Thangavel | | | | | | +--------+ 470344dc18SVelumani Thangavel | | | | | | IPMBn| | 480344dc18SVelumani Thangavel | | | | |-|------| Host-N | 490344dc18SVelumani Thangavel | | | | | | | | 500344dc18SVelumani Thangavel | +-----------+ +------------+ | +--------+ 510344dc18SVelumani Thangavel +------------------------------------+ 520344dc18SVelumani Thangavel``` 53*f4febd00SPatrick Williams 540344dc18SVelumani ThangavelHosts are connected with IPMB interface, the hosts can be 1 to N. The IPMB 55*f4febd00SPatrick Williamsrequest coming from the hosts are routed to ipmid by the ipmbbridged. The IPMB 56*f4febd00SPatrick Williamsrequests are routed from ipmid or any service by D-Bus interface and The 57*f4febd00SPatrick Williamsoutgoing IPMB requests are routed by ipmbbridged to IPMB interface. 580344dc18SVelumani Thangavel 590344dc18SVelumani Thangavel## Requirements 600344dc18SVelumani Thangavel 610344dc18SVelumani ThangavelThe current version of OpenBMC does not support multi-host implementation in 62*f4febd00SPatrick WilliamsIPMI commands handling. We have a multi-host system and proposing the design to 63*f4febd00SPatrick Williamssupport multi-host. 640344dc18SVelumani Thangavel 650344dc18SVelumani Thangavel## Proposed Design 660344dc18SVelumani Thangavel 670344dc18SVelumani ThangavelTo address issue1 and issue2, we propose the following design changes in 68*f4febd00SPatrick Williamsipmbbridged and ipmid. To address out-of-band IPMI command from the network,the 69*f4febd00SPatrick Williamsproposal is captured in section "Changes in netipmid". 700344dc18SVelumani Thangavel 71*f4febd00SPatrick Williams## Issue1: Changes in ipmbbridged: 72*f4febd00SPatrick Williams 730344dc18SVelumani Thangavelipmbbridged to send the channel details from where the request is received. 740344dc18SVelumani Thangavel 750344dc18SVelumani Thangavel**Change: Sending Host detail as additional parameter** 760344dc18SVelumani Thangavel 77*f4febd00SPatrick WilliamsWhile routing the IPMB requests coming from the host channel, We will be adding 78*f4febd00SPatrick Williamsnew entry in the json config file for the host ID '"devIndex": 0' ipmbbridged 79*f4febd00SPatrick Williamswill send '"devIndex": 0' as optional parameter(options) in D-Bus interface to 80*f4febd00SPatrick Williamsipmid.This can be used to get the information on which IPMB bus the message 81*f4febd00SPatrick Williamscomes from. 820344dc18SVelumani Thangavel 830344dc18SVelumani ThangavelThe json file looks like below. Each devIndex can have one "me" and "ipmb" 840344dc18SVelumani Thangavelchannel.To ensure existing platforms does not get affected, if the "devIndex" 850344dc18SVelumani Thangavelkey is not present in the file ipmbbridged uses default "devIndex" as 0. 860344dc18SVelumani Thangavel 87*f4febd00SPatrick Williams{ "type": "me", "slave-path": "/dev/ipmb-1", "bmc-addr": 32, "remote-addr": 64, 88*f4febd00SPatrick Williams"devIndex": 0 }, { "type": "ipmb", "slave-path": "/dev/ipmb-2", "bmc-addr": 32, 89*f4febd00SPatrick Williams"remote-addr": 64, "devIndex": 0 }, { "type": "me", "slave-path": "/dev/ipmb-3", 90*f4febd00SPatrick Williams"bmc-addr": 32, "remote-addr": 64, "devIndex": 1 }, { "type": "ipmb", 91*f4febd00SPatrick Williams"slave-path": "/dev/ipmb-4", "bmc-addr": 32, "remote-addr": 64, "devIndex": 1 }, 920344dc18SVelumani Thangavel 93*f4febd00SPatrick Williams## Issue2: Changes in ipmid: 94*f4febd00SPatrick Williams 950344dc18SVelumani ThangavelReceive the optional parameter sent by the ipmbbridged as host details, while 960344dc18SVelumani Thangavelreceiving the parameter in the executionEntry method call the same will be 970344dc18SVelumani Thangavelpassed to the command handlers in both common and oem handlers.The command 980344dc18SVelumani Thangavelhandlers can make use of the host information to fetch host specific data. 990344dc18SVelumani Thangavel 100*f4febd00SPatrick WilliamsFor example, host1 send a request to get boot order from BMC, BMC maintains data 101*f4febd00SPatrick Williamsseparately for each host. When this command comes to ipmid the commands handler 102*f4febd00SPatrick Williamsgets the host in which the command received. The handler will fetch host1 boot 103*f4febd00SPatrick Williamsorder details and respond from the command handler. This is applicable for both 104*f4febd00SPatrick Williamscommon and oem handlers. 1050344dc18SVelumani Thangavel 106*f4febd00SPatrick Williams## Changes in netipmid: 107*f4febd00SPatrick Williams 1080344dc18SVelumani ThangavelThe "options" parameter can be used for sending the host information from 1090344dc18SVelumani Thangavelnetipmid. The changes proposed for ipmbbridged can be used in netipmid as well. 110*f4febd00SPatrick WilliamsThe netipmid sends the "devIndex" on which channel the request comes from. There 111*f4febd00SPatrick Williamswill not be any further changes required in ipmid. The netipmid can have 112*f4febd00SPatrick Williamsmultiple approaches to handle multi-host.Some of the approaches are listed down 113*f4febd00SPatrick Williamsand but not limited to this list. 114*f4febd00SPatrick Williams 1150344dc18SVelumani Thangavel1. Virtual Ethernet interfaces - One virtual interface per host. 1160344dc18SVelumani Thangavel2. Different port numbers - Can have different port numbers for each host. 117*f4febd00SPatrick Williams3. VLAN Ids- VLAN IDs can be used to support multi host. The netipmid shall have 118*f4febd00SPatrick Williams a config file where in the interfaces can be configured for each host. Also 119*f4febd00SPatrick Williams one or more interfaces can be configured to each of the host. The interfaces 120*f4febd00SPatrick Williams can be virtual or physical. Below is the sample configuration file 1210344dc18SVelumani Thangavel 122*f4febd00SPatrick Williams{"Host":1, "Interface-1":"eth0", "Interface-2":"eth1", "Interface-3":"veth4", 123*f4febd00SPatrick Williams"Interface-4":"veth5" }, {"Host":2, "Interface-1":"eth2", "Interface-2":"eth3", 124*f4febd00SPatrick Williams"Interface-3":"veth1", "Interface-4":"veth2" }, 1250344dc18SVelumani Thangavel 1260344dc18SVelumani ThangavelExample implementation of approach1:Virtual ethernet interface. 1270344dc18SVelumani Thangavel 1280344dc18SVelumani Thangavel``` 1290344dc18SVelumani Thangavel+--------------------------------------------+ 1300344dc18SVelumani Thangavel| BMC | 1310344dc18SVelumani Thangavel| +--------+ +-----------+ +------+ | +--------+ 1320344dc18SVelumani Thangavel| | | D-Bus | | | | | | | 1330344dc18SVelumani Thangavel| | |-------| netipmid1 |---|veth1 |---------| Host-1 | 1340344dc18SVelumani Thangavel| | | | | | | | | | 1350344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 1360344dc18SVelumani Thangavel| | | | 1370344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 1380344dc18SVelumani Thangavel| | ipmid | D-Bus | | | | | | | 1390344dc18SVelumani Thangavel| | |-------| netipmid2 |---|veth2 |---------| Host-2 | 1400344dc18SVelumani Thangavel| | | | | | | | | | 1410344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 1420344dc18SVelumani Thangavel| | | | 1430344dc18SVelumani Thangavel| | | +-----------+ +------+ | +--------+ 1440344dc18SVelumani Thangavel| | | D-Bus | | | | | | | 1450344dc18SVelumani Thangavel| | |-------| netipmidN |---|vethN |---------| Host-N | 1460344dc18SVelumani Thangavel| | | | | | | | | | 1470344dc18SVelumani Thangavel| +--------+ +-----------+ +------+ | +--------+ 1480344dc18SVelumani Thangavel+--------------------------------------------+ 1490344dc18SVelumani Thangavel``` 150*f4febd00SPatrick Williams 151*f4febd00SPatrick WilliamsIn the above diagram one instance of netipmid runs per host. Each instance is 152*f4febd00SPatrick Williamstied to one virtual ethernet interface, The virtual interface ID can be used to 153*f4febd00SPatrick Williamsmake a devIndex. This represents the HostId. 1540344dc18SVelumani Thangavel 1550344dc18SVelumani Thangavel## Alternatives Considered 1560344dc18SVelumani Thangavel 157*f4febd00SPatrick Williams## Approach1:ipmbbridged to send host-id in the payload 158*f4febd00SPatrick Williams 1590344dc18SVelumani ThangavelThe ipmbbridged shall be modified to send the host id in data payload. This 1600344dc18SVelumani Thangavellooks to be a simple change but impacts the existing platforms which are already 1610344dc18SVelumani Thangavelusing it.This may not be a right approach. 1620344dc18SVelumani Thangavel 163*f4febd00SPatrick Williams## Approach2:Create multiple ipmid to handle multihost.One ipmid process per host. 1640344dc18SVelumani Thangavel 165*f4febd00SPatrick WilliamsThis is a multi service appoach,one instance of ipmid service shall be spawned 166*f4febd00SPatrick Williamsto respond each host.The changes looks simple and no major design change from 167*f4febd00SPatrick Williamsthe existing design. But many common handlers will be running as duplicate in 168*f4febd00SPatrick Williamsmultiple instances. 169*f4febd00SPatrick Williams 170*f4febd00SPatrick Williams## Approach3:Using a different IPMI channel for handling multiple host. 171*f4febd00SPatrick Williams 1720344dc18SVelumani ThangavelUsing a different IPMI channel for handling multiple hosts, in the ipmbbridged 1730344dc18SVelumani Thangavelthe channel id can be used to identify host. In this approach we will be having 1740344dc18SVelumani Thangavelmultiple instances of ipmbbridged and each instance will be registered with the 1750344dc18SVelumani Thangavela channel number.Maximum channel numbers are limited to 8 as per the 1760344dc18SVelumani Thangavelspecification.This limits the maximum hosts to be supported. 1770344dc18SVelumani Thangavel 1780344dc18SVelumani Thangavel## Impacts 1790344dc18SVelumani Thangavel 1800344dc18SVelumani ThangavelThere may not be an impact in ipmid command handler functions.This design will 1810344dc18SVelumani Thangavelnot affect the current functionality. 1820344dc18SVelumani Thangavel 1830344dc18SVelumani Thangavel## Testing 1840344dc18SVelumani Thangavel 185*f4febd00SPatrick WilliamsThe proposed design can be tested in a platform in which the multiple hosts are 186*f4febd00SPatrick Williamsconnected.The commands requests are received from all the hosts and responses 187*f4febd00SPatrick Williamsare host specific data.When the request coming from the host as IPMB command, 188*f4febd00SPatrick Williamsipmbbridged appends devIndex and ipmid receives the respective devIndex.ipmid 189*f4febd00SPatrick Williamsresponds based on the received devIndex(Host Id) and response reaches all the 190*f4febd00SPatrick Williamsway to host.The data can be validated in host. 191