1# Multi-host IPMI design
2
3Author:
4  Velumani T(velu),  [velumanit@hcl](mailto:velumanit@hcl.com)
5  Kumar T(kumar_t), [thangavel.k@hcl.com](mailto:thangavel.k@hcl.com)
6
7Other contributors:
8
9Created:
10 June 26, 2020
11
12## Problem Description
13The current version of OpenBMC does not support multi-host implementation in
14IPMI commands handling. We have a multi-host system and proposing the design
15to support multi-host.
16
17As detailed below the hosts are connected in the IPMB interface, all host
18related communication is based on IPMB. The OpenBMC uses ipmbbridged to manage
19IPMB buses and the IPMB messages are routed to ipmid.
20
21Issue 1: ipmbridged does not send the channel number (ie HostId)
22Issue 2: ipmid does not have the information on which IPMB channel the request
23has come from. The ipmid handlers should have the host details to fetch the
24host specific responses.
25
26## Background and References
27IPMI and IPMB System architecture:
28```
29 +------------------------------------+
30 |               BMC                  |
31 | +-----------+       +------------+ |      +--------+
32 | |           |       |            | | IPMB1|        |
33 | |           |       |            |-|------| Host-1 |
34 | |           |       |            | |      |        |
35 | |           |       |            | |      +--------+
36 | |           |       |            | |
37 | |           |       |            | |
38 | |           | D-Bus |            | |      +--------+
39 | | ipmid     |-------| ipmbbridged| | IPMB2|        |
40 | |           |       |            |-|------| Host-2 |
41 | |           |       |            | |      |        |
42 | |           |       |            | |      +--------+
43 | |           |       |            | |
44 | |           |       |            | |
45 | |           |       |            | |      +--------+
46 | |           |       |            | | IPMBn|        |
47 | |           |       |            |-|------| Host-N |
48 | |           |       |            | |      |        |
49 | +-----------+       +------------+ |      +--------+
50 +------------------------------------+
51```
52Hosts are connected with IPMB interface, the hosts can be 1 to N. The IPMB
53request coming from the hosts are routed to ipmid by the ipmbbridged.
54The IPMB requests are routed from ipmid or any service by D-Bus interface and
55The outgoing IPMB requests are routed by ipmbbridged to IPMB interface.
56
57## Requirements
58
59The current version of OpenBMC does not support multi-host implementation in
60IPMI commands handling. We have a multi-host system and proposing the design
61to support multi-host.
62
63## Proposed Design
64
65To address issue1 and issue2, we propose the following design changes in
66ipmbbridged and ipmid.
67To address out-of-band IPMI command from the network,the proposal is captured
68in section "Changes in netipmid".
69
70Issue1: Changes in ipmbbridged:
71-
72ipmbbridged to send the channel details from where the request is received.
73
74**Change: Sending Host detail as additional parameter**
75
76While routing the IPMB requests coming from the host channel, We will be
77adding new entry in the json config file for the host ID '"devIndex": 0'
78ipmbbridged will send '"devIndex": 0' as optional parameter(options) in D-Bus
79interface to ipmid.This can be used to get the information on which IPMB bus
80the message comes from.
81
82The json file looks like below. Each devIndex can have one "me" and "ipmb"
83channel.To ensure existing platforms does not get affected, if the "devIndex"
84key is not present in the file ipmbbridged uses default "devIndex" as 0.
85
86{ "type": "me",
87"slave-path": "/dev/ipmb-1",
88"bmc-addr": 32,
89"remote-addr": 64,
90"devIndex": 0
91},
92{ "type": "ipmb",
93"slave-path": "/dev/ipmb-2",
94"bmc-addr": 32,
95"remote-addr": 64,
96"devIndex": 0
97},
98{ "type": "me",
99"slave-path": "/dev/ipmb-3",
100"bmc-addr": 32,
101"remote-addr": 64,
102"devIndex": 1
103},
104{ "type": "ipmb",
105"slave-path": "/dev/ipmb-4",
106"bmc-addr": 32,
107"remote-addr": 64,
108"devIndex": 1
109},
110
111Issue2: Changes in ipmid:
112-
113Receive the optional parameter sent by the ipmbbridged as host details, while
114receiving the parameter in the executionEntry method call the same will be
115passed to the command handlers in both common and oem handlers.The command
116handlers can make use of the host information to fetch host specific data.
117
118For example, host1 send a request to get boot order from BMC, BMC maintains
119data separately for each host. When this command comes to ipmid the commands
120handler gets the host in which the command received. The handler will fetch
121host1 boot order details and respond from the command handler. This is
122applicable for both common and oem handlers.
123
124Changes in netipmid:
125-
126The "options" parameter can be used for sending the host information from
127netipmid. The changes proposed for ipmbbridged can be used in netipmid as well.
128The netipmid sends the "devIndex" on which channel the request comes from.
129There will not be any further changes required in ipmid.
130The netipmid can have multiple approaches to handle multi-host.Some of the
131approaches are listed down and but not limited to this list.
1321. Virtual Ethernet interfaces - One virtual interface per host.
1332. Different port numbers - Can have different port numbers for each host.
1343. VLAN Ids- VLAN IDs can be used to support multi host.
135The netipmid shall have a config file where in the interfaces can be configured
136for each host. Also one or more interfaces can be configured to each of the
137host. The interfaces can be virtual or physical.
138Below is the sample configuration file
139
140{"Host":1,
141"Interface-1":"eth0",
142"Interface-2":"eth1",
143"Interface-3":"veth4",
144"Interface-4":"veth5"
145},
146{"Host":2,
147"Interface-1":"eth2",
148"Interface-2":"eth3",
149"Interface-3":"veth1",
150"Interface-4":"veth2"
151},
152
153Example implementation of approach1:Virtual ethernet interface.
154
155```
156+--------------------------------------------+
157|           BMC                              |
158| +--------+       +-----------+   +------+  |      +--------+
159| |        | D-Bus |           |   |      |  |      |        |
160| |        |-------| netipmid1 |---|veth1 |---------| Host-1 |
161| |        |       |           |   |      |  |      |        |
162| |        |       +-----------+   +------+  |      +--------+
163| |        |                                 |
164| |        |       +-----------+   +------+  |      +--------+
165| | ipmid  | D-Bus |           |   |      |  |      |        |
166| |        |-------| netipmid2 |---|veth2 |---------| Host-2 |
167| |        |       |           |   |      |  |      |        |
168| |        |       +-----------+   +------+  |      +--------+
169| |        |                                 |
170| |        |       +-----------+   +------+  |      +--------+
171| |        | D-Bus |           |   |      |  |      |        |
172| |        |-------| netipmidN |---|vethN |---------| Host-N |
173| |        |       |           |   |      |  |      |        |
174| +--------+       +-----------+   +------+  |      +--------+
175+--------------------------------------------+
176```
177In the above diagram one instance of netipmid runs per host. Each instance
178is tied to one virtual ethernet interface, The virtual interface ID can be
179used to make a devIndex. This represents the HostId.
180
181## Alternatives Considered
182
183Approach1:ipmbbridged to send host-id in the payload
184-
185The ipmbbridged shall be modified to send the host id in data payload. This
186looks to be a simple change but impacts the existing platforms which are already
187using it.This may not be a right approach.
188
189Approach2:Create multiple ipmid to handle multihost.One ipmid process per host.
190-
191This is a multi service appoach,one instance of ipmid service shall be
192spawned to respond each host.The changes looks simple and no major design
193change from the existing design. But many common handlers will be running as
194duplicate in multiple instances.
195
196Approach3:Using a different IPMI channel for handling multiple host.
197-
198Using a different IPMI channel for handling multiple hosts, in the ipmbbridged
199the channel id can be used to identify host. In this approach we will be having
200multiple instances of ipmbbridged and each instance will be registered with the
201a channel number.Maximum channel numbers are limited to 8 as per the
202specification.This limits the maximum hosts to be supported.
203
204## Impacts
205
206There may not be an impact in ipmid command handler functions.This design will
207not affect the current functionality.
208
209## Testing
210
211The proposed design can be tested in a platform in which the multiple hosts
212are connected.The commands requests are received from all the hosts and
213responses are host specific data.When the request coming from the host as IPMB
214command, ipmbbridged appends devIndex and ipmid receives the respective
215devIndex.ipmid responds based on the received devIndex(Host Id) and
216response reaches all the way to host.The data can be validated in host.
217