1# Multi-host IPMI design
2
3Author:
4  Velumani T(velu),  [velumanit@hcl](mailto:velumanit@hcl.com)
5  Kumar T(kumar_t), [thangavel.k@hcl.com](mailto:thangavel.k@hcl.com)
6
7Primary assignee:
8
9Other contributors:
10
11Created:
12 June 26, 2020
13
14## Problem Description
15The current version of OpenBMC does not support multi-host implementation in
16IPMI commands handling. We have a multi-host system and proposing the design
17to support multi-host.
18
19As detailed below the hosts are connected in the IPMB interface, all host
20related communication is based on IPMB. The OpenBMC uses ipmbbridged to manage
21IPMB buses and the IPMB messages are routed to ipmid.
22
23Issue 1: ipmbridged does not send the channel number (ie HostId)
24Issue 2: ipmid does not have the information on which IPMB channel the request
25has come from. The ipmid handlers should have the host details to fetch the
26host specific responses.
27
28## Background and References
29IPMI and IPMB System architecture:
30```
31 +------------------------------------+
32 |               BMC                  |
33 | +-----------+       +------------+ |      +--------+
34 | |           |       |            | | IPMB1|        |
35 | |           |       |            |-|------| Host-1 |
36 | |           |       |            | |      |        |
37 | |           |       |            | |      +--------+
38 | |           |       |            | |
39 | |           |       |            | |
40 | |           | D-Bus |            | |      +--------+
41 | | ipmid     |-------| ipmbbridged| | IPMB2|        |
42 | |           |       |            |-|------| Host-2 |
43 | |           |       |            | |      |        |
44 | |           |       |            | |      +--------+
45 | |           |       |            | |
46 | |           |       |            | |
47 | |           |       |            | |      +--------+
48 | |           |       |            | | IPMBn|        |
49 | |           |       |            |-|------| Host-N |
50 | |           |       |            | |      |        |
51 | +-----------+       +------------+ |      +--------+
52 +------------------------------------+
53```
54Hosts are connected with IPMB interface, the hosts can be 1 to N. The IPMB
55request coming from the hosts are routed to ipmid by the ipmbbridged.
56The IPMB requests are routed from ipmid or any service by D-Bus interface and
57The outgoing IPMB requests are routed by ipmbbridged to IPMB interface.
58
59## Requirements
60
61The current version of OpenBMC does not support multi-host implementation in
62IPMI commands handling. We have a multi-host system and proposing the design
63to support multi-host.
64
65## Proposed Design
66
67To address issue1 and issue2, we propose the following design changes in
68ipmbbridged and ipmid.
69To address out-of-band IPMI command from the network,the proposal is captured
70in section "Changes in netipmid".
71
72Issue1: Changes in ipmbbridged:
73-
74ipmbbridged to send the channel details from where the request is received.
75
76**Change: Sending Host detail as additional parameter**
77
78While routing the IPMB requests coming from the host channel, We will be
79adding new entry in the json config file for the host ID '"devIndex": 0'
80ipmbbridged will send '"devIndex": 0' as optional parameter(options) in D-Bus
81interface to ipmid.This can be used to get the information on which IPMB bus
82the message comes from.
83
84The json file looks like below. Each devIndex can have one "me" and "ipmb"
85channel.To ensure existing platforms does not get affected, if the "devIndex"
86key is not present in the file ipmbbridged uses default "devIndex" as 0.
87
88{ "type": "me",
89"slave-path": "/dev/ipmb-1",
90"bmc-addr": 32,
91"remote-addr": 64,
92"devIndex": 0
93},
94{ "type": "ipmb",
95"slave-path": "/dev/ipmb-2",
96"bmc-addr": 32,
97"remote-addr": 64,
98"devIndex": 0
99},
100{ "type": "me",
101"slave-path": "/dev/ipmb-3",
102"bmc-addr": 32,
103"remote-addr": 64,
104"devIndex": 1
105},
106{ "type": "ipmb",
107"slave-path": "/dev/ipmb-4",
108"bmc-addr": 32,
109"remote-addr": 64,
110"devIndex": 1
111},
112
113Issue2: Changes in ipmid:
114-
115Receive the optional parameter sent by the ipmbbridged as host details, while
116receiving the parameter in the executionEntry method call the same will be
117passed to the command handlers in both common and oem handlers.The command
118handlers can make use of the host information to fetch host specific data.
119
120For example, host1 send a request to get boot order from BMC, BMC maintains
121data separately for each host. When this command comes to ipmid the commands
122handler gets the host in which the command received. The handler will fetch
123host1 boot order details and respond from the command handler. This is
124applicable for both common and oem handlers.
125
126Changes in netipmid:
127-
128The "options" parameter can be used for sending the host information from
129netipmid. The changes proposed for ipmbbridged can be used in netipmid as well.
130The netipmid sends the "devIndex" on which channel the request comes from.
131There will not be any further changes required in ipmid.
132The netipmid can have multiple approaches to handle multi-host.Some of the
133approaches are listed down and but not limited to this list.
1341. Virtual Ethernet interfaces - One virtual interface per host.
1352. Different port numbers - Can have different port numbers for each host.
1363. VLAN Ids- VLAN IDs can be used to support multi host.
137The netipmid shall have a config file where in the interfaces can be configured
138for each host. Also one or more interfaces can be configured to each of the
139host. The interfaces can be virtual or physical.
140Below is the sample configuration file
141
142{"Host":1,
143"Interface-1":"eth0",
144"Interface-2":"eth1",
145"Interface-3":"veth4",
146"Interface-4":"veth5"
147},
148{"Host":2,
149"Interface-1":"eth2",
150"Interface-2":"eth3",
151"Interface-3":"veth1",
152"Interface-4":"veth2"
153},
154
155Example implementation of approach1:Virtual ethernet interface.
156
157```
158+--------------------------------------------+
159|           BMC                              |
160| +--------+       +-----------+   +------+  |      +--------+
161| |        | D-Bus |           |   |      |  |      |        |
162| |        |-------| netipmid1 |---|veth1 |---------| Host-1 |
163| |        |       |           |   |      |  |      |        |
164| |        |       +-----------+   +------+  |      +--------+
165| |        |                                 |
166| |        |       +-----------+   +------+  |      +--------+
167| | ipmid  | D-Bus |           |   |      |  |      |        |
168| |        |-------| netipmid2 |---|veth2 |---------| Host-2 |
169| |        |       |           |   |      |  |      |        |
170| |        |       +-----------+   +------+  |      +--------+
171| |        |                                 |
172| |        |       +-----------+   +------+  |      +--------+
173| |        | D-Bus |           |   |      |  |      |        |
174| |        |-------| netipmidN |---|vethN |---------| Host-N |
175| |        |       |           |   |      |  |      |        |
176| +--------+       +-----------+   +------+  |      +--------+
177+--------------------------------------------+
178```
179In the above diagram one instance of netipmid runs per host. Each instance
180is tied to one virtual ethernet interface, The virtual interface ID can be
181used to make a devIndex. This represents the HostId.
182
183## Alternatives Considered
184
185Approach1:ipmbbridged to send host-id in the payload
186-
187The ipmbbridged shall be modified to send the host id in data payload. This
188looks to be a simple change but impacts the existing platforms which are already
189using it.This may not be a right approach.
190
191Approach2:Create multiple ipmid to handle multihost.One ipmid process per host.
192-
193This is a multi service appoach,one instance of ipmid service shall be
194spawned to respond each host.The changes looks simple and no major design
195change from the existing design. But many common handlers will be running as
196duplicate in multiple instances.
197
198Approach3:Using a different IPMI channel for handling multiple host.
199-
200Using a different IPMI channel for handling multiple hosts, in the ipmbbridged
201the channel id can be used to identify host. In this approach we will be having
202multiple instances of ipmbbridged and each instance will be registered with the
203a channel number.Maximum channel numbers are limited to 8 as per the
204specification.This limits the maximum hosts to be supported.
205
206## Impacts
207
208There may not be an impact in ipmid command handler functions.This design will
209not affect the current functionality.
210
211## Testing
212
213The proposed design can be tested in a platform in which the multiple hosts
214are connected.The commands requests are received from all the hosts and
215responses are host specific data.When the request coming from the host as IPMB
216command, ipmbbridged appends devIndex and ipmid receives the respective
217devIndex.ipmid responds based on the received devIndex(Host Id) and
218response reaches all the way to host.The data can be validated in host.
219