1# Multi-host IPMI design
2
3Author: Velumani T(velu), [velumanit@hcl](mailto:velumanit@hcl.com) Kumar
4T(kumar_t), [thangavel.k@hcl.com](mailto:thangavel.k@hcl.com)
5
6Other contributors:
7
8Created: June 26, 2020
9
10## Problem Description
11
12The current version of OpenBMC does not support multi-host implementation in
13IPMI commands handling. We have a multi-host system and proposing the design to
14support multi-host.
15
16As detailed below the hosts are connected in the IPMB interface, all host
17related communication is based on IPMB. The OpenBMC uses ipmbbridged to manage
18IPMB buses and the IPMB messages are routed to ipmid.
19
20Issue 1: ipmbridged does not send the channel number (ie HostId) Issue 2: ipmid
21does not have the information on which IPMB channel the request has come from.
22The ipmid handlers should have the host details to fetch the host specific
23responses.
24
25## Background and References
26
27IPMI and IPMB System architecture:
28
29```
30 +------------------------------------+
31 |               BMC                  |
32 | +-----------+       +------------+ |      +--------+
33 | |           |       |            | | IPMB1|        |
34 | |           |       |            |-|------| Host-1 |
35 | |           |       |            | |      |        |
36 | |           |       |            | |      +--------+
37 | |           |       |            | |
38 | |           |       |            | |
39 | |           | D-Bus |            | |      +--------+
40 | | ipmid     |-------| ipmbbridged| | IPMB2|        |
41 | |           |       |            |-|------| Host-2 |
42 | |           |       |            | |      |        |
43 | |           |       |            | |      +--------+
44 | |           |       |            | |
45 | |           |       |            | |
46 | |           |       |            | |      +--------+
47 | |           |       |            | | IPMBn|        |
48 | |           |       |            |-|------| Host-N |
49 | |           |       |            | |      |        |
50 | +-----------+       +------------+ |      +--------+
51 +------------------------------------+
52```
53
54Hosts are connected with IPMB interface, the hosts can be 1 to N. The IPMB
55request coming from the hosts are routed to ipmid by the ipmbbridged. The IPMB
56requests are routed from ipmid or any service by D-Bus interface and The
57outgoing IPMB requests are routed by ipmbbridged to IPMB interface.
58
59## Requirements
60
61The current version of OpenBMC does not support multi-host implementation in
62IPMI commands handling. We have a multi-host system and proposing the design to
63support multi-host.
64
65## Proposed Design
66
67To address issue1 and issue2, we propose the following design changes in
68ipmbbridged and ipmid. To address out-of-band IPMI command from the network,the
69proposal is captured in section "Changes in netipmid".
70
71## Issue1: Changes in ipmbbridged:
72
73ipmbbridged to send the channel details from where the request is received.
74
75**Change: Sending Host detail as additional parameter**
76
77While routing the IPMB requests coming from the host channel, We will be adding
78new entry in the json config file for the host ID '"devIndex": 0' ipmbbridged
79will send '"devIndex": 0' as optional parameter(options) in D-Bus interface to
80ipmid.This can be used to get the information on which IPMB bus the message
81comes from.
82
83The json file looks like below. Each devIndex can have one "me" and "ipmb"
84channel.To ensure existing platforms does not get affected, if the "devIndex"
85key is not present in the file ipmbbridged uses default "devIndex" as 0.
86
87{ "type": "me", "slave-path": "/dev/ipmb-1", "bmc-addr": 32, "remote-addr": 64,
88"devIndex": 0 }, { "type": "ipmb", "slave-path": "/dev/ipmb-2", "bmc-addr": 32,
89"remote-addr": 64, "devIndex": 0 }, { "type": "me", "slave-path": "/dev/ipmb-3",
90"bmc-addr": 32, "remote-addr": 64, "devIndex": 1 }, { "type": "ipmb",
91"slave-path": "/dev/ipmb-4", "bmc-addr": 32, "remote-addr": 64, "devIndex": 1 },
92
93## Issue2: Changes in ipmid:
94
95Receive the optional parameter sent by the ipmbbridged as host details, while
96receiving the parameter in the executionEntry method call the same will be
97passed to the command handlers in both common and oem handlers.The command
98handlers can make use of the host information to fetch host specific data.
99
100For example, host1 send a request to get boot order from BMC, BMC maintains data
101separately for each host. When this command comes to ipmid the commands handler
102gets the host in which the command received. The handler will fetch host1 boot
103order details and respond from the command handler. This is applicable for both
104common and oem handlers.
105
106## Changes in netipmid:
107
108The "options" parameter can be used for sending the host information from
109netipmid. The changes proposed for ipmbbridged can be used in netipmid as well.
110The netipmid sends the "devIndex" on which channel the request comes from. There
111will not be any further changes required in ipmid. The netipmid can have
112multiple approaches to handle multi-host.Some of the approaches are listed down
113and but not limited to this list.
114
1151. Virtual Ethernet interfaces - One virtual interface per host.
1162. Different port numbers - Can have different port numbers for each host.
1173. VLAN Ids- VLAN IDs can be used to support multi host. The netipmid shall have
118   a config file where in the interfaces can be configured for each host. Also
119   one or more interfaces can be configured to each of the host. The interfaces
120   can be virtual or physical. Below is the sample configuration file
121
122{"Host":1, "Interface-1":"eth0", "Interface-2":"eth1", "Interface-3":"veth4",
123"Interface-4":"veth5" }, {"Host":2, "Interface-1":"eth2", "Interface-2":"eth3",
124"Interface-3":"veth1", "Interface-4":"veth2" },
125
126Example implementation of approach1:Virtual ethernet interface.
127
128```
129+--------------------------------------------+
130|           BMC                              |
131| +--------+       +-----------+   +------+  |      +--------+
132| |        | D-Bus |           |   |      |  |      |        |
133| |        |-------| netipmid1 |---|veth1 |---------| Host-1 |
134| |        |       |           |   |      |  |      |        |
135| |        |       +-----------+   +------+  |      +--------+
136| |        |                                 |
137| |        |       +-----------+   +------+  |      +--------+
138| | ipmid  | D-Bus |           |   |      |  |      |        |
139| |        |-------| netipmid2 |---|veth2 |---------| Host-2 |
140| |        |       |           |   |      |  |      |        |
141| |        |       +-----------+   +------+  |      +--------+
142| |        |                                 |
143| |        |       +-----------+   +------+  |      +--------+
144| |        | D-Bus |           |   |      |  |      |        |
145| |        |-------| netipmidN |---|vethN |---------| Host-N |
146| |        |       |           |   |      |  |      |        |
147| +--------+       +-----------+   +------+  |      +--------+
148+--------------------------------------------+
149```
150
151In the above diagram one instance of netipmid runs per host. Each instance is
152tied to one virtual ethernet interface, The virtual interface ID can be used to
153make a devIndex. This represents the HostId.
154
155## Alternatives Considered
156
157## Approach1:ipmbbridged to send host-id in the payload
158
159The ipmbbridged shall be modified to send the host id in data payload. This
160looks to be a simple change but impacts the existing platforms which are already
161using it.This may not be a right approach.
162
163## Approach2:Create multiple ipmid to handle multihost.One ipmid process per host.
164
165This is a multi service appoach,one instance of ipmid service shall be spawned
166to respond each host.The changes looks simple and no major design change from
167the existing design. But many common handlers will be running as duplicate in
168multiple instances.
169
170## Approach3:Using a different IPMI channel for handling multiple host.
171
172Using a different IPMI channel for handling multiple hosts, in the ipmbbridged
173the channel id can be used to identify host. In this approach we will be having
174multiple instances of ipmbbridged and each instance will be registered with the
175a channel number.Maximum channel numbers are limited to 8 as per the
176specification.This limits the maximum hosts to be supported.
177
178## Impacts
179
180There may not be an impact in ipmid command handler functions.This design will
181not affect the current functionality.
182
183## Testing
184
185The proposed design can be tested in a platform in which the multiple hosts are
186connected.The commands requests are received from all the hosts and responses
187are host specific data.When the request coming from the host as IPMB command,
188ipmbbridged appends devIndex and ipmid receives the respective devIndex.ipmid
189responds based on the received devIndex(Host Id) and response reaches all the
190way to host.The data can be validated in host.
191