openbmc/openpower-host-ipmi-flash/hiomap.cpp

1 // SPDX-License-Identifier: Apache-2.0
7 #include <ipmid/api.h>
10 #include <systemd/sd-bus.h>
11 #include <systemd/sd-event.h>
13 #include <ipmid-host/cmd-utils.hpp>
14 #include <ipmid-host/cmd.hpp>
15 #include <ipmid/api.hpp>
16 #include <phosphor-logging/log.hpp>
17 #include <sdbusplus/bus.hpp>
18 #include <sdbusplus/bus/match.hpp>
37 The primary motivation of the Host I/O Mapping protocol (HIOMAP) is to mediate
38 host access to a BMC-controlled flash chip housing the host's boot firmware.
40 openpower-host-ipmi-flash facilitates the system design of transporting the
41 HIOMAP protocol[1] over IPMI. This is somewhat abusive of IPMI, basically
42 treating the BT interface as a mailbox with an interrupt each way between the
43 BMC and the host.
47 Using IPMI in this way has a number of challenges, a lot of them on the host
48 side where we need to bring up the LPC and BT interfaces to enable IPMI before
49 accessing the flash, and before any interrupts are enabled. There are also
50 challenges on the BMC side with the design of the current implementation. We
53 BMC-side System Design and Integration Issues
54 ---------------------------------------------
56 The current design is that we have the HIOMAP daemon, mboxd (to be renamed),
57 exposing a set of DBus interfaces. Whilst the spec defines the IPMI transport
58 message packing, mboxd knows nothing of IPMI itself, instead relying on the
59 DBus interface to receive messages from ipmid. ipmid in-turn knows nothing of
60 the interfaces communicating with it, also relying on DBus to receive messages
61 from interface-specific daemons, e.g. btbridged[2].
65 For this design to function correctly we must ensure that the daemons are
69 1. systemd uses Wants=/Before=/After= relationships in units to define both
70    start-up *and* shutdown order, in stack push / pop order respectively.
75    PropertiesChanged signal to notify of state changes.
79 grief with shutdown of the BMC, as mboxd needs to issue a state-change
80 notification when it is shut down to inform the host that will not respond to
81 future requests and that the protocol state has been reset. If mboxd has a
82 Wants=/Before= relationship with ipmid this message will never propagate to the
83 host, as ipmid will be shut by systemd before mboxd.
85 The above leads to mboxd having a Wants=/After= relationship with ipmid. This
86 ensures that if mboxd is restarted on its own the correct state changes will be
87 propagated to the host. The case where ipmid attempts to call into mboxd's DBus
88 interface before mboxd is ready is mitigated by the ready bit in the protocol's
93 no control over the *scheduling* of processes to ensure the PropertiesChanged
97   On the basis that mboxd has a Wants=/After= relationship with ipmid,
98   openpower-host-ipmi-flash will emit an HIOMAP BMC status event to the host
99   with the value BMC_EVENT_PROTOCOL_RESET upon receiving SIGTERM iff the BMC
102 If ipmid has received SIGTERM the assumption is that it is systemd that is
103 sending it, and that the Wants=/After= relationship requires that mboxd has
105 openpower-host-ipmi-flash emits the BMC event state we close the race where the
106 host is not informed of the termination of mboxd due to scheduling ipmid (to
107 deliver SIGTERM) prior to scheduling dbus-daemon, where the PropertiesChanged
110 Observations on the IPMI Specification and Design Details of ipmid
111 ------------------------------------------------------------------
113 In addition to the system-level design problems with delivering
116 delivered to the host.
118 The first necessary observation is that the mechanism for delivering BMC state
119 change events from mboxd to the host over IPMI uses the SMS ATN bit to indicate
120 a message is ready for delivery from the BMC to the host system. Retrieving the
121 BMC state data involves the host recognising that the SMS ATN bit is set,
122 performing Get Message Flags transaction with the BMC followed by a subsequent
123 Get Message transaction. Thus, delivery of the HIOMAP protocol's BMC status is
126 The second necessary observation is that the kernel delivers signals
128 ipmid can win the race against SIGTERM to receive the PropertiesChanged event
129 from mboxd, but lose the race to complete delivery to the host.
131   On this basis, we need to block the delivery of SIGTERM to ipmid until ipmid
132   has completed the set of `SMS ATN`/`Get Message Flags`/`Get Message`
133   transactions with the host
137 better approach that avoids the need for global application state is to simply
138 block the signal until we are ready to handle it, which we can do via
141 The existing design of ipmid makes it feasible to block and unblock
144 phosphor::host::command::Manager::getNextCommand(). The documentation for
147   @brief  Extracts the next entry in the queue and returns
148           Command and data part of it.
150   @detail Also calls into the registered handlers so that they can now
151           send the CommandComplete signal since the interface contract
152           is that we emit this signal once the message has been
153           passed to the host (which is required when calling this)
155           Also, if the queue has more commands, then it will alert the
158 However, its description is not entirely accurate. The callback function is
159 invoked when ipmid *dequeues* the data to send to the host: Delivery of the
160 data to the host occurs at some *after* the callback has been invoked.
162 Invoking the callback before completion of delivery of the data to the host
163 nullifies the approach of unblocking asynchronous SIGTERM in the callback
164 associated with sending the HIOMAP BMC state event to the host, as the BMC
165 kernel can asynchronously terminate the process between the callback being
166 invoked and the host receiving the BMC state event data.
168 Overcoming this issue hinges on a significant implementation detail of ipmid:
170   ipmid uses an sd_event loop in the main function to pump DBus events.
175   of Linux's signalfd(2) interface.
177 The fact that sd_event is used to pump DBus events means that ipmid can remain
178 a single-threaded process. By remaining single-threaded we know that events
180 corollary of this is that DBus events and UNIX signals are serialised with
183 The fourth necessary observation is that we do not need to pump sd_event in
184 order to complete DBus method calls; sd_bus will handle the pumping independent
185 of the main loop in order to complete the method invocation.
188 ------------------------------------------------------
190 We achieve reliable delivery of HIOMAP BMC status events in the following way:
194    openpower::flash::hiomap_protocol_reset() as the SIGTERM handler using
196 3. openpower::flash::hiomap_protocol_reset() implements the logic to send the
197    BMC_EVENT_PROTOCOL_RESET state to the host if necessary, otherwise terminate
198    the sd_event loop.
199 4. If it is necessary to send BMC_EVENT_PROTOCOL_RESET to the host in 3, assign
200    a callback handler that terminates the sd_event loop, which is only
201    processed after the current iteration is complete.
203 This process and its use of signalfd integration in the sd_event loop
204 eliminates the following three races:
206 1. The scheduler race between mboxd, dbus-daemon and ipmid, by having
207    openpower-host-ipmi-flash conditionally deliver the protocol reset event if
209 2. The race between delivering the BMC status event to the host and ipmid
210    receiving asynchronous SIGTERM after receiving the PropertiesChanged event
212 3. The race to deliver the BMC status data to the host after unblocking
213    asynchronous SIGTERM in the host command callback and before receiving
216 Ultimately, ipmid could benefit from a redesign that fires the callback *after*
217 delivering the associated data to the host, but brief inspection determined
218 that this involved a non-trivial amount of effort.
245 /* XXX: ipmid is currently single-threaded, pumping dbus events in sequence
246  * via the main event loop. Thus the code is not forced to be re-entrant. We
247  * also know that the callback and DBus event handling will not be running
251  * pointer, so instead use a global. active_event_updates gates manipulation of
258     bus_t* bus;  member
261     bus::match_t* properties;
299     {ETIMEDOUT, 0xc3}, /* FIXME: Replace when defined in ipmid-api.h */
300     {ENOSPC, 0xc4},    /* FIXME: Replace when defined in ipmid-api.h */
305     {-1, IPMI_CC_UNSPECIFIED_ERROR},
312     while (!(entry->err == err || entry->err == -1))  in hiomap_xlate_errno()
317     return entry->cc;  in hiomap_xlate_errno()
331     active_event_updates--;  in ipmi_hiomap_event_response()
359         if (!ctx->event_lookup.count(x.first))  in hiomap_handle_property_update()
365         uint8_t mask = ctx->event_lookup[x.first];  in hiomap_handle_property_update()
370             ctx->bmc_events |= mask;  in hiomap_handle_property_update()
374             ctx->bmc_events &= ~mask;  in hiomap_handle_property_update()
378     auto cmd = std::make_pair(IPMI_CMD_HIOMAP_EVENT, ctx->bmc_events);  in hiomap_handle_property_update()
389     // the event queue as the last signal handler  in hiomap_protocol_reset_response()
396     if (ctx->bmc_events == BMC_EVENT_PROTOCOL_RESET)  in hiomap_protocol_reset()
399         // the event queue as the last signal handler  in hiomap_protocol_reset()
405      * Send an attention indicating the hiomapd has died  in hiomap_protocol_reset()
406      * (BMC_EVENT_DAEMON_READY cleared) and that the protocol has been reset  in hiomap_protocol_reset()
407      * (BMC_EVENT_PROTOCOL_RESET set) to indicate to the host that it needs to  in hiomap_protocol_reset()
408      * wait for the BMC to come back and renegotiate the protocol.  in hiomap_protocol_reset()
410      * We know this to be the case in systems that integrate  in hiomap_protocol_reset()
411      * openpower-host-ipmi-flash, as hiomapd's unit depends on  in hiomap_protocol_reset()
412      * phosphor-ipmi-host, and thus hiomapd has been terminated before ipmid  in hiomap_protocol_reset()
423 static bus::match_t hiomap_match_properties(struct hiomap* ctx)  in hiomap_match_properties()
426         bus::match::rules::propertiesChanged(HIOMAPD_OBJECT, HIOMAPD_IFACE_V2);  in hiomap_match_properties()
428     bus::match_t match(  in hiomap_match_properties()
429         *ctx->bus, properties,  in hiomap_match_properties()
441     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_reset()
445         ctx->bus->call(m);  in hiomap_reset()
469     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_get_info()
475         auto reply = ctx->bus->call(m);  in hiomap_get_info()
505     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_get_flash_info()
509         auto reply = ctx->bus->call(m);  in hiomap_get_flash_info()
540     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_create_window()
547         auto reply = ctx->bus->call(m);  in hiomap_create_window()
599     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_close_window()
605         auto reply = ctx->bus->call(m);  in hiomap_close_window()
629     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_mark_dirty()
637         auto reply = ctx->bus->call(m);  in hiomap_mark_dirty()
655     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_flush()
661         auto reply = ctx->bus->call(m);  in hiomap_flush()
685     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_ack()
692         auto reply = ctx->bus->call(m);  in hiomap_ack()
716     auto m = ctx->bus->new_method_call(HIOMAPD_SERVICE, HIOMAPD_OBJECT,  in hiomap_erase()
724         auto reply = ctx->bus->call(m);  in hiomap_erase()
761 /* FIXME: Define this in the "right" place, wherever that is */
782     if (hiomap_cmd == 0 || hiomap_cmd > hiomap_commands.size() - 1)  in hiomap_dispatch()
791     if (!is_unversioned && ctx->seq == ipmi_req[1])  in hiomap_dispatch()
797     ctx->seq = ipmi_req[1];  in hiomap_dispatch()
800     size_t flash_len = *data_len - 2;  in hiomap_dispatch()
809     ipmi_ret_t cc = command->second(flash_req, flash_resp, &flash_len, context);  in hiomap_dispatch()
816     /* Populate the response command and sequence */  in hiomap_dispatch()
818     ipmi_resp[1] = ctx->seq;  in hiomap_dispatch()
832     struct hiomap* ctx = new hiomap();  in register_openpower_hiomap_commands()
835     ctx->event_lookup["DaemonReady"] = BMC_EVENT_DAEMON_READY;  in register_openpower_hiomap_commands()
836     ctx->event_lookup["FlashControlLost"] = BMC_EVENT_FLASH_CTRL_LOST;  in register_openpower_hiomap_commands()
837     ctx->event_lookup["WindowReset"] = BMC_EVENT_WINDOW_RESET;  in register_openpower_hiomap_commands()
838     ctx->event_lookup["ProtocolReset"] = BMC_EVENT_PROTOCOL_RESET;  in register_openpower_hiomap_commands()
840     ctx->bus = new bus_t(ipmid_get_sd_bus_connection());  in register_openpower_hiomap_commands()
846      * destruction (!?), so enjoy the weird wrapping.  in register_openpower_hiomap_commands()
848     ctx->properties = new bus::match_t(std::move(hiomap_match_properties(ctx)));  in register_openpower_hiomap_commands()