1# phosphor-audit 2 3Author: Ivan Mikhaylov [i.mikhaylov@yadro.com](mailto:i.mikhaylov@yadro.com) 4 5Other contributors: 6 7- Alexander Amelkin [a.amelkin@yadro.com](mailto:a.amelkin@yadro.com) 8- Alexander Filippov [a.filippov@yadro.com](mailto:a.filippov@yadro.com) 9 10Created: 2019-07-23 11 12## Problem Description 13 14End users of OpenBMC may take actions that change the system state and/or 15configuration. Such actions may be taken using any of the numerous interfaces 16provided by OpenBMC. That includes RedFish, IPMI, ssh or serial console shell, 17and other interfaces, including the future ones. 18 19Consequences of those actions may sometimes be harmful and an investigation may 20be conducted in order to find out the person responsible for the unwelcome 21changes. Currently, most changes leave no trace in OpenBMC logs, which hampers 22the aforementioned investigation. 23 24It is required to develop a mechanism that would allow for tracking such user 25activity, logging it, and taking certain actions if necessary. 26 27## Background and References 28 29YADRO had an internal solution for the problem. It was only applicable to an 30outdated version of OpenBMC and needed a redesign. There was also a parallel 31effort by IBM that can be found here: 32[REST and Redfish Traffic Logging](https://gerrit.openbmc.org/c/openbmc/bmcweb/+/22699) 33 34## Assumptions 35 36This design assumes that an end user is never given a direct access to the 37system shell. The shell allows for direct manipulation of user database 38(add/remove users, change passwords) and system configuration (network scripts, 39etc.), and it doesn't seem feasible to track such user actions taken within the 40shell. This design assumes that all user interaction with OpenBMC is limited to 41controlled interfaces served by other Phosphor OpenBMC components interacting 42via D-Bus. 43 44## Requirements 45 46- Provide a unified method of logging user actions independent of the user 47 interface, where possible user actions are: 48 - Redfish/REST PUT/POST/DELETE/PATCH 49 - IPMI 50 - PAM 51 - PLDM 52 - Any other suitable service 53- Provide a way to configure system response actions taken upon certain user 54 actions, where possible response actions are: 55 - Log an event 56 - Notify an administrator or an arbitrary notification receiver 57 - Run an arbitrary command 58- Provide a way to configure notification receivers: 59 - E-mail 60 - SNMP 61 - Instant messengers 62 - D-Bus 63 64## Proposed Design 65 66The main idea is to catch D-Bus requests sent by user interfaces, then handle 67the request according to the configuration. In future, support for flexible 68policies may be implemented that would allow for better flexibility in handling 69and tracking. 70 71The phosphor-audit service represents a service that provides user activity 72tracking and corresponding action taking in response of user actions. 73 74The key benefit of using phosphor-audit is that all action handling will be kept 75inside this project instead of spreading it across multiple dedicated interface 76services with a risk of missing a handler for some action in one of them and 77bloating the codebase. 78 79The component diagram below shows the example of service overview. 80 81```ascii 82 +----------------+ audit event +-----------------+ 83 | IPMI NET +-----------+ | action | 84 +----------------+ | | +-------------+ | 85 | | | logging | | 86 +----------------+ | | +-------------+ | 87 | IPMI HOST +-----------+ +--------------+ | | 88 +----------------+ | | audit | | +-------------+ | 89 +----->+ service +----->| | command | | 90 +----------------+ | | | | +-------------+ | 91 | RedFish/REST +-----------+ +--------------+ | | 92 +----------------+ | | +-------------+ | 93 | | | notify | | 94 +----------------+ | | +-------------+ | 95 | any service +-----------+ | | 96 +----------------+ | +-------------+ | 97 | | ... | | 98 | +-------------+ | 99 +-----------------+ 100``` 101 102The audit event from diagram generated by an application to track user activity. 103The application sends 'signal' to audit service via D-Bus. What is happening 104next in audit service's handler depends on user requirements and needs. It is 105possible to just store logs, run arbitrary command or notify someone in handler 106or we can do all of the above and all of this can be optional. 107 108### Audit event call 109 110Audit event call performs preprocessing of incoming data at application side 111before sending it to the audit service, if the request is filtered out, it will 112be dropped at this moment and will no longer be processed. After the filter 113check, the audit event call sends the data through D-Bus to the audit service 114which makes a decision regarding next steps. Also, it caches list of possible 115commands (blacklist or whitelist) and status of its service (disabled or 116enabled). If the service in undefined state, the call checks if service alive or 117not. 118 119> `audit_event(type, rc, request, user, host, data)` 120> 121> - type - type of event source : IPMI, REST, PAM, etc. 122> - rc - return code of the handler event (status, rc, etc.) 123> - request - a generalized identifier of the event, e.g. ipmi command 124> (cmd/netfn/lun), web path, or anything else that can describe the event. 125> - user - the user account on behalf of which the event was processed. depends 126> on context, NA/None in case of user inaccessibility. 127> - source - identifier of the host that the event has originated from. This can 128> be literally "host" for events originating from the local host (via locally 129> connected IPMI), or an IP address or a hostname of a remote host. 130> - data - any supplementary data that can help better identify the event (e.g., 131> some first bytes of the IPMI command data). 132 133Service itself can control flow of events with configuration on its side. 134 135Pseudocode for example: 136 137```cpp 138 audit_event(NET_IPMI, "access denied"(rc=-1), "ipmi cmd", "qwerty223", 139 "192.168.0.1", <some additional data if needed>) 140 audit_event(REST, "login successful"(rc=200), "rest login", 141 "qwerty223", "192.168.0.1", NULL) 142 audit_event(HOST_IPMI, "shutting down the host"(rc=0), "host poweroff", 143 NULL, NULL, NULL) 144``` 145 146`audit_event(blob_data)` Blob can be described as structure: 147 148```cpp 149 struct blob_audit 150 { 151 uint8_t type; 152 int32_t rc; 153 uint32_t request_id; 154 char *user; 155 sockaddr_in6 *addr; 156 struct iovec *data; 157 } 158``` 159 160When the call reaches the server destination via D-Bus, the server already knows 161that the call should be processed via predefined list of actions which are set 162in the server configuration. 163 164Step by step execution of call: 165 166- client's layer 167 1. checks if audit is enabled for such service 168 2. checks if audit event should be whitelisted or blacklisted at the audit 169 service side for preventing spamming of unneeded events to audit service 170 3. send the data to the audit service via D-Bus 171- server's layer 172 1. accept D-Bus request 173 2. goes through list of actions for each services 174 175How the checks will be processed at client's layer: 176 1771. check the status of service and cache that value 1782. check the list of possible actions which should be logged and cache them also 1793. listen on 'propertiesChanged' event in case of changing list or status of 180 service 181 182## Service configuration 183 184The configuration structure can be described as tree with set of options, as 185example of structure: 186 187```text 188[IPMI] 189 [Enabled] 190 [Whitelist] 191 [Cmd 0x01] ["reset request"] 192 [Cmd 0x02] ["hello world"] 193 [Cmd 0x03] ["goodbye cruel world"] 194 [Actions] 195 [Notify type1] [Recipient] 196 [Notify type2] [Recipient] 197 [Notify type3] [Recipient] 198 [Logging type] [Options] 199 [Exec] [ExternalCommand] 200[REST] 201 [Disabled] 202 [Blacklist] 203 [Path1] [Options] 204 [Path2] [Options] 205 [Actions] 206 [Notify type2] [Recipient] 207 [Logging type] [Options] 208``` 209 210Options can be updated via D-Bus properties. The audit service listens changes 211on configuration file and emit 'PropertiesChanged' signal with changed details. 212 213- The whitelisting and blacklisting 214 215> Possible list of requests which have to be filtered and processed. 'Whitelist' 216> filters possible requests which can be processed. 'Blacklist' blocks only 217> exact requests. 218 219- Enable/disable the event processing for directed services, where the directed 220 service is any suitable services which can use audit service. 221 222> Each audit processing type can be disabled or enabled at runtime via config 223> file or D-Bus property. 224 225- Notification setup via SNMP/E-mail/Instant messengers/D-Bus 226 227> The end recipient notification system with different transports. 228 229- Logging 230 231> phosphor-logging, journald or anything else suitable for. 232 233- User actions 234 235> Running a command as consequenced action. 236 237## Workflow 238 239An example of possible flow: 240 241```ascii 242 +----------------+ 243 | NET IPMI | 244 | REQUEST | 245 +----------------+ 246 | 247 +--------------------------------------------------------------------------+ 248 | +-------v--------+ IPMI | 249 | | NET IPMI | | 250 | +----------------+ | 251 | | | 252 | +-------v--------+ +---------------------------+ | 253 | | rc = handle() +------->| audit_event<NET_IPMI>() | | 254 | +----------------+ +---------------------------+ | 255 | | | | 256 | | | | 257 | +-------v--------+ | | 258 | | Processing | | | 259 | | further | | | 260 | +----------------+ | | 261 +--------------------------------------------------------------------------+ 262 | 263 | 264 +--------------------------------------------------------------------------+ 265 | +-----------------------------+ | 266 | | Audit Service | 267 | | | 268 | | | 269 | | | 270 | +-----v------+ | 271 | NO | Is logging | YES | 272 | +------+ enabled +--------------------+ | 273 | | | for type? | | | 274 | | +------------+ +-------v-----+ | 275 | | NO | Is request | YES | 276 | | +--------+ type +--------+ | 277 | | | | filtered? | | | 278 | | | +-------------+ | | 279 | | | | | 280 | | +-------v-------+ | | 281 | | | Notify | | | 282 | | | Administrator | | | 283 | | +---------------+ | | 284 | | | | | 285 | | +-------v-------+ | | 286 | | | Log Event | | | 287 | | +---------------+ | | 288 | | | | | 289 | | +-------v-------+ | | 290 | | | User | | | 291 | | | actions | | | 292 | | +---------------+ | | 293 | | | | | 294 | | +-------v-------+ | | 295 | +-------------->| End |<----------------------+ | 296 | +---------------+ | 297 | | 298 +--------------------------------------------------------------------------+ 299``` 300 301## Notification mechanisms 302 303The unified model for reporting accidents to the end user, where the transport 304can be: 305 306- E-mail 307 308 > Sending a note to directed recipient which set in configuration via sendmail 309 > or anything else. 310 311- SNMP 312 313 > Sending a notification via SNMP trap messages to directed recipient which 314 > set in configuration. 315 316- Instant messengers 317 318 > Sending a notification to directed recipient which set in configuration via 319 > jabber/sametime/gtalk/etc. 320 321- D-Bus 322 323 > Notify the other service which set in configuration via 'method_call' or 324 > 'signal'. 325 326Notifications will be skipped in case if there is no any of above configuration 327rules is set inside configuration. It is possible to pick up rules at runtime. 328 329## User Actions 330 331- Exec application via 'system' call. 332- The code for directed handling type inside handler itself. As example for 'net 333 ipmi' in case of unsuccessful user login inside handler: 334 - Sends a notification to administrator. 335 - echo heartbeat > /sys/class/leds/alarm_red/trigger 336 337## Alternatives Considered 338 339Processing user requests in each dedicated interface service and logging them 340separately for each of the interfaces. Scattered handling looks like an 341error-prone and rigid approach. 342 343## Impacts 344 345Improves system manageability and security. 346 347Impacts when phosphor-audit is not enabled: 348 349- Many services will have slightly larger code size and longer CPU path length 350 due to invocations of audit_event(). 351- Increased D-Bus traffic. 352 353Impacts when phosphor-audit is enabled: All of the above, plus: 354 355- Additional BMC processor time needed to handle audit events. 356- Additional BMC flash storage needed to store logged events. 357- Additional outbound network traffic to notify users. 358- Additional space for notification libraries. 359 360## Testing 361 362`dbus-send` as command-line tool for generating audit events. 363 364Scenarios: 365 366- For each supported service (such as Redfish, net IPMI, host IPMI, PLDM), 367 create audit events, and validate they get logged. 368- Ensure message-type and request-type filtering works as expected. 369- Ensure basic notification actions work as expected (log, command, notify). 370- When continuously generating audit-events, change the phosphor-audit service's 371 configuration, and validate no audit events are lost, and the new 372 configuration takes effect. 373