1### ECC Error SEL for BMC 2 3Author: Will Liang 4 5Created: 2019-02-26 6 7#### Problem Description 8 9The IPMI SELs only define memory Error Correction Code (ECC) errors for host 10memory rather than BMC. 11 12The aim of this proposal is to record ECC events from the BMC in the IPMI System 13Event Log (SEL). Whenever ECC occurs, the BMC generates an event with the 14appropriate information and adds it to the SEL. 15 16#### Background and References 17 18The IPMI specification defines memory system event log about ECC/other 19correctable or ECC/other uncorrectable and whether ECC/other correctable memory 20error logging limits are reached.[1]. The BMC ECC SEL will follow IPMI SEL 21format and creates BMC memory ECC event log. 22 23OpenBMC currently support for generating SEL entries based on parsing the D-Bus 24event log. It does not yet support the BMC ECC SEL feature in OpenBMC project. 25Therefore, the memory ECC information will be registered to D-Bus and generate 26memory ECC SEL as well. 27 28[[1]Intelligent Platform Management Interface Specification v2.0 rev 1.1, section 41](https://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html) 29 30#### Requirements 31 32Currently, the OpenBMC project does not support ECC event logs in D-Bus because 33there is no relevant ECC information in the OpenBMC D-Bus architecture. 34The new ECC D-Bus information will be added to the OpenBMC project and an ECC 35monitor service will be created to fetch the ECC count (ce_count/ue_count) from 36the EDAC driver. And make sure the EDAC driver must be loaded and ECC/other 37correctable or ECC/other uncorrectable counts need to be obtained from the EDAC 38driver. 39 40#### Proposed Design 41 42ECC-enabled memory controllers can detect and correct errors in operating 43systems (such as certain versions of Linux, macOS, and Windows) that allow 44detection and correction of memory errors, which helps identify problems before 45they become catastrophic faulty memory module. 46 47Many ECC memory systems use an "external" EDAC between the CPU and the memory 48to fix memory error. Most host integrate EDAC into the CPU's integrated memory 49controller. 50 51According to Section 42.2 of the IPMI specification, Table 42 [2], these SEL 52sensor types will be defined as `Memory` and `Event Data 3` field can be used to 53provide an event extension. Therefore, the BMC ECC event sets "Event Data 3" 54with the value FEh to identify the BMC ECC error. 55 56[[2] Intelligent Platform Management Interface Specification v2.0 rev 1.1, section 42.2](https://www.intel.com/content/www/us/en/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html) 57 58The main purpose of this function is to provide the BMC with the ability to 59record ECC error SELs. 60 61There are two new applications for this design: 62 63- poll the ECC error count 64- create the ECC SEL 65 66It also devised a mechanism to limit the "maximum number" of logs to avoid 67creating a large number of correctable ECC logs. When the `maximum quantity` is 68reached, the ECC service will stop to record the ECC log. The `maximum quantity` 69(default:100) is saved in the configuration file, and the user can modify the 70value if necessary. 71 72##### phosphor-ecc.service 73 74This will always run the application and look up the ECC error count every 75second after service is started. On first start, it resets all correctable ECC 76counts and uncorrectable ECC counts in the EDAC driver. 77 78It also provide the following path on D-Bus: 79 80- bus name : `xyz.openbmc_project.Memory.ECC` 81- object path : `/xyz/openbmc_project/metrics/memory/BmcECC` 82- interface : `xyz.openbmc_project.Memory.MemoryECC` 83 84The interface with the following properties: 85| Property | Type | Description | 86| -------- | ---- | ----------- | 87| isLoggingLimitReached | bool | ECC logging reach limits| 88| ceCount| int64 | correctable ECC events | 89| ueCount| int64 | uncorrectable ECC events | 90| state| string | bmc ECC event state | 91 92The error types for `xyz::openbmc_project::Memory::Ecc::Error::ceCount` and 93`ueCount` and `isLoggingLimitReached` will be created which generated the error 94type for the ECC logs. 95 96##### Create the ECC SEL 97 98Use the `phosphor-sel-logger` package to record the following logs in BMC SEL 99format. 100 101- correctable ECC log : when fetching the `ce_count` from EDAC driver parameter 102 and the count exceeds previous count. 103- uncorrectable ECC log : when fetching the `ue_count` from EDAC driver parameter 104 and the count exceeds previous count. 105- logging limit reached log : When the correctable ECC log reaches the 106 `maximum quantity`. 107 108#### Alternatives Considered 109 110Another consideration is that there is no stopping the recording of the ECC 111logging mechanism. 112When the checks `ce_count` and value exceeds the previous value, it will record 113the ECC log. But this will encounter a lot of ECC logs, and BMC memory will 114also be occupied. 115 116#### Impacts 117 118This application implementation only needs to make some changes when 119creating the event log, so it has minimal impact on the rest of the system. 120 121#### Testing 122 123Depending on the platform hardware design, this test requires an ECC 124driver to make fake ECC errors and then check the scenario is good. 125