psu-ng: Disable Communication, Logic, and Memory (CML) fault

Turn off CML fault logging as some of the power supplies don't support
all PMBus commands that could be sent from device driver or applic

psu-ng: Disable Communication, Logic, and Memory (CML) fault

Turn off CML fault logging as some of the power supplies don't support
all PMBus commands that could be sent from device driver or application.
The PSU sets the CML fault bit on when it receives unsupported PMBus
command.

Testing:
- Tested the change on simulation by setting the CML status bit on and
verified no error log reported.
- Tested the change on BMC System as follow and verified no error log
reported
* Powered on the chassis
* i2cget -f -y 3 0x68 0x6c w # causes information error log for
tracing purpose.

Change-Id: Ie1b52e5350a29f65bd517ad11293c6dc522ac83e
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>
Signed-off-by: Faisal Awada <faisal@us.ibm.com>

show more ...

psu-ng: Remove previous input history code

As mentioned in a previous commit, the org.open_power D-Bus interfaces
that host the 30 second maximum and average input power values are no
longer necessa

psu-ng: Remove previous input history code

As mentioned in a previous commit, the org.open_power D-Bus interfaces
that host the 30 second maximum and average input power values are no
longer necessary, and have been replaced by a single peak input power
sensor object.

Remove all of the code that dealt with them.

For now, the code remains that toggles the GPIO to sync up the input
history values between the power supplies on boots and plugs. As the
input history is used to populate the new peak sensor, its values across
power supplies will then be using the same time intervals.

Change-Id: Ide5b40e67bf3938606a75bcc0291bc57232ce8bc
Signed-off-by: Matt Spinler <spinler@us.ibm.com>

show more ...

psu-ng: Add peak input power sensor for some PSs

Some models of the ibm-cffps power supplies support an 'input history'
command that reports 30 second maximum and average input power values.
The cod

psu-ng: Add peak input power sensor for some PSs

Some models of the ibm-cffps power supplies support an 'input history'
command that reports 30 second maximum and average input power values.
The code was currently putting those on an 'org.open_power' D-Bus
interface so that the history of those values could be captured in a
single D-Bus call.

Now that there is a real telemetry feature in Redfish that can capture
the history of a single sensor value, we can drop the custom D-Bus
interface and just use the normal Sensor.Value interface that contains
the most recent maximum value from the input history command, The sensor
name will be 'psX_input_power_peak' where X is the PS instance number.

The average input power telemetry will now just be obtained from the
psX_input_power sensor provided by phosphor-hwmon so an equivalent
sensor to the peak isn't needed here.

This commit will add support for putting the new sensor on D-Bus, and a
future one will remove the previous input history support.

Like sensors in other daemons, it will be set to not available when a PS
is removed, and not functional when it has an access problem. There
will be associations to the parent chassis and to the power supply so it
will show up in Redfish output with the other sensors.

This commit did remove one of the input history testcases, as trying to
get the right sequence of EXPECT_CALLs would get tricky when both the
old and new are running together.

Tested:
- New sensor shows up when PS is present and supports it.
- All interfaces on the object path are correct.
- Sensor value matches what org.open_power Max property had.
- Works correctly when:
- PS is missing on startup
- PS is removed
- Previously present PS is replaced.

Change-Id: Id9c33aa753c9af32880a0cc874b39c113222568f
Signed-off-by: Matt Spinler <spinler@us.ibm.com>

show more ...

Fix PSU status when BMC is in standby

Without this patch:
Plug in the power supplies in any order, the power supply that is
plugged in first will report a `PSU_KILL_Fault` in standby, but the
proble

Fix PSU status when BMC is in standby

Without this patch:
Plug in the power supplies in any order, the power supply that is
plugged in first will report a `PSU_KILL_Fault` in standby, but the
problem will be restored after the host is powered on.

With this patch:
Regardless of whether it is in standby or host powered on, this
problem disappears and it works fine.

Signed-off-by: George Liu <liuxiwei@inspur.com>
Change-Id: I18e0e26a4922dd710e042048625da1cc8b08dd3c

show more ...

psu-ng: PGOOD_DEGLITCH_LIMIT of five

Turns out that the (IBM) power supplies take up to around 3 seconds to
get the PGOOD# bit in STATUS_WORD set to the good (0) state after the
input power is withi

psu-ng: PGOOD_DEGLITCH_LIMIT of five

Turns out that the (IBM) power supplies take up to around 3 seconds to
get the PGOOD# bit in STATUS_WORD set to the good (0) state after the
input power is within range. Adjust the de-glitch value to avoid logging
PGOOD errors for the power supplies, as they mostly ride through the
momentary drops in the input power.

Use PGOOD_DEGLITCH_LIMIT instead of DEGLITCH_LIMIT for the pgoodFault.

Tested:
Rainier 2S4U
1. Drop outlet power for 0.5 seconds, verify no 110015F1.
2. Drop outlet power for 1 second, verify no 110015F1.
3. Drop outlet power for 1.5 seconds, verify no 110015F1.
Rainier 2S2U
1. Drop outlet power for 1 second, verify no 110015F1.
2. Drop outlet power for 5 seconds, verify 110015F0 (VIN_UV) IS logged.
3. Drop outlet power for 1.5 seconds, verify no 110015F1.
Everest (powersupply0 unplugged, powersupply3 missing)
1. Drop outlet power for 0.5 seconds, verify no 110015F1.
2. Drop outlet power for 1 second, verify no 110015F1.
3. Drop outlet power for 1.5 seconds, verify no 110015F1.
4. Drop outlet power for 3 seconds, verify 110015F0 IS logged.
5. Drop all outlet power for 5 seconds, verify 110000AC.

Change-Id: I6ba12209e6c84548fec036101ca02d857ed900f6
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Disable INPUT_HISTORY for 1400W IBM PSU

If the device driver is ibm-cpffps, read the MFR_POUT_MAX value
(max_power_out). Only enable INPUT_HISTORY data collection if it is not
the 1400W IBM

psu-ng: Disable INPUT_HISTORY for 1400W IBM PSU

If the device driver is ibm-cpffps, read the MFR_POUT_MAX value
(max_power_out). Only enable INPUT_HISTORY data collection if it is not
the 1400W IBM power supply (MSB/LSB results in 30725 for 1400). The
1400W IBM power supply appears to cause problems on the bus when
an INPUT_HISTORY PMBus command read occurs.

Tested:
Simulated Rainier 2S4U
Fake 2nd and 3rd PSUs to return 1400W value
Verify 1st and 4th collecting INPUT_HISTORY
Verify 2nd and 3rd PSUs not getting INPUT_HISTORY collected
-----
Verify real Rainier 2S4U with 1600W working as expected.

Change-Id: Ia37cea9b0273ac5926e4bc581a2ea8a4079afa23
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: INPUT_HISTORY syncHistory

Add in the function that syncs the power supply input history data
between all the installed power supplies.

Use the GPIO line name instead of gpiochip and number.

psu-ng: INPUT_HISTORY syncHistory

Add in the function that syncs the power supply input history data
between all the installed power supplies.

Use the GPIO line name instead of gpiochip and number. Use libgpiod via
helper utility. Create a toggleLowHigh() to use for synchronizing the
input history.

Add in indicator and helper functions to indicate a syncHistory is
needed if a power supply goes from missing to present.

Trace when syncHistory is called. This should be infrequent enough that
I do not think it would be a problem.

Initial testing on Rainier 2S2U did not need a lengthy delay between
lowering and raising the power-ffs-sync-history GPIO, but testing on
Rainier 2S4u required a longer delay.

Depends-On: Ib1ac2456f7f715360d089dfa4b6b379b516439ab

Change-Id: I022806155139d70fb4a42cc27eb9f279f6a3aedc
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Power supply class updates for input history

Update the meson files to include the record_manager with the
phosphor-psu-monitor application.

Since we do not want to blindly enable input his

psu-ng: Power supply class updates for input history

Update the meson files to include the record_manager with the
phosphor-psu-monitor application.

Since we do not want to blindly enable input history for all power
supplies, base the enablement of the feature off of the driver name.
Change the PowerSupply class to require the driver name be passed in,
and pass that down via the PSUManager during the configuration
determination.

Add a server manager to the PSUManager to handle the INPUT HISTORY data
that will be under /org/open_power/sensors.

The INPUT_HISTORY command is handled via a sysfs file in binary format,
so add in a readBinary() base function to allow for mock testing.

Change-Id: Iea163892d5482e6f2dacacfbfa746f605af52ed5
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add a clearVinUVFault function

Create a clearVinUVFault() function specifically for clearing the VIN_UV
and unit is off fault bits.

Use this function when see a vinUVFault with good voltage

psu-ng: Add a clearVinUVFault function

Create a clearVinUVFault() function specifically for clearing the VIN_UV
and unit is off fault bits.

Use this function when see a vinUVFault with good voltage read back from
READ_VIN. This should clear the latched fault if the fault is no longer
active. If the fault is still active, the next read will show the fault
again.

Also call clearVinUVFault() from the clearFaults() function.

Signed-off-by: Brandon Wyman <bjwyman@gmail.com>
Change-Id: Ic26029fe64e3cd3e232c615b223cafb774bf77f9

show more ...

psu-ng: Clear faults if VIN_UV and READ_VIN good

If we see that we have set the VIN_UV fault indicator, but the READ_VIN
value is returning a non-zero voltage for input, attempt to clear the
faults.

psu-ng: Clear faults if VIN_UV and READ_VIN good

If we see that we have set the VIN_UV fault indicator, but the READ_VIN
value is returning a non-zero voltage for input, attempt to clear the
faults. This could be a latched input voltage under-voltage condition
that has now cleared.

Save off actualVoltage in PowerSupply member variable. This allows for
comparing the previous actual voltage with the new reading, to trace out
the actual voltage instead of the calculated 0/110/220 voltage.

Signed-off-by: Brandon Wyman <bjwyman@gmail.com>
Change-Id: Ibf1642a081ccf4d4a9c4202ce2536799806af017

show more ...

psu-ng: Add Availability support to power supplies

Add support for determining if a power supply is considered 'available'
or not, and set the Available D-Bus property on the
xyz.openbmc_project.Sta

psu-ng: Add Availability support to power supplies

Add support for determining if a power supply is considered 'available'
or not, and set the Available D-Bus property on the
xyz.openbmc_project.State.Decorator.Availability interface with the
result.

A power supply is considered unavailable if any of the following are
true:
- it isn't present
- there is a input fault active
- there is a Vin UV fault active
- there is a PS KILL fault active
- there is a Iout OC fault active

The latter four faults are the faults where it can't provide good
output power, hence the PS is unavailable. These faults also don't
call out the power supply directly, so the Functional property won't
get set to false, unlike other faults.

The Available D-Bus property is then able to be used in Redfish
responses for the heath and state properties in the power supply
schema.

Signed-off-by: Matt Spinler <spinler@us.ibm.com>
Change-Id: I349eddb8c052cf238276c832af0dde9e49f6d3f0

show more ...

psu-ng: Continue reading after readFail

Update the PMBus::read() function to allow for not creating journal
trace and elog, but default to continuing to trace and elog, the
previous behavior.

If we

psu-ng: Continue reading after readFail

Update the PMBus::read() function to allow for not creating journal
trace and elog, but default to continuing to trace and elog, the
previous behavior.

If we reach the limit of read failures that results in a communication
error log, continue to read, but stop logging failures.

If communication restores, we may be able to detect what caused the read
failure, or otherwise detect or clear new faults.

Change-Id: If59b86211ab54c31248ede78f8f117b607298923
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: De-glitch all faults

Use DEGLITCH_LIMIT to determine all the faults. If a fault bit is on, do
not consider that a fault until it is seen at least DEGLITCH_LIMIT
times. With DEGLITCH_LIMIT se

psu-ng: De-glitch all faults

Use DEGLITCH_LIMIT to determine all the faults. If a fault bit is on, do
not consider that a fault until it is seen at least DEGLITCH_LIMIT
times. With DEGLITCH_LIMIT set to 3, the monitor would need to see a
fault bit on 3 times in a row before indicating that the power supply
has that fault.

This was done earlier for the PGOOD fault detection.

Change-Id: I918c2fcdd1d90ae253ab268bd04aa7a0da0208b8
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Clear faults when voltage back in range

If the last read voltage (via READ_VIN) was below the minimum and
now it is back in a valid range (100 or 200 volt range valid), clear all
the faults

psu-ng: Clear faults when voltage back in range

If the last read voltage (via READ_VIN) was below the minimum and
now it is back in a valid range (100 or 200 volt range valid), clear all
the faults to allow for re-detection of faults and logging of new errors.

Trace if INPUT_FAULT_WARN or VIN_UV clear. We should not expect to see
that without sending a CLEAR_FAULTS command (or a power cycle).

Tested:
Rainier 2S2U real hardware.
ePDU outlet off/on allows re-detection of injected CML fault.
- input fault, vin_uv fault, pgood/off fault.
- repeat shows faults cleared, and new faults logged.
Simulator pgood fault, then low voltage followed by good voltage.
Verify simulator can re-detect faults after voltage back in range.
Simulator fake input fault/warn on, then off and other fault on.
- verified tracing input going off without clear faults sent.
Simulator fake input fault/warn on, then no faults.
- verified tracing input going off without clear faults sent.

Change-Id: Ic8022cf137978ff660680e9680f778853cbecf0d
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: gtest cleanup expectation warnings

A number of the tests are missing various EXPECT_CALL statements that
result in very verbose testlog.txt output. This becomes especially
pr

psu-ng: gtest cleanup expectation warnings

A number of the tests are missing various EXPECT_CALL statements that
result in very verbose testlog.txt output. This becomes especially
problematic when a test fails, as there are pages of output to look for
to narrow down what failed where and why.

Adding in the EXPECT_CALL statements that should be there, such as
findHwmonDir, the reading of "in1_input" as part of the fault clearing,
etc.

Change-Id: I9f2f88622ad7b682461069df980a50b0b13c44a6
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add in handling of specific MFR faults

Add in a function to determine what the various bits in statusMFR may be
indicating for a fault, based on the type of power supply (device

psu-ng: Add in handling of specific MFR faults

Add in a function to determine what the various bits in statusMFR may be
indicating for a fault, based on the type of power supply (device driver
bound).

Add in PS_Kill, 12Vcs, and 12V CS faults for IBM power supply types.

Add in creating error logs for PS_Kill, 12Vcs, and 12V CS faults. The
12Vcs and 12V CS faults can essentially be treated the same as VOUT_UV
faults (same error type, same call out).

Tested:
Verified no PS_Kill, 12Vcs, or 12V CS fault on normal Rainier 2S4U

Simulated PS_Kill fault:
MFR fault: STATUS_WORD = 0x1840 STATUS_MFR_SPECIFIC = 0x10

Simulated 12Vcs fault:
PGOOD fault: STATUS_WORD = 0x1840, STATUS_MFR_SPECIFIC = 0x40
MFR fault: STATUS_WORD = 0x1840 STATUS_MFR_SPECIFIC = 0x40

Simulated 12V CS fault/warning:
MFR fault: STATUS_WORD = 0x1000 STATUS_MFR_SPECIFIC = 0x80

Change-Id: Ie89a58836ecec86dfa2e124eb6ab03e9dccce929
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add DEGLITCH_LIMIT, deglitch pgoodFault

While the power supply should not arbitrarily report a PGOOD fault, and
then turn it back off, there is a perception that this is indeed

psu-ng: Add DEGLITCH_LIMIT, deglitch pgoodFault

While the power supply should not arbitrarily report a PGOOD fault, and
then turn it back off, there is a perception that this is indeed
possible, a glitch of some sort.

To avoid possibly logging an error for an erroneous fault reporting,
make sure the fault is reported more than once before considering it to
be a true fault (deglitch the signal).

Tested:
Real Rainier 2S2U:
Verify tracing PGOOD faults seen and cleared, no error logged
Verify PGOOD/OFF error logged when manually set ON_OFF_CONFIG & OPERATION.
Verify deglitched PGOOD again on restart service (ON_OFF_CONFIG reset).

Change-Id: I54f775004d2e363cff21ff0512bd9283408f1f72
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add in detection of fan faults

If the FANS bit in the STATUS_WORD turns on (A fan or airflow fault or
warning has occurred), set a fan fault indicator in the power supply
obj

psu-ng: Add in detection of fan faults

If the FANS bit in the STATUS_WORD turns on (A fan or airflow fault or
warning has occurred), set a fan fault indicator in the power supply
object. During analysis of the power supplies, if a fan fault has
occurred, prioritize that over a temperature fault, include the
STATUS_TEMPERATURE and STATUS_FANS_1_2 command responses in the error
created. Call out the power supply with the fault.

Tested:
Verify no faults detected or logged on real hardware (Rainier 2S4U).
Simulate fan 1 fault on Rainier 2S2U, 110015FF PEL created.

Change-Id: Ifff5b4d96efe44b081a33caa01d70fdb578e57e3
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add code to detect VOUT_UV_FAULT

If the output voltage hits an undervoltage condition, the VOUT
fault/warn bit in STATUS_WORD will be on, but the VOUT_OV_FAULT bit will
be of

psu-ng: Add code to detect VOUT_UV_FAULT

If the output voltage hits an undervoltage condition, the VOUT
fault/warn bit in STATUS_WORD will be on, but the VOUT_OV_FAULT bit will
be off.

Check for VOUT_UV_FAULT after check for VOUT_OV_FAULT, create error,
call out the power supply indicating that fault.

Tested:
Verify no fault detected or error logged in normal conditions on
real hardware.
Simulate VOUT_UV fault:
VOUT_UV_FAULT fault: STATUS_WORD = 0x8000, STATUS_MFR_SPECIFIC = 0x0, STATUS_VOUT = 0x0
and
VOUT_UV_FAULT fault: STATUS_WORD = 0x8000, STATUS_MFR_SPECIFIC = 0x0, STATUS_VOUT = 0x30

Change-Id: I2605a197634616c4c2ad49f0275eaccaef7cc5f0
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add in detection of IOUT_OC fault

If bit 4 of STATUS_BYTE (lower byte of STATUS_WORD) turns on, there is
an output over-current fault. Add in code to detect that and log an
e

psu-ng: Add in detection of IOUT_OC fault

If bit 4 of STATUS_BYTE (lower byte of STATUS_WORD) turns on, there is
an output over-current fault. Add in code to detect that and log an
error, has priority after VOUT_OV fault.

Change-Id: Ia9cdb04a16a9f980aa699a8fff247d0c47e0aa93
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add in detection of PGOOD fault

If STATUS_WORD has bit 11 high (POWER_GOOD# is inactive) or bit 6 high
(unit is OFF), consider that a PGOOD fault. Lowest priority error to
ch

psu-ng: Add in detection of PGOOD fault

If STATUS_WORD has bit 11 high (POWER_GOOD# is inactive) or bit 6 high
(unit is OFF), consider that a PGOOD fault. Lowest priority error to
check, log an error if a power supply has a PGOOD fault, call out the
power supply.

Tested:
Simulated POWER_GOOD# / OFF and verified error logged.
Verified normal operations on real hardware do not log error.
Get power supply in bad state via i2cset commands:
1. Write 0x19 to 0x02 (ON_OFF_CONFIG, OPERATION command only).
2. Write 0x00 to 0x01 (Immediate off OPERATION command).
3. Restart service, error logged for STATUS_WORD = 0x0800.
4. Write 0x19 to 0x02, 0x00 to 0x01 while powered on.
5. Error logged for STATUS_WORD=0x0840.

Change-Id: I24dd1db780510d39dc245e76977099f790a2e59e
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add code to detect temperature fault

If the low byte of STATUS_WORD has bit 2 on/1, there is a temperature
fault or warning.

Tested:
Verify no error logged for t

psu-ng: Add code to detect temperature fault

If the low byte of STATUS_WORD has bit 2 on/1, there is a temperature
fault or warning.

Tested:
Verify no error logged for temperature fault during normal operation
on real hardware.
Verify temperature fault error logged when simulated
over-temperature fault.

Change-Id: Ib5b0fe849699e72e517ea4d59a69bbb6de1e5283
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Refactor PMBus expectations helper

The long list of parameters with a default of 0 is getting a bit
unwieldy to deal with. Refactor this code to use a struct that
initializes

psu-ng: Refactor PMBus expectations helper

The long list of parameters with a default of 0 is getting a bit
unwieldy to deal with. Refactor this code to use a struct that
initializes the values to 0.

Use scoping to allow for defaults of 0, when that is the desired value.

Change-Id: I9548823fb286fc886cc2c073206ce78f264092b6
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Add code to detect VOUT_OV_FAULT

If the output voltage hits an overvoltage condition, the VOUT_OV_FAULT
bit (bit 5 of STATUS_WORD low byte) should turn on. The PMBus spec
ind

psu-ng: Add code to detect VOUT_OV_FAULT

If the output voltage hits an overvoltage condition, the VOUT_OV_FAULT
bit (bit 5 of STATUS_WORD low byte) should turn on. The PMBus spec
indicates that VOUT fault/warning (bit 7 high byte) and VOUT_OV_FAULT
are associated with status in the STATUS_VOUT command response.

Check for VOUT_OV_FAULT after check for VIN_UV_FAULT, create error if
the fault is indicated.

Change-Id: Ia68b4f986393f0ba0184401deb29b4f5d25a2ed0
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...

psu-ng: Refactor EXPECT calls for PMBus STATUS_*

Continuing to copy and paste and update all these EXPECT_CALL() lines
for the various STATUS_* commands seems a bit tedious. Refactor tha

psu-ng: Refactor EXPECT calls for PMBus STATUS_*

Continuing to copy and paste and update all these EXPECT_CALL() lines
for the various STATUS_* commands seems a bit tedious. Refactor that
code to pull it out into a helper function.

Change-Id: If981de19cbc771ac8da17a2dfe0df50e40a1a190
Signed-off-by: Brandon Wyman <bjwyman@gmail.com>

show more ...