Add failsafe logger for zones

Tested:
...
Nov 23 21:40:06 tmddp10-nfd01.prod.google.com swampd[4893]:
Zone `0` is in failsafe mode.
With update at `fleeting0`: The sensor has bad readings.
Nov 23 21

Add failsafe logger for zones

Tested:
...
Nov 23 21:40:06 tmddp10-nfd01.prod.google.com swampd[4893]:
Zone `0` is in failsafe mode.
With update at `fleeting0`: The sensor has bad readings.
Nov 23 21:40:06 tmddp10-nfd01.prod.google.com swampd[4893]:
Zone `1` is in failsafe mode.
With update at `fleeting1`: The sensor has bad readings.
Nov 23 21:40:06 tmddp10-nfd01.prod.google.com swampd[4893]:
Zone `1` leaves failsafe mode.
With update at `hotswap_in_Input_Power`: The sensor has recovered.
Nov 23 21:40:06 tmddp10-nfd01.prod.google.com swampd[4893]:
Zone `0` leaves failsafe mode.
With update at `hotswap_in_Input_Power`: The sensor has recovered.
...

Change-Id: I2c296addb7ad117c03c04a27de91204796cda036
Signed-off-by: James Zheng <alphetis@google.com>

show more ...

clang-format: re-format for clang-18

clang-format-18 isn't compatible with the clang-format-17 output, so we
need to reformat the code with the latest version. The way clang-18
handles lambda forma

clang-format: re-format for clang-18

clang-format-18 isn't compatible with the clang-format-17 output, so we
need to reformat the code with the latest version. The way clang-18
handles lambda formatting also changed, so we have made changes to the
organization default style format to better handle lambda formatting.

See I5e08687e696dd240402a2780158664b7113def0e for updated style.
See Iea0776aaa7edd483fa395e23de25ebf5a6288f71 for clang-18 enablement.

Change-Id: I0f105c3310e87172c65a09a8787a2db5a4041cc0
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>

show more ...

main: Gracefully handle SIGTERM

When systemd stops phosphor-pid-control.service it sends a SIGTERM.
Catch SIGTERM in the existing boost signal handler and stop the all
control loops to make sure the

main: Gracefully handle SIGTERM

When systemd stops phosphor-pid-control.service it sends a SIGTERM.
Catch SIGTERM in the existing boost signal handler and stop the all
control loops to make sure the destructor is called in each of them.

This functionality will be used in the following commit.

Tested: systemctl stop phosphor-pid-control.service and see dtor
being invoked before process terminates.

Change-Id: I5b1fe8f9191d703351b96e7ae19348f7ccab03d4
Signed-off-by: Patrick Rudolph <patrick.rudolph@9elements.com>

show more ...

clang-format: copy latest and re-format

clang-format-16 has some backwards incompatible changes that require
additional settings for best compatibility and re-running the formatter.
Copy the latest

clang-format: copy latest and re-format

clang-format-16 has some backwards incompatible changes that require
additional settings for best compatibility and re-running the formatter.
Copy the latest .clang-format from the docs repository and reformat the
repository.

Change-Id: Icbc50d46e84ee7ef756705e2b19741439a325074
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>

show more ...

fix default root directory logging path

Currently, log files will be written into root directory if we only
touch /etc/thermal.d/logging.

Before fix:
$ mkdir -p /etc/thermal.d && touch /etc/thermal

fix default root directory logging path

Currently, log files will be written into root directory if we only
touch /etc/thermal.d/logging.

Before fix:
$ mkdir -p /etc/thermal.d && touch /etc/thermal.d/logging && systemctl restart phosphor-pid-control.service
$ systemctl status -n 10000 phosphor-pid-control.service | grep enabled
swampd[10391]: Logging enabled:
$ ls /zone*
zone_0.log zone_1.log zone_2.log

After fix:
Logging enabled: /tmp
$ ls /tmp/zone*
/tmp/zone_0.log /tmp/zone_1.log /tmp/zone_2.log

Change-Id: I9f3e93b2d9c469b122f106fab54f3686524ec209
Signed-off-by: Jinliang Wang <jinliangw@google.com>

show more ...

Allow indefinite retries in tryRestartControlLoops()

On some platforms, when the BMC first starts up the sensors used in
swampd's PID loops may not yet exist, and may not start existing until
the ho

Allow indefinite retries in tryRestartControlLoops()

On some platforms, when the BMC first starts up the sensors used in
swampd's PID loops may not yet exist, and may not start existing until
the host is first powered on. Since that may not happen for an
arbitrarily long time, swampd can end up crashing after exhausting its
artificially-limited retry budget in tryRestartControlLoops(), at which
point it gets restarted by systemd and the loop continues. Each time it
crashes we generate a warning in the Redfish event log though, which
isn't ideal since nothing truly erroneous has actually happened (aside
from the daemon somewhat spuriously dying).

With this change we instead allow tryRestartControlLoops() to retry
indefinitely without crashing the process, so that it can simply wait
until the sensors it's configured to use eventually appear.

Signed-off-by: Zev Weiss <zev@bewilderbeest.net>
Change-Id: I13f591fded2563731cb5a8c3a67283b57f99f6bf

show more ...

main: add multiple search paths for config.json

For some platforms, /usr/share is read-only folder which is not allowed
to add new config.json.

Add multiple default paths for searching config.json

main: add multiple search paths for config.json

For some platforms, /usr/share is read-only folder which is not allowed
to add new config.json.

Add multiple default paths for searching config.json

1. pwd
2. /var/lib/swampd
3. /usr/share/swampd

Tested on Bletchley.

Signed-off-by: Potin Lai <potin.lai@quantatw.com>
Change-Id: I21f3963f5f33b65557b42c8e3700d73b139140a5

show more ...

Adding new feature of core PID loop logging

This differs from the normal logging,
as this focuses on the core of the PID loop computations.

All variables within the core pid/ec/pid.cpp pid()
functi

Adding new feature of core PID loop logging

This differs from the normal logging,
as this focuses on the core of the PID loop computations.

All variables within the core pid/ec/pid.cpp pid()
function are now logged, so math can be debugged.

Output is throttled to only log a new line
when it changes, or when 60 seconds have elapsed.

Creates 2 files for each PID loop,
one showing coefficients that were configured for it,
and one showing the variables changing over time.

Enable by --corelogging command line option,
or by creating /etc/thermal.d/corelogging file.

Tested:
Log files appear as expected, when enabled.
No changes noticed, when this feature is disabled.

Signed-off-by: Josh Lehan <krellan@google.com>
Change-Id: I3f37fe918e7cbc6fb885ffa2f268600d5a317d32

show more ...

Add debug mode

Description:
1. Could enable debug mode by adding file: /etc/thermal.d/debugging.
$ mkdir /etc/thermal.d
$ touch /etc/thermal.d/debugging
$ systemctl resta

Add debug mode

Description:
1. Could enable debug mode by adding file: /etc/thermal.d/debugging.
$ mkdir /etc/thermal.d
$ touch /etc/thermal.d/debugging
$ systemctl restart phosphor-pid-control

2. Could record fans output PWM, chosen temperature, PID/stepwise output PWM in debug mode.

Design:
1. Create debugging file and restart service to enable debug mode.

2. Check if debug mode is enabled to output fans output PWM, chosen temperature,
PID/stepwise output PWM, and so on.

Test Case:
1. Enable debug mode and check logs: pass

Change-Id: I8527ebcb81e723298ba7e786b4501f986ebd439e
Signed-off-by: Bonnie Lo <Bonnie_Lo@wiwynn.com>

show more ...

Change folder name build to buildjson

- If change to use meson build, meson will automake build folder,
and this will overwrite the origin folder and miss the buildjson
files.

Signed-off-by: Ha

Change folder name build to buildjson

- If change to use meson build, meson will automake build folder,
and this will overwrite the origin folder and miss the buildjson
files.

Signed-off-by: Harvey Wu <Harvey.Wu@quantatw.com>
Change-Id: Idfc4852bf56ed8f6c04e1b4be78ce40e08539132

show more ...

Add SIGHUP handling for reloading configuration

Add signal handling feature for handling the SIGHUP generated by
systemd ExecReload.

Tested log:
- Brfore reload
root@bletchley:~# systemctl status p

Add SIGHUP handling for reloading configuration

Add signal handling feature for handling the SIGHUP generated by
systemd ExecReload.

Tested log:
- Brfore reload
root@bletchley:~# systemctl status phosphor-pid-control.service
* phosphor-pid-control.service - Phosphor-Pid-Control Margin-based Fan Control Daemon
Loaded: loaded (/lib/systemd/system/phosphor-pid-control.service; enabled; preset: enabled)
Drop-In: /lib/systemd/system/phosphor-pid-control.service.d
`-10-bletchley.conf
Active: active (running) since Fri 2022-09-30 09:12:10 UTC; 1min 11s ago
Main PID: 884 (swampd)
CGroup: /system.slice/phosphor-pid-control.service
`-884 /usr/bin/swampd

Sep 30 09:12:32 bletchley swampd[884]: PID name: fan_tachs
Sep 30 09:12:32 bletchley swampd[884]: inputs: FAN0_TACH_IL, FAN0_TACH_OL, FAN1_TACH_IL, FAN1_TACH_OL, FAN2_TACH_IL, FAN2_TACH_OL, FAN3_TACH_IL, FAN3_TACH_OL,
Sep 30 09:12:32 bletchley swampd[884]: PID name: inlet_Temp
Sep 30 09:12:32 bletchley swampd[884]: inputs: Virtual_Inlet_Temp,
Sep 30 09:12:32 bletchley swampd[884]: PID name: retimer_temp
Sep 30 09:12:32 bletchley swampd[884]: inputs: MB_U19_THERM_LOCAL,
Sep 30 09:12:32 bletchley swampd[884]: PID name: switch_temp
Sep 30 09:12:32 bletchley swampd[884]: inputs: MB_U402_THERM_LOCAL,
Sep 30 09:12:32 bletchley swampd[884]: pushing zone 1
Sep 30 09:12:32 bletchley swampd[884]: PID Zone 1 max SetPoint 37 requested by inlet_Temp FAN0_TACH_IL FAN0_TACH_OL FAN1_TACH_IL FAN1_TACH_OL FAN2_TACH_IL FAN2_TACH_OL FAN3_TACH_IL FAN3_TACH_OL

- After reload called
root@bletchley:~# journalctl -u phosphor-pid-control.service | grep -i reload
Sep 30 09:13:46 bletchley systemd[1]: Reloading Phosphor-Pid-Control Margin-based Fan Control Daemon...
Sep 30 09:13:46 bletchley swampd[884]: reloading configuration
Sep 30 09:13:46 bletchley systemd[1]: Reloaded Phosphor-Pid-Control Margin-based Fan Control Daemon.
root@bletchley:~#
root@bletchley:~# systemctl status phosphor-pid-control
* phosphor-pid-control.service - Phosphor-Pid-Control Margin-based Fan Control Daemon
Loaded: loaded (/lib/systemd/system/phosphor-pid-control.service; enabled; preset: enabled)
Drop-In: /lib/systemd/system/phosphor-pid-control.service.d
`-10-bletchley.conf
Active: active (running) since Fri 2022-09-30 09:12:10 UTC; 2min 35s ago
Process: 1353 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 884 (swampd)
CGroup: /system.slice/phosphor-pid-control.service
`-884 /usr/bin/swampd

Sep 30 09:13:56 bletchley swampd[884]: PID name: fan_tachs
Sep 30 09:13:56 bletchley swampd[884]: inputs: FAN0_TACH_IL, FAN0_TACH_OL, FAN1_TACH_IL, FAN1_TACH_OL, FAN2_TACH_IL, FAN2_TACH_OL, FAN3_TACH_IL, FAN3_TACH_OL,
Sep 30 09:13:56 bletchley swampd[884]: PID name: inlet_Temp
Sep 30 09:13:56 bletchley swampd[884]: inputs: Virtual_Inlet_Temp,
Sep 30 09:13:56 bletchley swampd[884]: PID name: retimer_temp
Sep 30 09:13:56 bletchley swampd[884]: inputs: MB_U19_THERM_LOCAL,
Sep 30 09:13:56 bletchley swampd[884]: PID name: switch_temp
Sep 30 09:13:56 bletchley swampd[884]: inputs: MB_U402_THERM_LOCAL,
Sep 30 09:13:56 bletchley swampd[884]: pushing zone 1
Sep 30 09:13:56 bletchley swampd[884]: PID Zone 1 max SetPoint 37 requested by inlet_Temp FAN0_TACH_IL FAN0_TACH_OL FAN1_TACH_IL FAN1_TACH_OL FAN2_TACH_IL FAN2_TACH_OL FAN3_TACH_IL FAN3_TACH_OL

Signed-off-by: Potin Lai <potin.lai@quantatw.com>
Change-Id: Idea1ede908fe4a2da3018cc0624c78ac7e0530eb

show more ...

sdbusplus: use shorter type aliases

The sdbusplus headers provide shortened aliases for many types.
Switch to using them to provide better code clarity and shorter
lines. Possible replacements are

sdbusplus: use shorter type aliases

The sdbusplus headers provide shortened aliases for many types.
Switch to using them to provide better code clarity and shorter
lines. Possible replacements are for:
* bus_t
* exception_t
* manager_t
* match_t
* message_t
* object_t
* slot_t

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ie36d234f4580029a7832a0cd179f3bb78a1a403f

show more ...

main: Remove shared_ptr copy

GCC 11 complains about it and it isn't needed.

Change-Id: Ia0bb3d1aea29d0505a9868b1f922338a87710ddb
Signed-off-by: William A. Kennington III <wak@google.com>

Fix command-line parsing of loggingPath

The "-l" option for loggingPath was not enabling logging when given,
as it should have been. Now, logging can be enabled either by use of
command line or by e

Fix command-line parsing of loggingPath

The "-l" option for loggingPath was not enabling logging when given,
as it should have been. Now, logging can be enabled either by use of
command line or by existence of sentinel file, as intended.

Thanks to Anton Kachalov and Konstantin Klubnichkin for catching this
bug.

Also cleaned up the printing of the informational lines printed, for
loggingEnabled and tuningEnabled, to be correct as originally
intended: printed when enabled, no output when disabled.

Signed-off-by: Josh Lehan <krellan@google.com>
Change-Id: I3f20a4dee997e04985d32cd6020b719e0b4d117e

show more ...

Add object manager

Added object manager to FanCtrl node so that it can be queried mapper
and GetManagedObjects method can be called.

Tested: verified this using busctl command and see object manage

Add object manager

Added object manager to FanCtrl node so that it can be queried mapper
and GetManagedObjects method can be called.

Tested: verified this using busctl command and see object manager
interface has been added to this service.

Signed-off-by: Bruce Lee <Bruce_Lee@quantatw.com>
Change-Id: Ibbd6f63993aa26914fa42d0205263fd7a490f95e

show more ...

Eliminate ControlLoop init delay.

Introducing extra delay for the first loop is unnecessary and the
increasement of gap between dbus name request and object creation
sometimes crashes phosphor-objmg

Eliminate ControlLoop init delay.

Introducing extra delay for the first loop is unnecessary and the
increasement of gap between dbus name request and object creation
sometimes crashes phosphor-objmgr.

Notice this patch only recovers the gap to the previous status as
before 40786 instead of completely eliminating the gap.

Tested: internal project with IPMI sensors.
Signed-off-by: Hao Jiang <jianghao@google.com>
Change-Id: I0d689809347e9dffa9a1059577455f3224502600

show more ...

Bug fix: pidControlLoop may loop infinitely.

boost::basic_waitable_timer::cancel is a non-blocking function and only
sends cancellation to the pending operations. For other operations:
```
If the ti

Bug fix: pidControlLoop may loop infinitely.

boost::basic_waitable_timer::cancel is a non-blocking function and only
sends cancellation to the pending operations. For other operations:
```
If the timer has already expired when cancel() is called, then the
handlers for asynchronous wait operations will:
* have already been invoked; or
* have been queued for invocation in the near future.
These handlers can no longer be cancelled, and therefore are passed an
error code that indicates the successful completion of the wait
operation.
```

In our case, if pidControlLoop() has been invoked or in the invoke
queue while the timer cancellation, it will ignore the cancal ec and
infinitely call the pidControlLoop() chain.

Thus an extra cancel variable is introduced to break the chain.

Signed-off-by: Hao Jiang <jianghao@google.com>
Change-Id: Ie4e53454ee326bdf612abb511990610a6b528300

show more ...

Expose exception of restartControlLoops().

Expose the failure of restartControlLoops() for the last loop. The
orignal rethrowing an exception will lose the call stack infomation.
It leads to extreme

Expose exception of restartControlLoops().

Expose the failure of restartControlLoops() for the last loop. The
orignal rethrowing an exception will lose the call stack infomation.
It leads to extreme difficulties when debugging.

Comparing to embedding the traceback infomation into the new (nested)
exception, I would rather expose the exception directly to external
which preserves both call stack and varibale information.

Signed-off-by: Hao Jiang <jianghao@google.com>
Change-Id: I1535fc5768ea58e5af695948eec65fdfa84e8ff9

show more ...

Retry pidControlLoop in individual context handler

io.context is non-preemptive on single thread, which means if all
restartControlLoops are scheduled within one handler, the resources
which are hel

Retry pidControlLoop in individual context handler

io.context is non-preemptive on single thread, which means if all
restartControlLoops are scheduled within one handler, the resources
which are held by pidControlLoop will never be released even given that
their contexts have been cancelled, since their contexts cannot preempt
the context of restartControlLoops.

We need to schedule restartControlLoops in individual context handlers.
Thus when cancellation signal is broadcasted, context runtime will be
yield from restartControlLoops to pidControlLoop to release resources.

Tested:
With multiple reboot cycle (2 machines with 10 reboot cycle with
interval of 10 min), swampd is stable and functional.

Change-Id: I0c750dcfe73016b71c6cfd693c4f8ed99b12e5d0
Signed-off-by: Hao Jiang <jianghao@google.com>

show more ...

drop struct keyword for non-packed objects

As a style decision, struct is often used with packed structures to
indicate they are used like C-structs. Cleanup this codebase to not use
the extra stru

drop struct keyword for non-packed objects

As a style decision, struct is often used with packed structures to
indicate they are used like C-structs. Cleanup this codebase to not use
the extra struct keyword throughout.

Signed-off-by: Patrick Venture <venture@google.com>
Change-Id: I2f83bb1989e4d1f2f843ba3e45fb82e04f0fa61c

show more ...

dbus: parameterize dbusconfiguration init

Signed-off-by: Patrick Venture <venture@google.com>
Change-Id: I9cb6c0a19d4625d79da8613bd6eea13d6cc04220

main: Allowing logging and tuning to also be enabled by files

The logging and tuning flags, which come from the command line,
are now also able to be enabled by the presence of certain files.

This

main: Allowing logging and tuning to also be enabled by files

The logging and tuning flags, which come from the command line,
are now also able to be enabled by the presence of certain files.

This allows the logging and tuning features to be enabled,
at runtime, without having to change the command line.

The content of /etc/thermal.d/tuning does not matter,
only the existence of this file is checked for.
The content of /etc/thermal.d/logging can optionally
be a directory path, which will be used if it exists,
otherwise use the systemwide default temporary directory.

This makes it easy for people doing thermal calibration
to enable these options as they need, without having to
destabilize the system by hand-editing the systemd configuration,
which is non-trivial to change at runtime, due to systemd caching.

The directory choice of /etc/thermal.d has precedence,
as it was already hardcoded in the setpoint file during tuning.

Tested: Tuning mode was enabled and logging appeared in /tmp
mkdir -p /etc/thermal.d
touch /etc/thermal.d/logging
touch /etc/thermal.d/tuning
killall swampd

Signed-off-by: Josh Lehan <krellan@google.com>
Change-Id: Ie74673664de806cbb8a0c70c61310e646ec38014

show more ...

Eliminate swampd core dump after D-Bus updates sensors

The swamp daemon intializes a list of sensors and uses those to
periodically scan the state associated devices. Reading the sensors is
done wit

Eliminate swampd core dump after D-Bus updates sensors

The swamp daemon intializes a list of sensors and uses those to
periodically scan the state associated devices. Reading the sensors is
done with an async timer, that runs code to re-arm an async timer.

There is also a D-Bus update cycle that is independent of the async
timer reading the sensors. When the D-Bus updates the number of
sensors in the system a new list must be created. In order to create
the new list the timers using the old list must be stopped. Only after
those timers have stopped may a new list be generated, and a new set of
timers started.

The two processes are unware of each other. To safely perform the
change the pointers to the list of zones and timers must be kept alive
until all timer actions complete. Only after all references to the
pointers have been release may the new state be built, and new timers
started.

Prior to this change swampd would throw a SYSSEGV fault due to an
attempt to use a pointer that was no longer active.

Tested:
Issued a "reset -w" (Warm Reset command) from the EFI shell.
Waited for the system to reboot, and enter EFI
Checked for a core file in /var/lib/systemd/coredump
Repeated step 1 if coredump file was not present.
Completed 2900+ passes successfully when ealier code failed at less
than 800 passes.

Change-Id: Iff4607510db579c36dc34d6f76e6eb2f0250a03a
Signed-off-by: Johnathan Mantey <johnathanx.mantey@intel.com>

show more ...

transition dbus-configuration to run-time check

This now checks dbus if there is no json file configuration found.

Signed-off-by: Patrick Venture <venture@google.com>
Change-Id: Ic8f5b0531b7131a3d3

transition dbus-configuration to run-time check

This now checks dbus if there is no json file configuration found.

Signed-off-by: Patrick Venture <venture@google.com>
Change-Id: Ic8f5b0531b7131a3d333c2127043b7054924c156

show more ...

use ZoneInterface pointers where Dbus aspect not important

The implementation of the ZoneInterface used is the DbusPidZone, however
using the ZoneInterface when the Dbus aspect is unimportant provid

use ZoneInterface pointers where Dbus aspect not important

The implementation of the ZoneInterface used is the DbusPidZone, however
using the ZoneInterface when the Dbus aspect is unimportant provides for
trivial support of other implementations.

Signed-off-by: Patrick Venture <venture@google.com>
Change-Id: I0ed87322904e7f87e5b5c8a50c01144f3d843a10

show more ...