Add SSE filter param support

The Redfish spec require filtering of SSE entries to be supported.
This commit rearranges the code, and implements SSE sorting as well
as support for Last-Event-Id. To

Add SSE filter param support

The Redfish spec require filtering of SSE entries to be supported.
This commit rearranges the code, and implements SSE sorting as well
as support for Last-Event-Id. To do this it adds a dependency on
boost circular_buffer.

Tested:

SSE connections succeed. Show filtered results.

Change-Id: I7aeb266fc40471519674c7b65cd5cc4625019e68
Signed-off-by: Ed Tanous <etanous@nvidia.com>

show more ...

Fix Task Monitor URI

Fixes #272

The TaskMonitor urls we create aren't correct per Redfish. Per DSP0266
section 12.2, our TaskMonitor URIs should take the form

/redfish/v1/TaskService/TaskMonitors

Fix Task Monitor URI

Fixes #272

The TaskMonitor urls we create aren't correct per Redfish. Per DSP0266
section 12.2, our TaskMonitor URIs should take the form

/redfish/v1/TaskService/TaskMonitors/<id>

Note that even though this appears to be a collection, it is not, and
does not "exist" in the Redfish schema, hence why it is called out
explicitly.

Tested:
Started dump collection task with
POST
```
/redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData
```
GET /redfish/v1/Tasks/0

Returned TaskMonitor = /redfish/v1/Tasks/TaskMonitors/0

GET /redfish/v1/Tasks/TaskMonitors/0 returned 200

Change-Id: I9fb1d62090f7787d7649c077b748b51ac3202f8a
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Make journal log efficient

Journal logging currently loops over all entries to find even a single
entry. This was reasonable at the time when bmc couldn't really store a
lot, but now that BMCs are

Make journal log efficient

Journal logging currently loops over all entries to find even a single
entry. This was reasonable at the time when bmc couldn't really store a
lot, but now that BMCs are getting significantly more flash storage,
this simplification is insufficient. In an example system with an
AST2600, this API takes 32 seconds to respond. This is mediocre for
obvious reasons.

This commit updates to use the sd_journal APIs to let journald do the
skipping, which can use internal details and can be a lot more
efficient. To get the total size, bmcweb still needs to pull the
sequenceids of HEAD and TAIL to determine the complete size, but this is
still reasonable.

Tested:
Redfish service validator passes.

Various versions of top and skip return the correct result, pulling
various top sizes from 0, omitted to the limit.

https://gerrit.openbmc.org/c/openbmc/openbmc-tools/+/72975

To test all corner cases.

Change-Id: I0456bca4e037529f70eaee0bdd9191e9d5839226
Signed-off-by: Ed Tanous <etanous@nvidia.com>

show more ...

Break out Journal log into its own file

log_services.hpp probably shouldn't have been allowed to get as large
as it has. This commit starts by breaking out functions from
log_services.hpp, and move

Break out Journal log into its own file

log_services.hpp probably shouldn't have been allowed to get as large
as it has. This commit starts by breaking out functions from
log_services.hpp, and moves them to manager_logservices_journal.hpp.
Code is moved as-is with no functional changes.

Tested: Journal GET works as before. Redfish service validator passes.

Change-Id: I93c372ae3e39967e1b0eaf0cf496f84ac4114b5c
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Add tests for invalid datetimes in time_utils

Add unit tests for dateStringToEpoch method for various invalid datetime
strings and make sure that std::nullopt is returned as expected in each
test ca

Add tests for invalid datetimes in time_utils

Add unit tests for dateStringToEpoch method for various invalid datetime
strings and make sure that std::nullopt is returned as expected in each
test case.

Change-Id: I1eaa1beb471e02e7c11e575fa13bb175b48dc30d
Signed-off-by: Farah Rasheed <Farah.Rasheed1@dell.com>

show more ...

Make schemas selectable

Which schemas are installed should be selectable in both a meson config,
and trivially by forks. This commit gets us closer to that idea.

It does it in several ways, first,

Make schemas selectable

Which schemas are installed should be selectable in both a meson config,
and trivially by forks. This commit gets us closer to that idea.

It does it in several ways, first, the code for generating
JsonSchemaFile resources has been changed to be generated at runtime,
based on files on disk. This is slightly slower, but allows installing
schemas from anywhere, and matches the CSDL handling.

Next, the schema folders are separated into two sets
csdl -> This includes the complete schema pack from dmtf
installed -> this includes only the schemas the bmc includes

Similar folders exist for json-schema and json-schema-installed.

This allows any additional schemas to be a single symlink addition.
Note, this also checks in all of the dmtf json schemas, not just the
versions we use. This allows us to update the schema pack without
needing to break our versions we ship.

Because the static files are now selectable, all files need to be in a
folder. This forces the css and image for the redfish built-in gui to
be moved.

Tested:
/redfish/v1/JsonSchemas returns the correct result
/redfish/v1/JsonSchemas/UpdateService returns a JsonSchemaFile instance
/redfish/v1/JsonSchemas/UpdateService/UpdateService<version>json returns
the JsonSchemaFile contents.

Redfish service validator passes.

Change-Id: Ie96b2e4b623788dc2ec94eb40fcfd80325f0d826
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Allow fuzzy string comparisons in $filter expr

Filter allows comparing certain strings as numeric greater than or less
than operators. The most obvious example of this is something like

\$filter=C

Allow fuzzy string comparisons in $filter expr

Filter allows comparing certain strings as numeric greater than or less
than operators. The most obvious example of this is something like

\$filter=Created gt <timestamp>

Because internally timestamps are treated as strings, this requires
including and parsing out the timestamps again, which we have utilities
for.

In addition, "fuzzy" string comparisons, like

GPU_2 gt GPU_1

Should also be supported.

Tested: Unit tests pass

Change-Id: I39fc543921ed8cc93664d9cf297dad8ee902b68f
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Use lexme in redfish filter parser

Previously, the parser added space ignore instructions between every
node. This is because there was one place where we actually cared about
spaces, when doing op

Use lexme in redfish filter parser

Previously, the parser added space ignore instructions between every
node. This is because there was one place where we actually cared about
spaces, when doing operator comparisons (x eq y). If spaces are
ignored, it's impossible to determine the end of x and the beginning of
eq.

Spirit x3 has a lexeme, which allows us to ignore the parser skips
temporarily, which allows us to parse the operations in a much simpler
way. This also requires that we change to phrase_parse instead of
parse.

Tested: Unit tests pass. Good coverage.

Change-Id: Ifc6f1681e8524ba5032ee118cc3b3a18b30c639e
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Add filter parameter support

$filter is a parameter documented in the Redfish specification, section
7.3.4. It defines a mechanism for filtering arbitrary collections of
parameters based on a set o

Add filter parameter support

$filter is a parameter documented in the Redfish specification, section
7.3.4. It defines a mechanism for filtering arbitrary collections of
parameters based on a set of arbitrary language expressions.

From the specification, it supports the following language operators:

() Precedence grouping operator.
(Status/State eq 'Enabled' and Status/Health eq 'OK')
or SystemType eq 'Physical'

and Logical and operator.
ProcessorSummary/Count eq 2 and MemorySummary/TotalSystemMemoryGiB gt 64

eq Equal comparison operator.
ProcessorSummary/Count eq 2

ge Greater than or equal to comparison operator.
ProcessorSummary/Count ge 2

gt Great than comparison operator.
ProcessorSummary/Count gt 2

le Less than or equal to comparison operator
MemorySummary/TotalSystemMemoryGiB le 64

lt Less than comparison operator.
MemorySummary/TotalSystemMemoryGiB lt 64

ne Not equal comparison operator.
SystemType ne 'Physical'

not Logical negation operator.
not (ProcessorSummary/Count eq 2)

or Logical or operator.
ProcessorSummary/Count eq 2 or ProcessorSummary/Count eq 4

Support for these operators have been added in previous commits. This
commit enables them behind the insecure-enable-redfish-query meson
option. This is an arbitrary language, so the likelihood there's some
improper implementation in the patch is high. This gives folks the
ability to test it.

Tested:
Lots of unit tests included in this patch.

Functionally tested the basic operators:
```
GET /redfish/v1/Managers/bmc/LogServices/Journal/Entries?\$filter=EntryType+eq+'Oem'
GET /redfish/v1/Managers/bmc/LogServices/Journal/Entries?\$filter=EntryType+ne+'Oem'
```

Function as expected, producing multiple results or no results
respectively.

GET /redfish/v1 reports "FilterQuery": true

Redfish service validator passes.

Change-Id: Id568acc5dcfce868af12da5ee16c4f0caae8060a
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Filter Expression parser

This commit implements a parser for $filter expressions, per the redfish
specification and odata specification. This is intended to be used to
support $filter query for col

Filter Expression parser

This commit implements a parser for $filter expressions, per the redfish
specification and odata specification. This is intended to be used to
support $filter query for collections.

For parsing libraries, this commit chooses boost spirit x3. It's chosen
because it doesn't require a new external dependency, and is done
entirely in the compiler, using C++ syntax. While the syntax is still
somewhat difficult to read, there's a slew of unit tests included to
make sure that at least the common things we expect to work will parse
correctly.

Tested: Unit tests pass (good coverage). Code not yet used.

Signed-off-by: Ed Tanous <edtanous@google.com>
Change-Id: I1b0ab615bc49064acab4dad47f0a8aa499557bfc

show more ...

Make aggregator unit tests pass

When enabled, redfish aggregation causes unit tests to fail.

Tested: Unit tests now pass with redfish aggregation enabled.

Change-Id: Ieb92a4ad0a4be3ce757c816a676e0

Make aggregator unit tests pass

When enabled, redfish aggregation causes unit tests to fail.

Tested: Unit tests now pass with redfish aggregation enabled.

Change-Id: Ieb92a4ad0a4be3ce757c816a676e0a641cd7c85a
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Bump base registry to 1.18.1

72169 points out a mistake in 1.16 which is fixed in 1.18. Let's pull
1.18.1 in. 1.18.1 is the latest.

Changed 1 line in parse_registries.py and reran it.

Tested: None

Bump base registry to 1.18.1

72169 points out a mistake in 1.16 which is fixed in 1.18. Let's pull
1.18.1 in. 1.18.1 is the latest.

Changed 1 line in parse_registries.py and reran it.

Tested: None. In the past bumping these hasn't caused problems.

Change-Id: I79b135b8366d9b423f789c34802b1c0366d4e8a5
Signed-off-by: Gunnar Mills <gmills@us.ibm.com>
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Break out random ID methods

The method of creating a random ID from an openssl random generator of a
particular length is something that is generally useful, and something
we can write unit tests fo

Break out random ID methods

The method of creating a random ID from an openssl random generator of a
particular length is something that is generally useful, and something
we can write unit tests for. Add it.

Tested:
Redfish service validator login flows work correctly in redfish service
validator.

Change-Id: Ic3b58d33f1421f3eb39e2d57585958f87f6fb8ea
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Make cert generate for readonly directories

When run from a development PC, we shouldn't REQUIRE that the cert
directory exists or is writable.

This commit reworks the SSL cert generation to genera

Make cert generate for readonly directories

When run from a development PC, we shouldn't REQUIRE that the cert
directory exists or is writable.

This commit reworks the SSL cert generation to generate a string with
the certification info, instead of writing it to disk and reading it
back. This allows bmcweb to start up in read-only environments, or
environments where there isn't access to the key information.

Tested: Launching the application on a dev desktop without an ssl
directory present no longer crashes.

Change-Id: I0d44eb1ce8d298986c5560803ca2d72958d3707c
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Add ipv6 URL parsing test

It was reported that ipv6 URLS failed to parse now. This test seems
useful to have.

Tested: Unit tests pass.

Change-Id: I21f1b11039c1b35dff5c85ecab784dd3cd6228dc
Signed-

Add ipv6 URL parsing test

It was reported that ipv6 URLS failed to parse now. This test seems
useful to have.

Tested: Unit tests pass.

Change-Id: I21f1b11039c1b35dff5c85ecab784dd3cd6228dc
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Generate metadata at runtime

In the initial implementation of metadata indexing the bmc knew at
compile time what schemas it could potentially publish. bmcweb took the
approach of adding all schema

Generate metadata at runtime

In the initial implementation of metadata indexing the bmc knew at
compile time what schemas it could potentially publish. bmcweb took the
approach of adding all schemas of all versions to the $metadata
resource. Since that was made, two major changes have happened.
First, Redfish has added significantly more versions of each schema, as
well as significantly more schemas to the point where the metadata index
is now 213KB. While this file compresses fairly well, the size is
obvious from the large amount of time that redfish service validator
takes to parse the schemas, compared to actually acquiring BMC redfish
resources.
Second, aggregation was added, where an aggregated Redfish service might
implement any number of schemas, including OEM ones.

In an effort to fix this, this patch takes the compile-time algorithm in
update_schemas.py, and moves it into bmcweb itself, parsing the files on
disk as needed on demand. This has some immediate benefits; First, is
that now schemas can be potentially installed from anywhere, not only
from within the bmcweb build, and they will be resolved at runtime.

Second, patches that want to add support for a given schema need to only
symlink the schema into the correct folder, without needing to rerun
update_schemas.py. This saves time in review.

Finally, this opens to door to reducing the schema versions present in
the metadata to the unique set of only what this bmcweb instance, and
its aggregated BMCs expose.

Tested: Redfish service validator passes. Need A/B checking to verify
the file is byte for byte the same.
GET /redfish/v1/$metadata returns what looks like sane results, with a
correct content-type.
Unit tests require the use of TemporaryFileHandle, so that class is
moved into a more general folder, outside of test/http.

Change-Id: I326159099c6b6c4056023b2e173c5f074ed88ce1
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Add https parsing

This is yet another step in parsing HTTP requests.

Tested:
'''
curl -vvvv -k --user "root:0penBmc" -H "Content-Type: application/json" \
-X POST https://192.168.7.2/redfish/v1/Upd

Add https parsing

This is yet another step in parsing HTTP requests.

Tested:
'''
curl -vvvv -k --user "root:0penBmc" -H "Content-Type: application/json" \
-X POST https://192.168.7.2/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate \
-d '{ \
"TransferProtocol":"TFTP", \
"ImageURI":"https://192.168.7.1/myfile.bin" \
}'
'''

Returns ActionParameterNotSupported

TransferProtocol: Omitted
ImageURI: https://192.168.7.1/myfile.bin
Returns ActionParameterNotSupported

TransferProtocol: Omitted
ImageURI: 192.168.7.1/myfile.bin
Returns ActionParameterValueTypeError

TransferProtocol: Bad
ImageURI: https:/192.168.7.1/myfile.bin
Returns: ActionParameterNotSupported

No changes to GET requests, so Redfish Service Validator not necessary.

Change-Id: Ibf4b69877031f3b8617412c06d40f2d0d0827ac3
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Manage Request with shared_ptr

This is an attempt to solve a class of use-after-move bugs on the
Request objects which have popped up several times. This more clearly
identifies code which owns the

Manage Request with shared_ptr

This is an attempt to solve a class of use-after-move bugs on the
Request objects which have popped up several times. This more clearly
identifies code which owns the Request objects and has a need to keep it
alive. Currently it's just the `Connection` (or `HTTP2Connection`)
(which needs to access Request headers while sending the response), and
the `validatePrivilege()` function (which needs to temporarily own the
Request while doing an asynchronous D-Bus call). Route handlers are
provided a non-owning `Request&` for immediate use and required to not
hold the `Request&` for future use.

Tested: Redfish validator passes (with a few unrelated fails).
Redfish URLs are sent to a browser as HTML instead of raw JSON.

Change-Id: Id581fda90b6bceddd08a5dc7ff0a04b91e7394bf
Signed-off-by: Jonathan Doman <jonathan.doman@intel.com>
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Bring consistency to config options

The configuration options that exist in bmcweb are an amalgimation of
CROW options, CMAKE options using #define, pre-bmcweb ifdef mechanisms
and meson options usi

Bring consistency to config options

The configuration options that exist in bmcweb are an amalgimation of
CROW options, CMAKE options using #define, pre-bmcweb ifdef mechanisms
and meson options using a config file. This history has led to a lot of
different ways to configure code in the codebase itself, which has led
to problems, and issues in consistency.

ifdef options do no compile time checking of code not within the branch.
This is good when you have optional dependencies, but not great when
you're trying to ensure both options compile.

This commit moves all internal configuration options to:
1. A namespace called bmcweb
2. A naming scheme matching the meson option. hyphens are replaced with
underscores, and the option is uppercased. This consistent transform
allows matching up option keys with their code counterparts, without
naming changes.
3. All options are bool true = enabled, and any options with _ENABLED or
_DISABLED postfixes have those postfixes removed. (note, there are
still some options with disable in the name, those are left as-is)
4. All options are now constexpr booleans, without an explicit compare.

To accomplish this, unfortunately an option list in config/meson.build
is required, given that meson doesn't provide a way to dump all options,
as is a manual entry in bmcweb_config.h.in, in addition to the
meson_options. This obsoletes the map in the main meson.build, which
helps some of the complexity.

Now that we've done this, we have some rules that will be documented.
1. Runtime behavior changes should be added as a constexpr bool to
bmcweb_config.h
2. Options that require optionally pulling in a dependency shall use an
ifdef, defined in the primary meson.build. (note, there are no
options that currently meet this class, but it's included for
completeness.)

Note, that this consolidation means that at configure time, all options
are printed. This is a good thing and allows direct comparison of
configs in log files.

Tested: Code compiles
Server boots, and shows options configured in the default build. (HTTPS,
log level, etc)

Change-Id: I94e79a56bcdc01755036e4e7278c7e69e25809ce
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Break out formatters

In the change made to move to std::format, we defined some custom type
formatters in logging.hpp. This had the unintended effect of making
all compile units pull in the majorit

Break out formatters

In the change made to move to std::format, we defined some custom type
formatters in logging.hpp. This had the unintended effect of making
all compile units pull in the majority of boost::url, and nlohmann::json
as includes.

This commit breaks out boost and json formatters into their own separate
includes.

Tested: Code compiles. Logging changes only.

Change-Id: I6a788533169f10e19130a1910cd3be0cc729b020
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Move to process v2

Boost process v2 brings some significant benefits to our launching of
processes[1]. In bmcweb terms:
1. The code is radically simpler, which decreaeses compile times, and
redu

Move to process v2

Boost process v2 brings some significant benefits to our launching of
processes[1]. In bmcweb terms:
1. The code is radically simpler, which decreaeses compile times, and
reduces the scope for code scanning tools.
2. The code now uses standard asio pipes instead of inventing its own.
3. Separate compilation.

Tested:
We don't have a lot of unit tests for the virtual media stuff that I can
run, but we do have unit tests for credentials pipe, which in this
change have been ported over, so the feature works. Unit tests are
passing.

[1] https://www.boost.org/doc/libs/1_80_0/doc/html/boost_process/v2.html#boost_process.v2.introduction
Change-Id: Ia20226819d75ff6e492f8852185f0b73e8f5cf83
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Implement a Content-Security-Policy TODO

This TODO has been in bmcweb for a very long time. Implement it.

W3 sets rules for what security policies apply to which content
types[1]. Reading through

Implement a Content-Security-Policy TODO

This TODO has been in bmcweb for a very long time. Implement it.

W3 sets rules for what security policies apply to which content
types[1]. Reading through this, essentially CSP should only apply to
HTML files.

Tested: Unit tests pass. Webui loads properly. Chrome network window
Shows headers show up as expected.

[1] https://www.w3.org/TR/CSP2/#which-policy-applies

Change-Id: I5467d0373832668763c72a66da2a8872e07bfb58
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Add missing headers

Most of these were found by breaking every redfish class handler into
its own compile unit:

When that's done, these missing headers become compile errors. We
should just fix the

Add missing headers

Most of these were found by breaking every redfish class handler into
its own compile unit:

When that's done, these missing headers become compile errors. We
should just fix them.

In addition, this allows us to enable automatic header checking in
clang-tidy using misc-header-cleaner. Because the compiler can now
"see" all the defines, it no longer tries to remove headers that it
thinks are unused.

[1] https://github.com/openbmc/bmcweb/commit/4fdee9e39e9f03122ee16a6fb251a380681f56ac

Tested: Code compiles.

Change-Id: Ifa27ac4a512362b7ded7cc3068648dc4aea6ad7b
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Remove ibm locks feature

This feature was introduced to manage the operation sync at BMC while
multiple clients manage the BMC.

This feature scope has gone away and it is not a simple code to maint

Remove ibm locks feature

This feature was introduced to manage the operation sync at BMC while
multiple clients manage the BMC.

This feature scope has gone away and it is not a simple code to maintain
as per the growing standards of bmcweb.

This commit removes the feature from this repo.

Tested by: Locks routes are not available anymore

Change-Id: I257225cfb1f43d7d5dadb21a28a2ee5345c5112a
Signed-off-by: Sunitha Harish <sunithaharish04@gmail.com>
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...

Refactor tftp parser

This function in the next patch will be used for more than just TFTP, so
rename it to match intent, and refactor to use non-TFTP specific types.

Tested: Rename only. Need help

Refactor tftp parser

This function in the next patch will be used for more than just TFTP, so
rename it to match intent, and refactor to use non-TFTP specific types.

Tested: Rename only. Need help on TFTP setups if we need it.

Change-Id: Ifc7485aa60ec53407c38b3d1bec530bdacf50075
Signed-off-by: Ed Tanous <ed@tanous.net>

show more ...