scripts: auto-generate list of schemas

Since the list of schemas is growing and we already have a directory, it
does not need to be updated manually. Make a script for it and check
that that's been

scripts: auto-generate list of schemas

Since the list of schemas is growing and we already have a directory, it
does not need to be updated manually. Make a script for it and check
that that's been done in CI.

Tested: Using 'meson install', check that schemas are installed like
before.
Script works and updates the files as expected.

Change-Id: Ia1b3d99890477c4077e8e3a52a5338805d810440
Signed-off-by: Alexander Hansen <alexander.hansen@9elements.com>

show more ...

validate_configs: use referencing library

The script was raising the following deprecation warning:
```
DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0,
in favor of the https:

validate_configs: use referencing library

The script was raising the following deprecation warning:
```
DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0,
in favor of the https://github.com/python-jsonschema/referencing library,
which provides more compliant referencing behavior as well as more
flexible APIs for customization. A future release will remove
RefResolver. Please file a feature request (on referencing) if you are
missing an API for the kind of customization you need.
```

Update the script to use the referencing library. Unfortunately, this
moves the URI crawling into our code rather than as handled by the
RefResolver.

Change-Id: I742e4dff9470849947f24866bd0cfcbd0173170c
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>

show more ...

validate_configs: run config validation in parallel

Use a Python process pool for the schema validation to give parallelism
and add an option to determine the number of worker threads to use. By
def

validate_configs: run config validation in parallel

Use a Python process pool for the schema validation to give parallelism
and add an option to determine the number of worker threads to use. By
default use a worker for each CPU available on the system.

Tested:

```
# Before
./scripts/validate_configs.py --verbose 14.94s user 0.02s system 99% cpu 15.008 total
# After
./scripts/validate_configs.py --verbose 48.53s user 1.27s system 1802% cpu 2.762 total
```

Change-Id: I8df68c2319d416273bd741dbc6ca7acac131cc87
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>

show more ...

scripts: refactor validate_configs.py part 2

Extract function 'validator_from_file' to prepare a validator object
from a given schema file path.

Tested:

An invalid config still results in exit cod

scripts: refactor validate_configs.py part 2

Extract function 'validator_from_file' to prepare a validator object
from a given schema file path.

Tested:

An invalid config still results in exit code 1 and error output.
When all configs are valid, exit code is still 0.

Change-Id: Ie3e664c9f5098627c6e3c31e6699cd304ce4ce31
Signed-off-by: Alexander Hansen <alexander.hansen@9elements.com>

show more ...

scripts: refactor validate_configs.py

Config validation for a single config is extracted to a separate
function.

Tested:

An invalid config still results in exit code 1 and error output.
When all c

scripts: refactor validate_configs.py

Config validation for a single config is extracted to a separate
function.

Tested:

An invalid config still results in exit code 1 and error output.
When all configs are valid, exit code is still 0.

Change-Id: I98367dd5aebe0c4249f55d5056252761571528a6
Signed-off-by: Alexander Hansen <alexander.hansen@9elements.com>

show more ...

generate_config_list: set LANG variable

Commands such as 'sort' can yield different results depending on the
language chosen. We don't want to rely on the user's LANG variable
because we will end u

generate_config_list: set LANG variable

Commands such as 'sort' can yield different results depending on the
language chosen. We don't want to rely on the user's LANG variable
because we will end up with potentially different results than CI.

Force LANG to "C" everywhere when we run the script.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I01d488ed47255d2b523438148ca10c509b7d3354

show more ...

meson: generate configurations list

To avoid manually listing all the configuration files, use a script to
make a list of them.

There is a tradeoff when running a command to get the filenames, as i

meson: generate configurations list

To avoid manually listing all the configuration files, use a script to
make a list of them.

There is a tradeoff when running a command to get the filenames, as it
may not regenerate when file is added/removed [1].

So the list can still be kept as source, but generated in its own
separate file.

References:
[1] https://mesonbuild.com/FAQ.html#but-i-really-want-to-use-wildcards

Change-Id: I760486bafe94fce6485dfb7627946d0a77fdd322
Signed-off-by: Alexander Hansen <alexander.hansen@9elements.com>

show more ...

autojson: handle recursion

In order to allow the configurations directory to have vendor
subdirectories, modify the script slightly for directory traversal
to handle subdirectories.

Tested:

After

autojson: handle recursion

In order to allow the configurations directory to have vendor
subdirectories, modify the script slightly for directory traversal
to handle subdirectories.

Tested:

After performing a `git mv`, the subdirectory files show up in the
`format-code` output.
```
...
formatting file configurations/meta/terminus_2x100g_nic_tsff.json
...
```

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ib8cea8053d558c60e0a89dd6dcddbc4bc0d970fb

show more ...

jsonschema: update validator to the latest draft

The JSON Schema "draft 7" is now old and the latest is
"2020-12". Switch the validator object we use to be the
latest version.

Signed-off-by: Patri

jsonschema: update validator to the latest draft

The JSON Schema "draft 7" is now old and the latest is
"2020-12". Switch the validator object we use to be the
latest version.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I90a0d9b8669bade600ca3d48041f31727e410e14

show more ...

scripts/validate_configs.py: Ensure schemas are validated as well

The quick'n'dirty method of validating a schema is via
`jsonschema.validate()`. From `help(jsonschema.validate)` we have:

> :func

scripts/validate_configs.py: Ensure schemas are validated as well

The quick'n'dirty method of validating a schema is via
`jsonschema.validate()`. From `help(jsonschema.validate)` we have:

> :func:`validate` will first verify that the provided schema is
> itself valid, since not doing so can lead to less obvious error
> messages and fail in less obvious or consistent ways.
>
> If you know you have a valid schema already, especially if you
> intend to validate multiple instances with the same schema, you
> likely would prefer using the `Validator.validate` method directly
> on a specific validator (e.g. ``Draft7Validator.validate``).

The implication is that explicitly instantiating a validator does not
validate the provided schema.

The implementation of validate_configs.py directly instantiates the
Draft7Validator() in order to provide a RefResolver. However, it does
not do the required work of validating the provided schema.

Rework the code instantiating the validator to validate the schema prior
to use.

Tested:

```
$ git revert -n $(git log --grep 'I2b8102e7d6047c8a7e624e182465065268fb8ad9' --format='%H') && ./scripts/validate_configs.py -v -k; git reset --hard
Traceback (most recent call last):
File "/home/andrew/src/openbmc.org/openbmc/entity-manager/origin/./scripts/validate_configs.py", line 187, in <module>
main()
File "/home/andrew/src/openbmc.org/openbmc/entity-manager/origin/./scripts/validate_configs.py", line 138, in main
spec.check_schema(schema)
File "/usr/lib/python3/dist-packages/jsonschema/validators.py", line 204, in check_schema
raise exceptions.SchemaError.create_from(error)
jsonschema.exceptions.SchemaError: ['The schema for an entity manager configuration file. An entity ', 'mananger configuration file can consist of a single object, or an ', 'array of objects. Objects must be of type EMConfig.'] is not of type 'string'

Failed validating 'type' in metaschema['properties']['description']:
{'type': 'string'}

On schema['description']:
['The schema for an entity manager configuration file. An entity ',
'mananger configuration file can consist of a single object, or an ',
'array of objects. Objects must be of type EMConfig.']
HEAD is now at cecbd8129645 scripts/validate_configs.py: Ensure schemas are validated as well
```

Change-Id: Ib4d41c0f7a07c1ea4b9a483893969f21870a50e1
Signed-off-by: Andrew Jeffery <andrew@codeconstruct.com.au>

show more ...

Support C-style comments for configuration JSON parsing

1. Add and set ignore_comment to true to all nlohmann::json::parse().
2. Add remove_c_comments() in `validate_configs.py` to remove C-style

Support C-style comments for configuration JSON parsing

1. Add and set ignore_comment to true to all nlohmann::json::parse().
2. Add remove_c_comments() in `validate_configs.py` to remove C-style
comments before loading.
3. Attempt to reformat comments in the `autojson.py` taking liberal
short-cuts which are documented in the script.

Supported comment examples:

- Single-line style comments
```
{
// Single-line style comment (new line)
"Key": "Value" // Single-line comment (end of content)
}
```

- Multi-line style comments
```
{
/* Multi-line style comment */
/*
* Multi-line style comments
*/
}
```

Tested on harma system with manual applied patch below, which contains
a c-style comment in harma-pttv.json file.
Link: https://gerrit.openbmc.org/c/openbmc/entity-manager/+/67469/25

- scripts/autojson.py
Run autojson.py on harma-pttv.json, the output as same as original file.

- scripts/validate_configs.py
Run validate_configs.py passed.

- EntityManager service
EntityManager service loads and probes harma-pttv.json successfully.
```
root@harma:~# busctl introspect xyz.openbmc_project.EntityManager \
> /xyz/openbmc_project/inventory/system/board/Harma_PTTV \
> xyz.openbmc_project.Inventory.Item.Board
NAME TYPE SIGNATURE RESULT/VALUE FLAGS
.Name property s "Harma PTTV" emits-change
.Probe property s "xyz.openbmc_project.FruDevice({\'BOA... emits-change
.Type property s "Board" emits-change
```

Signed-off-by: Potin Lai <potin.lai@quantatw.com>
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ib235f2aa6a724615dc4c8184577f57abda8e17a6

show more ...

format: reformat with latest openbmc-build-scripts

Run all the latest formatters from openbmc-build-scripts. Disable
prettier on the configurations subdirectory because these are
script-formatted a

format: reformat with latest openbmc-build-scripts

Run all the latest formatters from openbmc-build-scripts. Disable
prettier on the configurations subdirectory because these are
script-formatted already with the repo-local `format-code` extension.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ifa4f2ec6688d60fde2a5f42efc96b6046e9be642

show more ...

python: fix flake8 warnings

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: If040c21bb65b613d75574d3154daa35c88c43f76

Remove mux-closing detritus

It's been a while since we had python in the build; this script is
obsolete, as is the reference to it in the FruDevice service file.

Signed-off-by: Zev Weiss <zev@bewil

Remove mux-closing detritus

It's been a while since we had python in the build; this script is
obsolete, as is the reference to it in the FruDevice service file.

Signed-off-by: Zev Weiss <zev@bewilderbeest.net>
Change-Id: If0bb9a4ff7c30d2e5b9ca370e11918ab36c820f8

show more ...

scripts: use lower_snake_case for filenames

To be consistent with lower_snake_case source code filenames, use
lower_snake_case for scripts file names as well.

run-ci.sh is not renamed because openb

scripts: use lower_snake_case for filenames

To be consistent with lower_snake_case source code filenames, use
lower_snake_case for scripts file names as well.

run-ci.sh is not renamed because openbmc-build-scripts specifically
looks for run-ci.sh or run-ci.

Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
Change-Id: I4535bac41375d2c78e71da5020cb5e2410ff3eaa

show more ...

autojson.py: add newline character before the end of file

POSIX standard defines line as something terminating with the newline
character. So the last line of a text file should also have that.

Pop

autojson.py: add newline character before the end of file

POSIX standard defines line as something terminating with the newline
character. So the last line of a text file should also have that.

Popular editors add it automatically so to avoid unnecessary noise in
the diffs between manually edited files and something that went through
autojson.py it should add it as well.

Signed-off-by: Paul Fertser <fercerpav@gmail.com>
Change-Id: I1e04b85eb41b8a7d00d506f49f994c6b40eb5fdf

show more ...

Revert "autojson.py: Check the sensor name"

This reverts commit 086f19d287fdc7d7be9c52ad9aee7704e4f339f5.

The recent script change causes errors.
formatting file configurations/1Ux16 Riser.json
Tra

Revert "autojson.py: Check the sensor name"

This reverts commit 086f19d287fdc7d7be9c52ad9aee7704e4f339f5.

The recent script change causes errors.
formatting file configurations/1Ux16 Riser.json
Traceback (most recent call last):
File "/build/openbmcup/build/workspace/sources/entity-manager/scripts/autojson.py", line 30, in <module>
nl = [s['Name'] for s in j['Exposes']]
TypeError: list indices must be integers or slices, not str

Considering the files this fail on have been around for a while, it's
clear they were never tested.

Change-Id: Ibda22e9ce43d1ba8e7a3c02609ef4d135ebb3044

show more ...

autojson.py: Check the sensor name

We may be careless and write down duplicate sensor names
in the json, which will cause the entity-manager to
generate duplicate dbus paths and coredump.

This chan

autojson.py: Check the sensor name

We may be careless and write down duplicate sensor names
in the json, which will cause the entity-manager to
generate duplicate dbus paths and coredump.

This change counts the sensor name and outputs a red alert
message if there are duplicates

Signed-off-by: John Wang <wangzhiqiang.bj@bytedance.com>
Change-Id: I3363e744d7e0ddf8cb8894e120368a45af4f8de9

show more ...

run-ci: modify permissions to execute

Per discussion in Ib420c69e88e88ca0a50863c83ad43fac750d3e7a, we
are going to require these scripts to have execute permission.

Signed-off-by: Patrick Williams

run-ci: modify permissions to execute

Per discussion in Ib420c69e88e88ca0a50863c83ad43fac750d3e7a, we
are going to require these scripts to have execute permission.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I04fefb6a10b195ae2782014bf305d2cf80f706b3

show more ...

test: run validation script from CI

Run the schema validation script when in the OpenBMC continuous
integration environment. This ensures that both the schema and EM JSON
documents are always valid

test: run validation script from CI

Run the schema validation script when in the OpenBMC continuous
integration environment. This ensures that both the schema and EM JSON
documents are always valid.

A handful of schema are known to fail - add an xfail file until those
can be fixed up.

Change-Id: Ifabfcce0f7953cd0e3bd5489bb0bbc2ce3e6989d
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

scripts: add configuration validation script

Add a script to validate configurations against the schema. Use
https://github.com/Julian/jsonschema to perform the json validation.

The script is inte

scripts: add configuration validation script

Add a script to validate configurations against the schema. Use
https://github.com/Julian/jsonschema to perform the json validation.

The script is intended to be run from a continuous integration
environment or by configuration/schema developers. A key
assumption/feature of the script is that its users will always prefer to
resolve relative references to the local filesystem. As such, the
script computes a base URI that points to the filesystem and instructs
the validator to use that in place of whatever base_uri it derives from
the global $id attribute. For additional reading see:

https://json-schema.org/understanding-json-schema/structuring.html#the-id-property
https://github.com/Julian/jsonschema/issues/98

Without any options the script assumes it is being run from an
entity-manager source distribution and attempts to find the schema and
configuration files relative to the location of the script.

Alternatively, the script can validate arbitrary json files against
arbitrary schema:

./validate-configs.py -s foo.schema.json -c test1.json -c test2.json

By default the validation stops as soon as a configuration does not
validate. Use -k to override this behavior and validate as many
configurations as possible.

Provide an option to instruct the script to ignore a list of
configurations that are expected to fail validation to be used in
continuous integration setups - similar in concept to xfail mechanisms
provided by most build systems with unit test support.

Change-Id: I7d67a54993a6d5e00daf552d9d350c80411b997b
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

autojson: don't check non-json files

Skip files that don't have a json suffix so other files (like makefiles)
can be placed in the configurations directory.

Change-Id: Ibdb4d29a6a024762d3f90ca410d1

autojson: don't check non-json files

Skip files that don't have a json suffix so other files (like makefiles)
can be placed in the configurations directory.

Change-Id: Ibdb4d29a6a024762d3f90ca410d11815fe641070
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Add build step to verify json formatting

This will run autojson during the build to verify
that all files are formatted correctly.

Tested: Before this change build failed, after
fixing json file, i

Add build step to verify json formatting

This will run autojson during the build to verify
that all files are formatted correctly.

Tested: Before this change build failed, after
fixing json file, it passed

Change-Id: I119cc898536a972bb7a248143b40d695f880ff2f
Signed-off-by: James Feist <james.feist@linux.intel.com>

show more ...

Sort Exposes Alphabetically By Type

This makes it much easier to find components of
a specific type because they will be grouped together.

Tested: Sensor list was still the same

Change-Id: Iab15be

Sort Exposes Alphabetically By Type

This makes it much easier to find components of
a specific type because they will be grouped together.

Tested: Sensor list was still the same

Change-Id: Iab15be34e003be1aac120864c18a80eaaa9b391c
Signed-off-by: James Feist <james.feist@linux.intel.com>

show more ...

Move closeMuxes script to a regex

This is to allow muxes of any type to be closed correctly on restart.

Change-Id: Ib0e15d3ca6309e7a297c910bded2ce172d16f956
Signed-off-by: Ed Tanous <ed.tanous@inte

Move closeMuxes script to a regex

This is to allow muxes of any type to be closed correctly on restart.

Change-Id: Ib0e15d3ca6309e7a297c910bded2ce172d16f956
Signed-off-by: Ed Tanous <ed.tanous@intel.com>

show more ...