build-unit-test-docker: Fix https_proxy

The https_proxy was not correctly set and if build with proxy settings,
it gives below error:

base: WARNING: Retrying (Retry(total=4, connect=None, read=

build-unit-test-docker: Fix https_proxy

The https_proxy was not correctly set and if build with proxy settings,
it gives below error:

base: WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.',
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
object at 0x7efe419e99d0>: Failed to establish a new connection:
[Errno -2] Name or service not known'))': /simple/inflection/
RAN: /usr/bin/docker build --build-arg http_proxy=http://<myproxy>
--build-arg 'https_proxy={https_proxy}' --network=host --force-rm
--no-cache=false -t
openbmc/ubuntu-unit-test-base:2021-W07-773f263e895d8ed6 -

Tested: Fix the https_proxy string and verify the CI passes.

Signed-off-by: Lei YU <yulei.sh@bytedance.com>
Change-Id: I87f543d33747b4957af83990f80bd8e832f4e5e3

show more ...

build-qemu-robot-docker: upgrade to focal

The 'pip' version that comes with bionic is no longer able to
install some of the python modules we need. Specificially, we
end up with a failure as:

build-qemu-robot-docker: upgrade to focal

The 'pip' version that comes with bionic is no longer able to
install some of the python modules we need. Specificially, we
end up with a failure as:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-lz9dedw0/cryptography/setup.py", line 14, in <module>
from setuptools_rust import RustExtension
ModuleNotFoundError: No module named 'setuptools_rust'

Rather than pin the cryptography module, which is actually a
dependency from a few levels down, upgrade the distro to focal to
pick up a new version of pip.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ief9aaccbe14b835a2dcae5c7d53cc5969f044cf3

show more ...

build-unit-test-docker: switch json to use cmake

nlohmann/json supports CMake and we use it in the bitbake
recipe, so use it here as well. The bitbake recipe has a
'install_append' to symlink the f

build-unit-test-docker: switch json to use cmake

nlohmann/json supports CMake and we use it in the bitbake
recipe, so use it here as well. The bitbake recipe has a
'install_append' to symlink the final header directly into
'{includedir}/json.hpp' so we need to continue to do the
same here.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ibf61a7a471216c43427109907a54e81dde1ae8fc

show more ...

build-unit-test-docker: pythonic refactoring

Do a bunch of clean-up to the script to better / modern Python
practices.

* Document the PackageDef dictionary.

* Add type hinting so that 'myp

build-unit-test-docker: pythonic refactoring

Do a bunch of clean-up to the script to better / modern Python
practices.

* Document the PackageDef dictionary.

* Add type hinting so that 'mypy' static analysis passes (and
code self-documents better).

* Create a Package class to encapsulate everything related to
package stage builds and a Docker namespace to encapsulate
Docker operations.

* Overall better documentation.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I841d844aa9401889d428ed55c10bee6c1b0a7109

show more ...

build-unit-test-docker: update numerous packages

- Update packages per new bitbake recipes.
- Favor using the tag-name rather than a git hash, when available.
- Linux headers now requires rsync.

Si

build-unit-test-docker: update numerous packages

- Update packages per new bitbake recipes.
- Favor using the tag-name rather than a git hash, when available.
- Linux headers now requires rsync.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I90d045fa7ef69067ddf09685dfde3265cc1e948f

show more ...

clean-unit-test-docker: script to clean old images

Since Docker images created by build-unit-test-docker now have
a date-stamp and content hash as part of their tag, a system's
Docker image reposito

clean-unit-test-docker: script to clean old images

Since Docker images created by build-unit-test-docker now have
a date-stamp and content hash as part of their tag, a system's
Docker image repository could grow quite large if not cleaned up.
Add a script that easily identifies old Docker images created by
that script and removes them (anything older than the current ISO
week).

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I316e71ba0b336035e33ebee223f7faf702eaeb2c

show more ...

build-unit-test-docker: create hashed tags for final image

Create a tag for the final image in the same format as the stage
images. Display this tag to stdout at the end of the docker build
so that

build-unit-test-docker: create hashed tags for final image

Create a tag for the final image in the same format as the stage
images. Display this tag to stdout at the end of the docker build
so that other scripts can consume it via calls like:
$(scripts/build-unit-test-docker)

Modify `run-unit-test-docker.sh` to get the docker image from
`build-unit-test-docker` rather than a static image name. This
ensures that the docker image used in unit testing is the exact
image requested to be built. (Soon this will allow us to incorporate
Gerrit changes into the docker image itself.)

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Idd7a9a7b0157f1c520b33b3ec7ac0d5b52018d9e

show more ...

build-unit-test-docker: invalidate base image weekly

Docker will cache an image built with the exact same commands,
so we only get latest Ubuntu packages rarely. Add a date-based
echo command into

build-unit-test-docker: invalidate base image weekly

Docker will cache an image built with the exact same commands,
so we only get latest Ubuntu packages rarely. Add a date-based
echo command into the early stages of the base image build so
that we regularly get updates to all the packages in the base
image.

This invalidation, via echo, is only performed when running under
a Jenkins-like context. The purpose of this distinction is so that
we are more likely to cache failures induced by an underlying OS
update in CI, but when developers run this script locally they rarely
see an invalidation that is not due to a code package update.

Add an environment variable option "FORCE_DOCKER_BUILD" which can be
set by developers to bypass the Docker cache and force all images
to be rebuilt, just in case they really want to do that sort of thing.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Iec92ebada513975d0806099083384dfca623f144

show more ...

build-unit-test-docker: build docker stages in parallel

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I1899b2cacdf8f668c3f1226959d1b5a044b5fc6c

build-unit-test-docker: split docker stages into invocations

Rather than a single large Dockerfile, split each stage into its
own Dockerfile with its own invocation. This is in preparation
for buil

build-unit-test-docker: split docker stages into invocations

Rather than a single large Dockerfile, split each stage into its
own Dockerfile with its own invocation. This is in preparation
for building the docker stages in parallel.

Since multiple invocations of this script could be running in
parallel, possibly with different content, this requires creating
well-named tags for each stage. Create these tags as a combination
of the current date and the content that went into the stage
Dockerfile. The date is used so that we can refresh the base
image more regularly.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I77ac232f1603b6ce74c55f0f9466482ff30eb32e

show more ...

build-unit-test-docker: switch cmake to ninja backend

Ninja is faster and what is used by Yocto.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Icbb17ab466e03844b97c27869646cc580f77

build-unit-test-docker: switch cmake to ninja backend

Ninja is faster and what is used by Yocto.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Icbb17ab466e03844b97c27869646cc580f77c558

show more ...

build-unit-test-docker: generate package directives

Generate the docker build directives for all packages by
performing a depth-first dependency search.

Signed-off-by: Patrick Williams <patrick@stw

build-unit-test-docker: generate package directives

Generate the docker build directives for all packages by
performing a depth-first dependency search.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ia3b6778e0f15bd9d5c4cff5e5e7bf2a620c0e156

show more ...

build-unit-test-docker: templatize build directives

Create 'pkg_build' function that generates the docker RUN template
necessary to build all the packages.

Signed-off-by: Patrick Williams <patrick@

build-unit-test-docker: templatize build directives

Create 'pkg_build' function that generates the docker RUN template
necessary to build all the packages.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ie7e7a3b3e78c67e6d97a83f3b43c3c54d10b77e6

show more ...

build-unit-test-docker: add dependencies to package def

Add dependencies to the package definition for each package and remove
the manual "COPY" commands to create the staged-dependencies.
Eventuall

build-unit-test-docker: add dependencies to package def

Add dependencies to the package definition for each package and remove
the manual "COPY" commands to create the staged-dependencies.
Eventually this will also be used to parallelize the docker build
process.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ifb7e26fccea62ae9bc8d450a91d1fc4d9ef0db06

show more ...

build-unit-test-docker: simplify fetching

Most packages we use come from github, except for boost. Create a
'pkg_url' function to get the URL for a revision for curl calls.
This function automatica

build-unit-test-docker: simplify fetching

Most packages we use come from github, except for boost. Create a
'pkg_url' function to get the URL for a revision for curl calls.
This function automatically creates the URL from the package name,
unless the package has a 'url' property. If the 'url' property exists,
treat it as a function that generates the URL from the package name and
revision (to handle the boost case and any future non-github packages).

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ifc47a1b580e6ec3728d8deeaa4c568017a80e03f

show more ...

build-unit-test-docker: use docker stagename everywhere

In some cases the script used a 'stagename' as the temporary docker
image used to create dependencies. Use this throughout for
consistency.

build-unit-test-docker: use docker stagename everywhere

In some cases the script used a 'stagename' as the temporary docker
image used to create dependencies. Use this throughout for
consistency.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I8b2d8624747b8a36630dd27f6aee32ddf10defa7

show more ...

build-unit-test-docker: automatically lookup HEAD rev

Switch from a special 'head_pkgs' list to automatically looking
up the HEAD revision of any package that doesn't have a specified
revision. Als

build-unit-test-docker: automatically lookup HEAD rev

Switch from a special 'head_pkgs' list to automatically looking
up the HEAD revision of any package that doesn't have a specified
revision. Also, switch the 'pkg_rev' to a dictionary so we can add
additional fields later on.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I8ead996469966f087845c6b3cbd6eb5226f3e235

show more ...

build-unit-test-docker: convert to Python

Do a near identical conversion from bash to Python for the script which
builds the unit-test docker containers. I plan to do additional
enhancements to bui

build-unit-test-docker: convert to Python

Do a near identical conversion from bash to Python for the script which
builds the unit-test docker containers. I plan to do additional
enhancements to build the sub-containers in parallel, so I needed to
first do this conversion to Python in order to make that implementation
sane.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I6df6f294b41f3a01f95fbe23a6dcd1ea21a315a9

show more ...

Updated firefox and geckodriver to the latest

Change-Id: I77ffde190a58d17a372de586eff94e8310cd3803
Signed-off-by: Rahul Maheshwari <rahulmaheshwari@in.ibm.com>

scripts: Add hacky qemu verification script

Document how to test qemu before we update openbmc/qemu master.

Change-Id: I7ffc83a24ca58056d2678330b6e0e422849b231b
Signed-off-by: Andrew Jeffery <andre

scripts: Add hacky qemu verification script

Document how to test qemu before we update openbmc/qemu master.

Change-Id: I7ffc83a24ca58056d2678330b6e0e422849b231b
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

format-code: ensure default used if no other found

The default format-code.sh provided by this repo is installed in the
WORKSPACE dir (which is not searched). Use the default script if no
other one

format-code: ensure default used if no other found

The default format-code.sh provided by this repo is installed in the
WORKSPACE dir (which is not searched). Use the default script if no
other one is provided by the repo under test.

Tested:
- Verified default was run when no other was present
- Verified the one in the repo under test was used when present

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I064d72197aaccee50943221d595a8b5ab56dc917

show more ...

unit-test: allow non-shell scripts for format extension

Switch the 'format-code.sh' extension to look recursively for any
'format-code.sh' or 'format-code' script. This allows non-shell
scripts to

unit-test: allow non-shell scripts for format extension

Switch the 'format-code.sh' extension to look recursively for any
'format-code.sh' or 'format-code' script. This allows non-shell
scripts to be used and allows the unit-test extensions to be moved
to a subdirectory (such as .openbmc).

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I2dfed1230bd70b871eca7c5b3d0078786f4d9460

show more ...

unit-test: fix and use CODE_SCAN_DIR

There are two different patterns for doing the equivalent of
os.path.join(WORKSPACE, UNIT_TEST_PKG)

Convert them all to use the CODE_SCAN_DIR variable and s

unit-test: fix and use CODE_SCAN_DIR

There are two different patterns for doing the equivalent of
os.path.join(WORKSPACE, UNIT_TEST_PKG)

Convert them all to use the CODE_SCAN_DIR variable and set CODE_SCAN_DIR
using `os.path.join`.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I1f2386f105dae100176247c40092a93c35974422

show more ...

unit-test: allow non-shell scripts for extensions

The unit-test script currently allows extension scripts of the
name 'run-ci.sh', but there are examples within openbmc where
the extensions are not

unit-test: allow non-shell scripts for extensions

The unit-test script currently allows extension scripts of the
name 'run-ci.sh', but there are examples within openbmc where
the extensions are not actually shell (at least one is Python).

Remove the extension requirement on the script by looking both
for 'run-ci.sh' and 'run-ci'.

Tested the 'find_file' script with:

> find_file('run-ci', '.../openbmc-sources')
['.../openbmc-sources/phosphor-dbus-interfaces/run-ci']

> find_file('run-ci.sh', '.../openbmc-sources')
['.../openbmc-sources/phosphor-logging/extensions/openpower-pels/registry/run-ci.sh',
'.../openbmc-sources/phosphor-power/phosphor-regulators/test/run-ci.sh',
'.../openbmc-sources/entity-manager/scripts/run-ci.sh' ]

> find_file(['run-ci', 'run-ci.sh'], '.../openbmc-sources')
['.../openbmc-sources/phosphor-dbus-interfaces/run-ci',
'.../openbmc-sources/phosphor-logging/extensions/openpower-pels/registry/run-ci.sh',
'.../openbmc-sources/phosphor-power/phosphor-regulators/test/run-ci.sh',
'.../openbmc-sources/entity-manager/scripts/run-ci.sh']

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ib420c69e88e88ca0a50863c83ad43fac750d3e7a

show more ...

ci-test: Add options to disable integration tests

Enable integration tests by default.
Give the user an option(--no-integration-tests) to disable them.

```
usage: unit-test.py [-h] -w WORKSPACE -p

ci-test: Add options to disable integration tests

Enable integration tests by default.
Give the user an option(--no-integration-tests) to disable them.

```
usage: unit-test.py [-h] -w WORKSPACE -p PACKAGE [-t]
[--integration-tests | --no-integration-tests] [-v]
[-r REPEAT] [-b BRANCH] [-n]
...
--integration-tests Enable integration tests [default].
--no-integration-tests
Disable integration tests.
```

This is needed, because some tests are run for a certain period
that increases the test execution time.

Tested on pid-control and host-ipmid proposed integration tests
with/without this option.

Signed-off-by: Ramin Izadpanah <iramin@google.com>
Change-Id: I21656dcdf300d990fef8d387725813e0dd1827e1

show more ...