build-unit-test-docker: use Gerrit revision if available

If Jenkins is triggering us to run CI on a package that is included
in the Docker image itself, use the requested revision-under-

build-unit-test-docker: use Gerrit revision if available

If Jenkins is triggering us to run CI on a package that is included
in the Docker image itself, use the requested revision-under-test as
part of the Docker image build process.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I61a0a249123800aba860ab1535e96f5cdb2ec63f

show more ...

build-unit-test-docker: Fix https_proxy

The https_proxy was not correctly set and if build with proxy settings,
it gives below error:

base: WARNING: Retrying (Retry(total=4,

build-unit-test-docker: Fix https_proxy

The https_proxy was not correctly set and if build with proxy settings,
it gives below error:

base: WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.',
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
object at 0x7efe419e99d0>: Failed to establish a new connection:
[Errno -2] Name or service not known'))': /simple/inflection/
RAN: /usr/bin/docker build --build-arg http_proxy=http://<myproxy>
--build-arg 'https_proxy={https_proxy}' --network=host --force-rm
--no-cache=false -t
openbmc/ubuntu-unit-test-base:2021-W07-773f263e895d8ed6 -

Tested: Fix the https_proxy string and verify the CI passes.

Signed-off-by: Lei YU <yulei.sh@bytedance.com>
Change-Id: I87f543d33747b4957af83990f80bd8e832f4e5e3

show more ...

build-unit-test-docker: switch json to use cmake

nlohmann/json supports CMake and we use it in the bitbake
recipe, so use it here as well. The bitbake recipe has a
'install_append'

build-unit-test-docker: switch json to use cmake

nlohmann/json supports CMake and we use it in the bitbake
recipe, so use it here as well. The bitbake recipe has a
'install_append' to symlink the final header directly into
'{includedir}/json.hpp' so we need to continue to do the
same here.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ibf61a7a471216c43427109907a54e81dde1ae8fc

show more ...

build-unit-test-docker: pythonic refactoring

Do a bunch of clean-up to the script to better / modern Python
practices.

* Document the PackageDef dictionary.

* A

build-unit-test-docker: pythonic refactoring

Do a bunch of clean-up to the script to better / modern Python
practices.

* Document the PackageDef dictionary.

* Add type hinting so that 'mypy' static analysis passes (and
code self-documents better).

* Create a Package class to encapsulate everything related to
package stage builds and a Docker namespace to encapsulate
Docker operations.

* Overall better documentation.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I841d844aa9401889d428ed55c10bee6c1b0a7109

show more ...

build-unit-test-docker: update numerous packages

- Update packages per new bitbake recipes.
- Favor using the tag-name rather than a git hash, when available.
- Linux headers now req

build-unit-test-docker: update numerous packages

- Update packages per new bitbake recipes.
- Favor using the tag-name rather than a git hash, when available.
- Linux headers now requires rsync.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I90d045fa7ef69067ddf09685dfde3265cc1e948f

show more ...

build-unit-test-docker: create hashed tags for final image

Create a tag for the final image in the same format as the stage
images. Display this tag to stdout at the end of the docker b

build-unit-test-docker: create hashed tags for final image

Create a tag for the final image in the same format as the stage
images. Display this tag to stdout at the end of the docker build
so that other scripts can consume it via calls like:
$(scripts/build-unit-test-docker)

Modify `run-unit-test-docker.sh` to get the docker image from
`build-unit-test-docker` rather than a static image name. This
ensures that the docker image used in unit testing is the exact
image requested to be built. (Soon this will allow us to incorporate
Gerrit changes into the docker image itself.)

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Idd7a9a7b0157f1c520b33b3ec7ac0d5b52018d9e

show more ...

build-unit-test-docker: invalidate base image weekly

Docker will cache an image built with the exact same commands,
so we only get latest Ubuntu packages rarely. Add a date-based
ec

build-unit-test-docker: invalidate base image weekly

Docker will cache an image built with the exact same commands,
so we only get latest Ubuntu packages rarely. Add a date-based
echo command into the early stages of the base image build so
that we regularly get updates to all the packages in the base
image.

This invalidation, via echo, is only performed when running under
a Jenkins-like context. The purpose of this distinction is so that
we are more likely to cache failures induced by an underlying OS
update in CI, but when developers run this script locally they rarely
see an invalidation that is not due to a code package update.

Add an environment variable option "FORCE_DOCKER_BUILD" which can be
set by developers to bypass the Docker cache and force all images
to be rebuilt, just in case they really want to do that sort of thing.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Iec92ebada513975d0806099083384dfca623f144

show more ...

build-unit-test-docker: build docker stages in parallel

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I1899b2cacdf8f668c3f1226959d1b5a044b5fc6c

build-unit-test-docker: split docker stages into invocations

Rather than a single large Dockerfile, split each stage into its
own Dockerfile with its own invocation. This is in preparat

build-unit-test-docker: split docker stages into invocations

Rather than a single large Dockerfile, split each stage into its
own Dockerfile with its own invocation. This is in preparation
for building the docker stages in parallel.

Since multiple invocations of this script could be running in
parallel, possibly with different content, this requires creating
well-named tags for each stage. Create these tags as a combination
of the current date and the content that went into the stage
Dockerfile. The date is used so that we can refresh the base
image more regularly.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I77ac232f1603b6ce74c55f0f9466482ff30eb32e

show more ...

build-unit-test-docker: switch cmake to ninja backend

Ninja is faster and what is used by Yocto.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Icbb17ab466e03844

build-unit-test-docker: switch cmake to ninja backend

Ninja is faster and what is used by Yocto.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Icbb17ab466e03844b97c27869646cc580f77c558

show more ...

build-unit-test-docker: generate package directives

Generate the docker build directives for all packages by
performing a depth-first dependency search.

Signed-off-by: Patrick W

build-unit-test-docker: generate package directives

Generate the docker build directives for all packages by
performing a depth-first dependency search.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ia3b6778e0f15bd9d5c4cff5e5e7bf2a620c0e156

show more ...

build-unit-test-docker: templatize build directives

Create 'pkg_build' function that generates the docker RUN template
necessary to build all the packages.

Signed-off-by: Patric

build-unit-test-docker: templatize build directives

Create 'pkg_build' function that generates the docker RUN template
necessary to build all the packages.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ie7e7a3b3e78c67e6d97a83f3b43c3c54d10b77e6

show more ...

build-unit-test-docker: add dependencies to package def

Add dependencies to the package definition for each package and remove
the manual "COPY" commands to create the staged-dependencie

build-unit-test-docker: add dependencies to package def

Add dependencies to the package definition for each package and remove
the manual "COPY" commands to create the staged-dependencies.
Eventually this will also be used to parallelize the docker build
process.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ifb7e26fccea62ae9bc8d450a91d1fc4d9ef0db06

show more ...

build-unit-test-docker: simplify fetching

Most packages we use come from github, except for boost. Create a
'pkg_url' function to get the URL for a revision for curl calls.
This fun

build-unit-test-docker: simplify fetching

Most packages we use come from github, except for boost. Create a
'pkg_url' function to get the URL for a revision for curl calls.
This function automatically creates the URL from the package name,
unless the package has a 'url' property. If the 'url' property exists,
treat it as a function that generates the URL from the package name and
revision (to handle the boost case and any future non-github packages).

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Ifc47a1b580e6ec3728d8deeaa4c568017a80e03f

show more ...

build-unit-test-docker: use docker stagename everywhere

In some cases the script used a 'stagename' as the temporary docker
image used to create dependencies. Use this throughout for

build-unit-test-docker: use docker stagename everywhere

In some cases the script used a 'stagename' as the temporary docker
image used to create dependencies. Use this throughout for
consistency.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I8b2d8624747b8a36630dd27f6aee32ddf10defa7

show more ...

build-unit-test-docker: automatically lookup HEAD rev

Switch from a special 'head_pkgs' list to automatically looking
up the HEAD revision of any package that doesn't have a specified

build-unit-test-docker: automatically lookup HEAD rev

Switch from a special 'head_pkgs' list to automatically looking
up the HEAD revision of any package that doesn't have a specified
revision. Also, switch the 'pkg_rev' to a dictionary so we can add
additional fields later on.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I8ead996469966f087845c6b3cbd6eb5226f3e235

show more ...

build-unit-test-docker: convert to Python

Do a near identical conversion from bash to Python for the script which
builds the unit-test docker containers. I plan to do additional
enh

build-unit-test-docker: convert to Python

Do a near identical conversion from bash to Python for the script which
builds the unit-test docker containers. I plan to do additional
enhancements to build the sub-containers in parallel, so I needed to
first do this conversion to Python in order to make that implementation
sane.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I6df6f294b41f3a01f95fbe23a6dcd1ea21a315a9

show more ...