xref: /openbmc/openbmc/poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst (revision 03514f19)

.. SPDX-License-Identifier: CC-BY-2.5

=========
Execution
=========

|

The primary purpose for running BitBake is to produce some kind of
output such as a single installable package, a kernel, a software
development kit, or even a full, board-specific bootable Linux image,
complete with bootloader, kernel, and root filesystem. Of course, you
can execute the ``bitbake`` command with options that cause it to
execute single tasks, compile single recipe files, capture or clear
data, or simply return information about the execution environment.

This chapter describes BitBake's execution process from start to finish
when you use it to create an image. The execution process is launched
using the following command form::

  $ bitbake target

For information on
the BitBake command and its options, see ":ref:`The BitBake Command
<bitbake-user-manual-command>`" section.

.. note::

   Prior to executing BitBake, you should take advantage of available
   parallel thread execution on your build host by setting the
   :term:`BB_NUMBER_THREADS` variable in
   your project's ``local.conf`` configuration file.

   A common method to determine this value for your build host is to run
   the following::

     $ grep processor /proc/cpuinfo

   This command returns
   the number of processors, which takes into account hyper-threading.
   Thus, a quad-core build host with hyper-threading most likely shows
   eight processors, which is the value you would then assign to
   :term:`BB_NUMBER_THREADS`.

   A possibly simpler solution is that some Linux distributions (e.g.
   Debian and Ubuntu) provide the ``ncpus`` command.

Parsing the Base Configuration Metadata
=======================================

The first thing BitBake does is parse base configuration metadata. Base
configuration metadata consists of your project's ``bblayers.conf`` file
to determine what layers BitBake needs to recognize, all necessary
``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
data itself is of various types:

-  **Recipes:** Details about particular pieces of software.

-  **Class Data:** An abstraction of common build information (e.g. how to
   build a Linux kernel).

-  **Configuration Data:** Machine-specific settings, policy decisions,
   and so forth. Configuration data acts as the glue to bind everything
   together.

The ``layer.conf`` files are used to construct key variables such as
:term:`BBPATH` and :term:`BBFILES`.
:term:`BBPATH` is used to search for configuration and class files under the
``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used
to locate both recipe and recipe append files (``.bb`` and
``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment.

Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable
that was just constructed. The ``bitbake.conf`` file may also include
other configuration files using the ``include`` or ``require``
directives.

Prior to parsing configuration files, BitBake looks at certain
variables, including:

-  :term:`BB_ENV_PASSTHROUGH`
-  :term:`BB_ENV_PASSTHROUGH_ADDITIONS`
-  :term:`BB_PRESERVE_ENV`
-  :term:`BB_ORIGENV`
-  :term:`BITBAKE_UI`

The first four variables in this list relate to how BitBake treats shell
environment variables during task execution. By default, BitBake cleans
the environment variables and provides tight control over the shell
execution environment. However, through the use of these first four
variables, you can apply your control regarding the environment
variables allowed to be used by BitBake in the shell during execution of
tasks. See the
":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
section and the information about these variables in the variable
glossary for more information on how they work and on how to use them.

The base configuration metadata is global and therefore affects all
recipes and tasks that are executed.

BitBake first searches the current working directory for an optional
``conf/bblayers.conf`` configuration file. This file is expected to
contain a :term:`BBLAYERS` variable that is a
space-delimited list of 'layer' directories. Recall that if BitBake
cannot find a ``bblayers.conf`` file, then it is assumed the user has
set the :term:`BBPATH` and :term:`BBFILES` variables directly in the
environment.

For each directory (layer) in this list, a ``conf/layer.conf`` file is
located and parsed with the :term:`LAYERDIR` variable
being set to the directory where the layer was found. The idea is these
files automatically set up :term:`BBPATH` and other
variables correctly for a given build directory.

BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
the user-specified :term:`BBPATH`. That configuration file generally has
include directives to pull in any other metadata such as files specific
to the architecture, the machine, the local environment, and so forth.

Only variable definitions and include directives are allowed in BitBake
``.conf`` files. Some variables directly influence BitBake's behavior.
These variables might have been set from the environment depending on
the environment variables previously mentioned or set in the
configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
chapter presents a full list of
variables.

After parsing configuration files, BitBake uses its rudimentary
inheritance mechanism, which is through class files, to inherit some
standard classes. BitBake parses a class when the inherit directive
responsible for getting that class is encountered.

The ``base.bbclass`` file is always included. Other classes that are
specified in the configuration using the
:term:`INHERIT` variable are also included. BitBake
searches for class files in a ``classes`` subdirectory under the paths
in :term:`BBPATH` in the same way as configuration files.

A good way to get an idea of the configuration files and the class files
used in your execution environment is to run the following BitBake
command::

  $ bitbake -e > mybb.log

Examining the top of the ``mybb.log``
shows you the many configuration files and class files used in your
execution environment.

.. note::

   You need to be aware of how BitBake parses curly braces. If a recipe
   uses a closing curly brace within the function and the character has
   no leading spaces, BitBake produces a parsing error. If you use a
   pair of curly braces in a shell function, the closing curly brace
   must not be located at the start of the line without leading spaces.

   Here is an example that causes BitBake to produce a parsing error::

      fakeroot create_shar() {
         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
      usage()
      {
         echo "test"
         ######  The following "}" at the start of the line causes a parsing error ######
      }
      EOF
      }

      Writing the recipe this way avoids the error:
      fakeroot create_shar() {
         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
      usage()
      {
         echo "test"
         ###### The following "}" with a leading space at the start of the line avoids the error ######
       }
      EOF
      }

Locating and Parsing Recipes
============================

During the configuration phase, BitBake will have set
:term:`BBFILES`. BitBake now uses it to construct a
list of recipes to parse, along with any append files (``.bbappend``) to
apply. :term:`BBFILES` is a space-separated list of available files and
supports wildcards. An example would be::

  BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"

BitBake parses each
recipe and append file located with :term:`BBFILES` and stores the values of
various variables into the datastore.

.. note::

   Append files are applied in the order they are encountered in BBFILES.

For each file, a fresh copy of the base configuration is made, then the
recipe is parsed line by line. Any inherit statements cause BitBake to
find and then parse class files (``.bbclass``) using
:term:`BBPATH` as the search path. Finally, BitBake
parses in order any append files found in :term:`BBFILES`.

One common convention is to use the recipe filename to define pieces of
metadata. For example, in ``bitbake.conf`` the recipe name and version
are used to set the variables :term:`PN` and
:term:`PV`::

   PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
   PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"

In this example, a recipe called "something_1.2.3.bb" would set
:term:`PN` to "something" and :term:`PV` to "1.2.3".

By the time parsing is complete for a recipe, BitBake has a list of
tasks that the recipe defines and a set of data consisting of keys and
values as well as dependency information about the tasks.

BitBake does not need all of this information. It only needs a small
subset of the information to make decisions about the recipe.
Consequently, BitBake caches the values in which it is interested and
does not store the rest of the information. Experience has shown it is
faster to re-parse the metadata than to try and write it out to the disk
and then reload it.

Where possible, subsequent BitBake commands reuse this cache of recipe
information. The validity of this cache is determined by first computing
a checksum of the base configuration data (see
:term:`BB_HASHCONFIG_IGNORE_VARS`) and
then checking if the checksum matches. If that checksum matches what is
in the cache and the recipe and class files have not changed, BitBake is
able to use the cache. BitBake then reloads the cached information about
the recipe instead of reparsing it from scratch.

Recipe file collections exist to allow the user to have multiple
repositories of ``.bb`` files that contain the same exact package. For
example, one could easily use them to make one's own local copy of an
upstream repository, but with custom modifications that one does not
want upstream. Here is an example::

  BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
  BBFILE_COLLECTIONS = "upstream local"
  BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
  BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
  BBFILE_PRIORITY_upstream = "5"
  BBFILE_PRIORITY_local = "10"

.. note::

   The layers mechanism is now the preferred method of collecting code.
   While the collections code remains, its main use is to set layer
   priorities and to deal with overlap (conflicts) between layers.

.. _bb-bitbake-providers:

Providers
=========

Assuming BitBake has been instructed to execute a target and that all
the recipe files have been parsed, BitBake starts to figure out how to
build the target. BitBake looks through the :term:`PROVIDES` list for each
of the recipes. A :term:`PROVIDES` list is the list of names by which the
recipe can be known. Each recipe's :term:`PROVIDES` list is created
implicitly through the recipe's :term:`PN` variable and
explicitly through the recipe's :term:`PROVIDES`
variable, which is optional.

When a recipe uses :term:`PROVIDES`, that recipe's functionality can be
found under an alternative name or names other than the implicit :term:`PN`
name. As an example, suppose a recipe named ``keyboard_1.0.bb``
contained the following::

  PROVIDES += "fullkeyboard"

The :term:`PROVIDES`
list for this recipe becomes "keyboard", which is implicit, and
"fullkeyboard", which is explicit. Consequently, the functionality found
in ``keyboard_1.0.bb`` can be found under two different names.

.. _bb-bitbake-preferences:

Preferences
===========

The :term:`PROVIDES` list is only part of the solution for figuring out a
target's recipes. Because targets might have multiple providers, BitBake
needs to prioritize providers by determining provider preferences.

A common example in which a target has multiple providers is
"virtual/kernel", which is on the :term:`PROVIDES` list for each kernel
recipe. Each machine often selects the best kernel provider by using a
line similar to the following in the machine configuration file::

  PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"

The default :term:`PREFERRED_PROVIDER` is the provider
with the same name as the target. BitBake iterates through each target
it needs to build and resolves them and their dependencies using this
process.

Understanding how providers are chosen is made complicated by the fact
that multiple versions might exist for a given provider. BitBake
defaults to the highest version of a provider. Version comparisons are
made using the same method as Debian. You can use the
:term:`PREFERRED_VERSION` variable to
specify a particular version. You can influence the order by using the
:term:`DEFAULT_PREFERENCE` variable.

By default, files have a preference of "0". Setting
:term:`DEFAULT_PREFERENCE` to "-1" makes the recipe unlikely to be used
unless it is explicitly referenced. Setting :term:`DEFAULT_PREFERENCE` to
"1" makes it likely the recipe is used. :term:`PREFERRED_VERSION` overrides
any :term:`DEFAULT_PREFERENCE` setting. :term:`DEFAULT_PREFERENCE` is often used
to mark newer and more experimental recipe versions until they have
undergone sufficient testing to be considered stable.

When there are multiple "versions" of a given recipe, BitBake defaults
to selecting the most recent version, unless otherwise specified. If the
recipe in question has a
:term:`DEFAULT_PREFERENCE` set lower than
the other recipes (default is 0), then it will not be selected. This
allows the person or persons maintaining the repository of recipe files
to specify their preference for the default selected version.
Additionally, the user can specify their preferred version.

If the first recipe is named ``a_1.1.bb``, then the
:term:`PN` variable will be set to "a", and the
:term:`PV` variable will be set to 1.1.

Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
default. However, if you define the following variable in a ``.conf``
file that BitBake parses, you can change that preference::

  PREFERRED_VERSION_a = "1.1"

.. note::

   It is common for a recipe to provide two versions -- a stable,
   numbered (and preferred) version, and a version that is automatically
   checked out from a source code repository that is considered more
   "bleeding edge" but can be selected only explicitly.

   For example, in the OpenEmbedded codebase, there is a standard,
   versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
   is also a Git-based version, ``busybox_git.bb``, which explicitly
   contains the line ::

     DEFAULT_PREFERENCE = "-1"

   to ensure that the
   numbered, stable version is always preferred unless the developer
   selects otherwise.

.. _bb-bitbake-dependencies:

Dependencies
============

Each target BitBake builds consists of multiple tasks such as ``fetch``,
``unpack``, ``patch``, ``configure``, and ``compile``. For best
performance on multi-core systems, BitBake considers each task as an
independent entity with its own set of dependencies.

Dependencies are defined through several variables. You can find
information about variables BitBake uses in the
:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
basic level, it is sufficient to know that BitBake uses the
:term:`DEPENDS` and
:term:`RDEPENDS` variables when calculating
dependencies.

For more information on how BitBake handles dependencies, see the
:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
section.

.. _ref-bitbake-tasklist:

The Task List
=============

Based on the generated list of providers and the dependency information,
BitBake can now calculate exactly what tasks it needs to run and in what
order it needs to run them. The
:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
section has more information on how BitBake chooses which task to
execute next.

The build now starts with BitBake forking off threads up to the limit
set in the :term:`BB_NUMBER_THREADS`
variable. BitBake continues to fork threads as long as there are tasks
ready to run, those tasks have all their dependencies met, and the
thread threshold has not been exceeded.

It is worth noting that you can greatly speed up the build time by
properly setting the :term:`BB_NUMBER_THREADS` variable.

As each task completes, a timestamp is written to the directory
specified by the :term:`STAMP` variable. On subsequent
runs, BitBake looks in the build directory within ``tmp/stamps`` and
does not rerun tasks that are already completed unless a timestamp is
found to be invalid. Currently, invalid timestamps are only considered
on a per recipe file basis. So, for example, if the configure stamp has
a timestamp greater than the compile timestamp for a given target, then
the compile task would rerun. Running the compile task again, however,
has no effect on other providers that depend on that target.

The exact format of the stamps is partly configurable. In modern
versions of BitBake, a hash is appended to the stamp so that if the
configuration changes, the stamp becomes invalid and the task is
automatically rerun. This hash, or signature used, is governed by the
signature policy that is configured (see the
:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
section for information). It is also
possible to append extra metadata to the stamp using the
``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
flag to make some tasks machine-specific.

.. note::

   Some tasks are marked as "nostamp" tasks. No timestamp file is
   created when these tasks are run. Consequently, "nostamp" tasks are
   always rerun.

For more information on tasks, see the
:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.

Executing Tasks
===============

Tasks can be either a shell task or a Python task. For shell tasks,
BitBake writes a shell script to
``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
executes the script. The generated shell script contains all the
exported variables, and the shell functions with all variables expanded.
Output from the shell script goes to the file
``${``\ :term:`T`\ ``}/log.do_taskname.pid``. Looking at the expanded shell functions in
the run file and the output in the log files is a useful debugging
technique.

For Python tasks, BitBake executes the task internally and logs
information to the controlling terminal. Future versions of BitBake will
write the functions to files similar to the way shell tasks are handled.
Logging will be handled in a way similar to shell tasks as well.

The order in which BitBake runs the tasks is controlled by its task
scheduler. It is possible to configure the scheduler and define custom
implementations for specific use cases. For more information, see these
variables that control the behavior:

-  :term:`BB_SCHEDULER`

-  :term:`BB_SCHEDULERS`

It is possible to have functions run before and after a task's main
function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
flags of the task that lists the functions to run.

.. _checksums:

Checksums (Signatures)
======================

A checksum is a unique signature of a task's inputs. The signature of a
task can be used to determine if a task needs to be run. Because it is a
change in a task's inputs that triggers running the task, BitBake needs
to detect all the inputs to a given task. For shell tasks, this turns
out to be fairly easy because BitBake generates a "run" shell script for
each task and it is possible to create a checksum that gives you a good
idea of when the task's data changes.

To complicate the problem, some things should not be included in the
checksum. First, there is the actual specific build path of a given task
- the working directory. It does not matter if the working directory
changes because it should not affect the output for target packages. The
simplistic approach for excluding the working directory is to set it to
some fixed value and create the checksum for the "run" script. BitBake
goes one step better and uses the
:term:`BB_BASEHASH_IGNORE_VARS` variable
to define a list of variables that should never be included when
generating the signatures.

Another problem results from the "run" scripts containing functions that
might or might not get called. The incremental build solution contains
code that figures out dependencies between shell functions. This code is
used to prune the "run" scripts down to the minimum set, thereby
alleviating this problem and making the "run" scripts much more readable
as a bonus.

So far we have solutions for shell scripts. What about Python tasks? The
same approach applies even though these tasks are more difficult. The
process needs to figure out what variables a Python function accesses
and what functions it calls. Again, the incremental build solution
contains code that first figures out the variable and function
dependencies, and then creates a checksum for the data used as the input
to the task.

Like the working directory case, situations exist where dependencies
should be ignored. For these cases, you can instruct the build process
to ignore a dependency by using a line like the following::

  PACKAGE_ARCHS[vardepsexclude] = "MACHINE"

This example ensures that the
``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
even if it does reference it.

Equally, there are cases where we need to add dependencies BitBake is
not able to find. You can accomplish this by using a line like the
following::

  PACKAGE_ARCHS[vardeps] = "MACHINE"

This example explicitly
adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.

Consider a case with in-line Python, for example, where BitBake is not
able to figure out dependencies. When running in debug mode (i.e. using
``-DDD``), BitBake produces output when it discovers something for which
it cannot figure out dependencies.

Thus far, this section has limited discussion to the direct inputs into
a task. Information based on direct inputs is referred to as the
"basehash" in the code. However, there is still the question of a task's
indirect inputs --- the things that were already built and present in the
build directory. The checksum (or signature) for a particular task needs
to add the hashes of all the tasks on which the particular task depends.
Choosing which dependencies to add is a policy decision. However, the
effect is to generate a master checksum that combines the basehash and
the hashes of the task's dependencies.

At the code level, there are a variety of ways both the basehash and the
dependent task hashes can be influenced. Within the BitBake
configuration file, we can give BitBake some extra information to help
it construct the basehash. The following statement effectively results
in a list of global variable dependency excludes --- variables never
included in any checksum. This example uses variables from OpenEmbedded
to help illustrate the concept::

   BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
       SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
       USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
       PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
       CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"

The previous example excludes the work directory, which is part of
``TMPDIR``.

The rules for deciding which hashes of dependent tasks to include
through dependency chains are more complex and are generally
accomplished with a Python function. The code in
``meta/lib/oe/sstatesig.py`` shows two examples of this and also
illustrates how you can insert your own policy into the system if so
desired. This file defines the basic signature generator
OpenEmbedded-Core uses: "OEBasicHash". By default, there
is a dummy "noop" signature handler enabled in BitBake. This means that
behavior is unchanged from previous versions. ``OE-Core`` uses the
"OEBasicHash" signature handler by default through this setting in the
``bitbake.conf`` file::

  BB_SIGNATURE_HANDLER ?= "OEBasicHash"

The main feature of the "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is that
it adds the task hash to the stamp files. Thanks to this, any metadata
change will change the task hash, automatically causing the task to be run
again. This removes the need to bump :term:`PR` values, and changes to
metadata automatically ripple across the build.

It is also worth noting that the end result of signature
generators is to make some dependency and hash information available to
the build. This information includes:

-  ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
   recipe.

-  ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
   dependent task.

-  :term:`BB_TASKHASH`: The hash of the currently running task.

It is worth noting that BitBake's "-S" option lets you debug BitBake's
processing of signatures. The options passed to -S allow different
debugging modes to be used, either using BitBake's own debug functions
or possibly those defined in the metadata/signature handler itself. The
simplest parameter to pass is "none", which causes a set of signature
information to be written out into ``STAMPS_DIR`` corresponding to the
targets specified. The other currently available parameter is
"printdiff", which causes BitBake to try to establish the most recent
signature match it can (e.g. in the sstate cache) and then run
compare the matched signatures to determine the stamps and delta
where these two stamp trees diverge. This can be used to determine why
tasks need to be re-run in situations where that is not expected.

.. note::

   It is likely that future versions of BitBake will provide other
   signature handlers triggered through additional "-S" parameters.

You can find more information on checksum metadata in the
:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
section.

Setscene
========

The setscene process enables BitBake to handle "pre-built" artifacts.
The ability to handle and reuse these artifacts allows BitBake the
luxury of not having to build something from scratch every time.
Instead, BitBake can use, when possible, existing build artifacts.

BitBake needs to have reliable data indicating whether or not an
artifact is compatible. Signatures, described in the previous section,
provide an ideal way of representing whether an artifact is compatible.
If a signature is the same, an object can be reused.

If an object can be reused, the problem then becomes how to replace a
given task or set of tasks with the pre-built artifact. BitBake solves
the problem with the "setscene" process.

When BitBake is asked to build a given target, before building anything,
it first asks whether cached information is available for any of the
targets it's building, or any of the intermediate targets. If cached
information is available, BitBake uses this information instead of
running the main tasks.

BitBake first calls the function defined by the
:term:`BB_HASHCHECK_FUNCTION` variable
with a list of tasks and corresponding hashes it wants to build. This
function is designed to be fast and returns a list of the tasks for
which it believes in can obtain artifacts.

Next, for each of the tasks that were returned as possibilities, BitBake
executes a setscene version of the task that the possible artifact
covers. Setscene versions of a task have the string "_setscene" appended
to the task name. So, for example, the task with the name ``xxx`` has a
setscene task named ``xxx_setscene``. The setscene version of the task
executes and provides the necessary artifacts returning either success
or failure.

As previously mentioned, an artifact can cover more than one task. For
example, it is pointless to obtain a compiler if you already have the
compiled binary. To handle this, BitBake calls the
:term:`BB_SETSCENE_DEPVALID` function for
each successful setscene task to know whether or not it needs to obtain
the dependencies of that task.

You can find more information on setscene metadata in the
:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
section.

Logging
=======

In addition to the standard command line option to control how verbose
builds are when execute, bitbake also supports user defined
configuration of the `Python
logging <https://docs.python.org/3/library/logging.html>`__ facilities
through the :term:`BB_LOGCONFIG` variable. This
variable defines a JSON or YAML `logging
configuration <https://docs.python.org/3/library/logging.config.html>`__
that will be intelligently merged into the default configuration. The
logging configuration is merged using the following rules:

-  The user defined configuration will completely replace the default
   configuration if top level key ``bitbake_merge`` is set to the value
   ``False``. In this case, all other rules are ignored.

-  The user configuration must have a top level ``version`` which must
   match the value of the default configuration.

-  Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
   will be merged into the same section in the default configuration,
   with the user specified keys taking replacing a default one if there
   is a conflict. In practice, this means that if both the default
   configuration and user configuration specify a handler named
   ``myhandler``, the user defined one will replace the default. To
   prevent the user from inadvertently replacing a default handler,
   formatter, or filter, all of the default ones are named with a prefix
   of "``BitBake.``"

-  If a logger is defined by the user with the key ``bitbake_merge`` set
   to ``False``, that logger will be completely replaced by user
   configuration. In this case, no other rules will apply to that
   logger.

-  All user defined ``filter`` and ``handlers`` properties for a given
   logger will be merged with corresponding properties from the default
   logger. For example, if the user configuration adds a filter called
   ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
   adds a filter called ``BitBake.defaultFilter``, both filters will be
   applied to the logger

As a first example, you can create a ``hashequiv.json`` user logging
configuration file to log all Hash Equivalence related messages of ``VERBOSE``
or higher priority to a file called ``hashequiv.log``::

   {
       "version": 1,
       "handlers": {
           "autobuilderlog": {
               "class": "logging.FileHandler",
               "formatter": "logfileFormatter",
               "level": "DEBUG",
               "filename": "hashequiv.log",
               "mode": "w"
           }
       },
       "formatters": {
               "logfileFormatter": {
                   "format": "%(name)s: %(levelname)s: %(message)s"
               }
       },
       "loggers": {
           "BitBake.SigGen.HashEquiv": {
               "level": "VERBOSE",
               "handlers": ["autobuilderlog"]
           },
           "BitBake.RunQueue.HashEquiv": {
               "level": "VERBOSE",
               "handlers": ["autobuilderlog"]
           }
       }
   }

Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``::

   BB_LOGCONFIG = "hashequiv.json"

Another example is this ``warn.json`` file to log all ``WARNING`` and
higher priority messages to a ``warn.log`` file::

  {
      "version": 1,
      "formatters": {
          "warnlogFormatter": {
              "()": "bb.msg.BBLogFormatter",
              "format": "%(levelname)s: %(message)s"
          }
      },

      "handlers": {
          "warnlog": {
              "class": "logging.FileHandler",
              "formatter": "warnlogFormatter",
              "level": "WARNING",
              "filename": "warn.log"
          }
      },

      "loggers": {
          "BitBake": {
              "handlers": ["warnlog"]
          }
      },

      "@disable_existing_loggers": false
  }

Note that BitBake's helper classes for structured logging are implemented in
``lib/bb/msg.py``.