1.. SPDX-License-Identifier: CC-BY-2.5
2
3=========
4Execution
5=========
6
7|
8
9The primary purpose for running BitBake is to produce some kind of
10output such as a single installable package, a kernel, a software
11development kit, or even a full, board-specific bootable Linux image,
12complete with bootloader, kernel, and root filesystem. Of course, you
13can execute the ``bitbake`` command with options that cause it to
14execute single tasks, compile single recipe files, capture or clear
15data, or simply return information about the execution environment.
16
17This chapter describes BitBake's execution process from start to finish
18when you use it to create an image. The execution process is launched
19using the following command form::
20
21  $ bitbake target
22
23For information on
24the BitBake command and its options, see ":ref:`The BitBake Command
25<bitbake-user-manual-command>`" section.
26
27.. note::
28
29   Prior to executing BitBake, you should take advantage of available
30   parallel thread execution on your build host by setting the
31   :term:`BB_NUMBER_THREADS` variable in
32   your project's ``local.conf`` configuration file.
33
34   A common method to determine this value for your build host is to run
35   the following::
36
37     $ grep processor /proc/cpuinfo
38
39   This command returns
40   the number of processors, which takes into account hyper-threading.
41   Thus, a quad-core build host with hyper-threading most likely shows
42   eight processors, which is the value you would then assign to
43   :term:`BB_NUMBER_THREADS`.
44
45   A possibly simpler solution is that some Linux distributions (e.g.
46   Debian and Ubuntu) provide the ``ncpus`` command.
47
48Parsing the Base Configuration Metadata
49=======================================
50
51The first thing BitBake does is parse base configuration metadata. Base
52configuration metadata consists of your project's ``bblayers.conf`` file
53to determine what layers BitBake needs to recognize, all necessary
54``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
55data itself is of various types:
56
57-  **Recipes:** Details about particular pieces of software.
58
59-  **Class Data:** An abstraction of common build information (e.g. how to
60   build a Linux kernel).
61
62-  **Configuration Data:** Machine-specific settings, policy decisions,
63   and so forth. Configuration data acts as the glue to bind everything
64   together.
65
66The ``layer.conf`` files are used to construct key variables such as
67:term:`BBPATH` and :term:`BBFILES`.
68:term:`BBPATH` is used to search for configuration and class files under the
69``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used
70to locate both recipe and recipe append files (``.bb`` and
71``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
72user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment.
73
74Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable
75that was just constructed. The ``bitbake.conf`` file may also include
76other configuration files using the ``include`` or ``require``
77directives.
78
79Prior to parsing configuration files, BitBake looks at certain
80variables, including:
81
82-  :term:`BB_ENV_PASSTHROUGH`
83-  :term:`BB_ENV_PASSTHROUGH_ADDITIONS`
84-  :term:`BB_PRESERVE_ENV`
85-  :term:`BB_ORIGENV`
86-  :term:`BITBAKE_UI`
87
88The first four variables in this list relate to how BitBake treats shell
89environment variables during task execution. By default, BitBake cleans
90the environment variables and provides tight control over the shell
91execution environment. However, through the use of these first four
92variables, you can apply your control regarding the environment
93variables allowed to be used by BitBake in the shell during execution of
94tasks. See the
95":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
96section and the information about these variables in the variable
97glossary for more information on how they work and on how to use them.
98
99The base configuration metadata is global and therefore affects all
100recipes and tasks that are executed.
101
102BitBake first searches the current working directory for an optional
103``conf/bblayers.conf`` configuration file. This file is expected to
104contain a :term:`BBLAYERS` variable that is a
105space-delimited list of 'layer' directories. Recall that if BitBake
106cannot find a ``bblayers.conf`` file, then it is assumed the user has
107set the :term:`BBPATH` and :term:`BBFILES` variables directly in the
108environment.
109
110For each directory (layer) in this list, a ``conf/layer.conf`` file is
111located and parsed with the :term:`LAYERDIR` variable
112being set to the directory where the layer was found. The idea is these
113files automatically set up :term:`BBPATH` and other
114variables correctly for a given build directory.
115
116BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
117the user-specified :term:`BBPATH`. That configuration file generally has
118include directives to pull in any other metadata such as files specific
119to the architecture, the machine, the local environment, and so forth.
120
121Only variable definitions and include directives are allowed in BitBake
122``.conf`` files. Some variables directly influence BitBake's behavior.
123These variables might have been set from the environment depending on
124the environment variables previously mentioned or set in the
125configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
126chapter presents a full list of
127variables.
128
129After parsing configuration files, BitBake uses its rudimentary
130inheritance mechanism, which is through class files, to inherit some
131standard classes. BitBake parses a class when the inherit directive
132responsible for getting that class is encountered.
133
134The ``base.bbclass`` file is always included. Other classes that are
135specified in the configuration using the
136:term:`INHERIT` variable are also included. BitBake
137searches for class files in a ``classes`` subdirectory under the paths
138in :term:`BBPATH` in the same way as configuration files.
139
140A good way to get an idea of the configuration files and the class files
141used in your execution environment is to run the following BitBake
142command::
143
144  $ bitbake -e > mybb.log
145
146Examining the top of the ``mybb.log``
147shows you the many configuration files and class files used in your
148execution environment.
149
150.. note::
151
152   You need to be aware of how BitBake parses curly braces. If a recipe
153   uses a closing curly brace within the function and the character has
154   no leading spaces, BitBake produces a parsing error. If you use a
155   pair of curly braces in a shell function, the closing curly brace
156   must not be located at the start of the line without leading spaces.
157
158   Here is an example that causes BitBake to produce a parsing error::
159
160      fakeroot create_shar() {
161         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
162      usage()
163      {
164         echo "test"
165         ######  The following "}" at the start of the line causes a parsing error ######
166      }
167      EOF
168      }
169
170      Writing the recipe this way avoids the error:
171      fakeroot create_shar() {
172         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
173      usage()
174      {
175         echo "test"
176         ###### The following "}" with a leading space at the start of the line avoids the error ######
177       }
178      EOF
179      }
180
181Locating and Parsing Recipes
182============================
183
184During the configuration phase, BitBake will have set
185:term:`BBFILES`. BitBake now uses it to construct a
186list of recipes to parse, along with any append files (``.bbappend``) to
187apply. :term:`BBFILES` is a space-separated list of available files and
188supports wildcards. An example would be::
189
190  BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
191
192BitBake parses each
193recipe and append file located with :term:`BBFILES` and stores the values of
194various variables into the datastore.
195
196.. note::
197
198   Append files are applied in the order they are encountered in BBFILES.
199
200For each file, a fresh copy of the base configuration is made, then the
201recipe is parsed line by line. Any inherit statements cause BitBake to
202find and then parse class files (``.bbclass``) using
203:term:`BBPATH` as the search path. Finally, BitBake
204parses in order any append files found in :term:`BBFILES`.
205
206One common convention is to use the recipe filename to define pieces of
207metadata. For example, in ``bitbake.conf`` the recipe name and version
208are used to set the variables :term:`PN` and
209:term:`PV`::
210
211   PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
212   PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
213
214In this example, a recipe called "something_1.2.3.bb" would set
215:term:`PN` to "something" and :term:`PV` to "1.2.3".
216
217By the time parsing is complete for a recipe, BitBake has a list of
218tasks that the recipe defines and a set of data consisting of keys and
219values as well as dependency information about the tasks.
220
221BitBake does not need all of this information. It only needs a small
222subset of the information to make decisions about the recipe.
223Consequently, BitBake caches the values in which it is interested and
224does not store the rest of the information. Experience has shown it is
225faster to re-parse the metadata than to try and write it out to the disk
226and then reload it.
227
228Where possible, subsequent BitBake commands reuse this cache of recipe
229information. The validity of this cache is determined by first computing
230a checksum of the base configuration data (see
231:term:`BB_HASHCONFIG_IGNORE_VARS`) and
232then checking if the checksum matches. If that checksum matches what is
233in the cache and the recipe and class files have not changed, BitBake is
234able to use the cache. BitBake then reloads the cached information about
235the recipe instead of reparsing it from scratch.
236
237Recipe file collections exist to allow the user to have multiple
238repositories of ``.bb`` files that contain the same exact package. For
239example, one could easily use them to make one's own local copy of an
240upstream repository, but with custom modifications that one does not
241want upstream. Here is an example::
242
243  BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
244  BBFILE_COLLECTIONS = "upstream local"
245  BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
246  BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
247  BBFILE_PRIORITY_upstream = "5"
248  BBFILE_PRIORITY_local = "10"
249
250.. note::
251
252   The layers mechanism is now the preferred method of collecting code.
253   While the collections code remains, its main use is to set layer
254   priorities and to deal with overlap (conflicts) between layers.
255
256.. _bb-bitbake-providers:
257
258Providers
259=========
260
261Assuming BitBake has been instructed to execute a target and that all
262the recipe files have been parsed, BitBake starts to figure out how to
263build the target. BitBake looks through the :term:`PROVIDES` list for each
264of the recipes. A :term:`PROVIDES` list is the list of names by which the
265recipe can be known. Each recipe's :term:`PROVIDES` list is created
266implicitly through the recipe's :term:`PN` variable and
267explicitly through the recipe's :term:`PROVIDES`
268variable, which is optional.
269
270When a recipe uses :term:`PROVIDES`, that recipe's functionality can be
271found under an alternative name or names other than the implicit :term:`PN`
272name. As an example, suppose a recipe named ``keyboard_1.0.bb``
273contained the following::
274
275  PROVIDES += "fullkeyboard"
276
277The :term:`PROVIDES`
278list for this recipe becomes "keyboard", which is implicit, and
279"fullkeyboard", which is explicit. Consequently, the functionality found
280in ``keyboard_1.0.bb`` can be found under two different names.
281
282.. _bb-bitbake-preferences:
283
284Preferences
285===========
286
287The :term:`PROVIDES` list is only part of the solution for figuring out a
288target's recipes. Because targets might have multiple providers, BitBake
289needs to prioritize providers by determining provider preferences.
290
291A common example in which a target has multiple providers is
292"virtual/kernel", which is on the :term:`PROVIDES` list for each kernel
293recipe. Each machine often selects the best kernel provider by using a
294line similar to the following in the machine configuration file::
295
296  PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
297
298The default :term:`PREFERRED_PROVIDER` is the provider
299with the same name as the target. BitBake iterates through each target
300it needs to build and resolves them and their dependencies using this
301process.
302
303Understanding how providers are chosen is made complicated by the fact
304that multiple versions might exist for a given provider. BitBake
305defaults to the highest version of a provider. Version comparisons are
306made using the same method as Debian. You can use the
307:term:`PREFERRED_VERSION` variable to
308specify a particular version. You can influence the order by using the
309:term:`DEFAULT_PREFERENCE` variable.
310
311By default, files have a preference of "0". Setting
312:term:`DEFAULT_PREFERENCE` to "-1" makes the recipe unlikely to be used
313unless it is explicitly referenced. Setting :term:`DEFAULT_PREFERENCE` to
314"1" makes it likely the recipe is used. :term:`PREFERRED_VERSION` overrides
315any :term:`DEFAULT_PREFERENCE` setting. :term:`DEFAULT_PREFERENCE` is often used
316to mark newer and more experimental recipe versions until they have
317undergone sufficient testing to be considered stable.
318
319When there are multiple "versions" of a given recipe, BitBake defaults
320to selecting the most recent version, unless otherwise specified. If the
321recipe in question has a
322:term:`DEFAULT_PREFERENCE` set lower than
323the other recipes (default is 0), then it will not be selected. This
324allows the person or persons maintaining the repository of recipe files
325to specify their preference for the default selected version.
326Additionally, the user can specify their preferred version.
327
328If the first recipe is named ``a_1.1.bb``, then the
329:term:`PN` variable will be set to "a", and the
330:term:`PV` variable will be set to 1.1.
331
332Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
333default. However, if you define the following variable in a ``.conf``
334file that BitBake parses, you can change that preference::
335
336  PREFERRED_VERSION_a = "1.1"
337
338.. note::
339
340   It is common for a recipe to provide two versions -- a stable,
341   numbered (and preferred) version, and a version that is automatically
342   checked out from a source code repository that is considered more
343   "bleeding edge" but can be selected only explicitly.
344
345   For example, in the OpenEmbedded codebase, there is a standard,
346   versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
347   is also a Git-based version, ``busybox_git.bb``, which explicitly
348   contains the line ::
349
350     DEFAULT_PREFERENCE = "-1"
351
352   to ensure that the
353   numbered, stable version is always preferred unless the developer
354   selects otherwise.
355
356.. _bb-bitbake-dependencies:
357
358Dependencies
359============
360
361Each target BitBake builds consists of multiple tasks such as ``fetch``,
362``unpack``, ``patch``, ``configure``, and ``compile``. For best
363performance on multi-core systems, BitBake considers each task as an
364independent entity with its own set of dependencies.
365
366Dependencies are defined through several variables. You can find
367information about variables BitBake uses in the
368:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
369basic level, it is sufficient to know that BitBake uses the
370:term:`DEPENDS` and
371:term:`RDEPENDS` variables when calculating
372dependencies.
373
374For more information on how BitBake handles dependencies, see the
375:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
376section.
377
378.. _ref-bitbake-tasklist:
379
380The Task List
381=============
382
383Based on the generated list of providers and the dependency information,
384BitBake can now calculate exactly what tasks it needs to run and in what
385order it needs to run them. The
386:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
387section has more information on how BitBake chooses which task to
388execute next.
389
390The build now starts with BitBake forking off threads up to the limit
391set in the :term:`BB_NUMBER_THREADS`
392variable. BitBake continues to fork threads as long as there are tasks
393ready to run, those tasks have all their dependencies met, and the
394thread threshold has not been exceeded.
395
396It is worth noting that you can greatly speed up the build time by
397properly setting the :term:`BB_NUMBER_THREADS` variable.
398
399As each task completes, a timestamp is written to the directory
400specified by the :term:`STAMP` variable. On subsequent
401runs, BitBake looks in the build directory within ``tmp/stamps`` and
402does not rerun tasks that are already completed unless a timestamp is
403found to be invalid. Currently, invalid timestamps are only considered
404on a per recipe file basis. So, for example, if the configure stamp has
405a timestamp greater than the compile timestamp for a given target, then
406the compile task would rerun. Running the compile task again, however,
407has no effect on other providers that depend on that target.
408
409The exact format of the stamps is partly configurable. In modern
410versions of BitBake, a hash is appended to the stamp so that if the
411configuration changes, the stamp becomes invalid and the task is
412automatically rerun. This hash, or signature used, is governed by the
413signature policy that is configured (see the
414:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
415section for information). It is also
416possible to append extra metadata to the stamp using the
417``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
418flag to make some tasks machine-specific.
419
420.. note::
421
422   Some tasks are marked as "nostamp" tasks. No timestamp file is
423   created when these tasks are run. Consequently, "nostamp" tasks are
424   always rerun.
425
426For more information on tasks, see the
427:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.
428
429Executing Tasks
430===============
431
432Tasks can be either a shell task or a Python task. For shell tasks,
433BitBake writes a shell script to
434``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
435executes the script. The generated shell script contains all the
436exported variables, and the shell functions with all variables expanded.
437Output from the shell script goes to the file
438``${``\ :term:`T`\ ``}/log.do_taskname.pid``. Looking at the expanded shell functions in
439the run file and the output in the log files is a useful debugging
440technique.
441
442For Python tasks, BitBake executes the task internally and logs
443information to the controlling terminal. Future versions of BitBake will
444write the functions to files similar to the way shell tasks are handled.
445Logging will be handled in a way similar to shell tasks as well.
446
447The order in which BitBake runs the tasks is controlled by its task
448scheduler. It is possible to configure the scheduler and define custom
449implementations for specific use cases. For more information, see these
450variables that control the behavior:
451
452-  :term:`BB_SCHEDULER`
453
454-  :term:`BB_SCHEDULERS`
455
456It is possible to have functions run before and after a task's main
457function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
458flags of the task that lists the functions to run.
459
460.. _checksums:
461
462Checksums (Signatures)
463======================
464
465A checksum is a unique signature of a task's inputs. The signature of a
466task can be used to determine if a task needs to be run. Because it is a
467change in a task's inputs that triggers running the task, BitBake needs
468to detect all the inputs to a given task. For shell tasks, this turns
469out to be fairly easy because BitBake generates a "run" shell script for
470each task and it is possible to create a checksum that gives you a good
471idea of when the task's data changes.
472
473To complicate the problem, some things should not be included in the
474checksum. First, there is the actual specific build path of a given task
475- the working directory. It does not matter if the working directory
476changes because it should not affect the output for target packages. The
477simplistic approach for excluding the working directory is to set it to
478some fixed value and create the checksum for the "run" script. BitBake
479goes one step better and uses the
480:term:`BB_BASEHASH_IGNORE_VARS` variable
481to define a list of variables that should never be included when
482generating the signatures.
483
484Another problem results from the "run" scripts containing functions that
485might or might not get called. The incremental build solution contains
486code that figures out dependencies between shell functions. This code is
487used to prune the "run" scripts down to the minimum set, thereby
488alleviating this problem and making the "run" scripts much more readable
489as a bonus.
490
491So far we have solutions for shell scripts. What about Python tasks? The
492same approach applies even though these tasks are more difficult. The
493process needs to figure out what variables a Python function accesses
494and what functions it calls. Again, the incremental build solution
495contains code that first figures out the variable and function
496dependencies, and then creates a checksum for the data used as the input
497to the task.
498
499Like the working directory case, situations exist where dependencies
500should be ignored. For these cases, you can instruct the build process
501to ignore a dependency by using a line like the following::
502
503  PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
504
505This example ensures that the
506``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
507even if it does reference it.
508
509Equally, there are cases where we need to add dependencies BitBake is
510not able to find. You can accomplish this by using a line like the
511following::
512
513  PACKAGE_ARCHS[vardeps] = "MACHINE"
514
515This example explicitly
516adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
517
518Consider a case with in-line Python, for example, where BitBake is not
519able to figure out dependencies. When running in debug mode (i.e. using
520``-DDD``), BitBake produces output when it discovers something for which
521it cannot figure out dependencies.
522
523Thus far, this section has limited discussion to the direct inputs into
524a task. Information based on direct inputs is referred to as the
525"basehash" in the code. However, there is still the question of a task's
526indirect inputs --- the things that were already built and present in the
527build directory. The checksum (or signature) for a particular task needs
528to add the hashes of all the tasks on which the particular task depends.
529Choosing which dependencies to add is a policy decision. However, the
530effect is to generate a master checksum that combines the basehash and
531the hashes of the task's dependencies.
532
533At the code level, there are a variety of ways both the basehash and the
534dependent task hashes can be influenced. Within the BitBake
535configuration file, we can give BitBake some extra information to help
536it construct the basehash. The following statement effectively results
537in a list of global variable dependency excludes --- variables never
538included in any checksum. This example uses variables from OpenEmbedded
539to help illustrate the concept::
540
541   BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
542       SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \
543       USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
544       PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
545       CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
546
547The previous example excludes the work directory, which is part of
548``TMPDIR``.
549
550The rules for deciding which hashes of dependent tasks to include
551through dependency chains are more complex and are generally
552accomplished with a Python function. The code in
553``meta/lib/oe/sstatesig.py`` shows two examples of this and also
554illustrates how you can insert your own policy into the system if so
555desired. This file defines the basic signature generator
556OpenEmbedded-Core uses: "OEBasicHash". By default, there
557is a dummy "noop" signature handler enabled in BitBake. This means that
558behavior is unchanged from previous versions. ``OE-Core`` uses the
559"OEBasicHash" signature handler by default through this setting in the
560``bitbake.conf`` file::
561
562  BB_SIGNATURE_HANDLER ?= "OEBasicHash"
563
564The main feature of the "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is that
565it adds the task hash to the stamp files. Thanks to this, any metadata
566change will change the task hash, automatically causing the task to be run
567again. This removes the need to bump :term:`PR` values, and changes to
568metadata automatically ripple across the build.
569
570It is also worth noting that the end result of signature
571generators is to make some dependency and hash information available to
572the build. This information includes:
573
574-  ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
575   recipe.
576
577-  ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
578   dependent task.
579
580-  :term:`BB_TASKHASH`: The hash of the currently running task.
581
582It is worth noting that BitBake's "-S" option lets you debug BitBake's
583processing of signatures. The options passed to -S allow different
584debugging modes to be used, either using BitBake's own debug functions
585or possibly those defined in the metadata/signature handler itself. The
586simplest parameter to pass is "none", which causes a set of signature
587information to be written out into ``STAMPS_DIR`` corresponding to the
588targets specified. The other currently available parameter is
589"printdiff", which causes BitBake to try to establish the most recent
590signature match it can (e.g. in the sstate cache) and then run
591compare the matched signatures to determine the stamps and delta
592where these two stamp trees diverge. This can be used to determine why
593tasks need to be re-run in situations where that is not expected.
594
595.. note::
596
597   It is likely that future versions of BitBake will provide other
598   signature handlers triggered through additional "-S" parameters.
599
600You can find more information on checksum metadata in the
601:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
602section.
603
604Setscene
605========
606
607The setscene process enables BitBake to handle "pre-built" artifacts.
608The ability to handle and reuse these artifacts allows BitBake the
609luxury of not having to build something from scratch every time.
610Instead, BitBake can use, when possible, existing build artifacts.
611
612BitBake needs to have reliable data indicating whether or not an
613artifact is compatible. Signatures, described in the previous section,
614provide an ideal way of representing whether an artifact is compatible.
615If a signature is the same, an object can be reused.
616
617If an object can be reused, the problem then becomes how to replace a
618given task or set of tasks with the pre-built artifact. BitBake solves
619the problem with the "setscene" process.
620
621When BitBake is asked to build a given target, before building anything,
622it first asks whether cached information is available for any of the
623targets it's building, or any of the intermediate targets. If cached
624information is available, BitBake uses this information instead of
625running the main tasks.
626
627BitBake first calls the function defined by the
628:term:`BB_HASHCHECK_FUNCTION` variable
629with a list of tasks and corresponding hashes it wants to build. This
630function is designed to be fast and returns a list of the tasks for
631which it believes in can obtain artifacts.
632
633Next, for each of the tasks that were returned as possibilities, BitBake
634executes a setscene version of the task that the possible artifact
635covers. Setscene versions of a task have the string "_setscene" appended
636to the task name. So, for example, the task with the name ``xxx`` has a
637setscene task named ``xxx_setscene``. The setscene version of the task
638executes and provides the necessary artifacts returning either success
639or failure.
640
641As previously mentioned, an artifact can cover more than one task. For
642example, it is pointless to obtain a compiler if you already have the
643compiled binary. To handle this, BitBake calls the
644:term:`BB_SETSCENE_DEPVALID` function for
645each successful setscene task to know whether or not it needs to obtain
646the dependencies of that task.
647
648You can find more information on setscene metadata in the
649:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
650section.
651
652Logging
653=======
654
655In addition to the standard command line option to control how verbose
656builds are when execute, bitbake also supports user defined
657configuration of the `Python
658logging <https://docs.python.org/3/library/logging.html>`__ facilities
659through the :term:`BB_LOGCONFIG` variable. This
660variable defines a JSON or YAML `logging
661configuration <https://docs.python.org/3/library/logging.config.html>`__
662that will be intelligently merged into the default configuration. The
663logging configuration is merged using the following rules:
664
665-  The user defined configuration will completely replace the default
666   configuration if top level key ``bitbake_merge`` is set to the value
667   ``False``. In this case, all other rules are ignored.
668
669-  The user configuration must have a top level ``version`` which must
670   match the value of the default configuration.
671
672-  Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
673   will be merged into the same section in the default configuration,
674   with the user specified keys taking replacing a default one if there
675   is a conflict. In practice, this means that if both the default
676   configuration and user configuration specify a handler named
677   ``myhandler``, the user defined one will replace the default. To
678   prevent the user from inadvertently replacing a default handler,
679   formatter, or filter, all of the default ones are named with a prefix
680   of "``BitBake.``"
681
682-  If a logger is defined by the user with the key ``bitbake_merge`` set
683   to ``False``, that logger will be completely replaced by user
684   configuration. In this case, no other rules will apply to that
685   logger.
686
687-  All user defined ``filter`` and ``handlers`` properties for a given
688   logger will be merged with corresponding properties from the default
689   logger. For example, if the user configuration adds a filter called
690   ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
691   adds a filter called ``BitBake.defaultFilter``, both filters will be
692   applied to the logger
693
694As a first example, you can create a ``hashequiv.json`` user logging
695configuration file to log all Hash Equivalence related messages of ``VERBOSE``
696or higher priority to a file called ``hashequiv.log``::
697
698   {
699       "version": 1,
700       "handlers": {
701           "autobuilderlog": {
702               "class": "logging.FileHandler",
703               "formatter": "logfileFormatter",
704               "level": "DEBUG",
705               "filename": "hashequiv.log",
706               "mode": "w"
707           }
708       },
709       "formatters": {
710               "logfileFormatter": {
711                   "format": "%(name)s: %(levelname)s: %(message)s"
712               }
713       },
714       "loggers": {
715           "BitBake.SigGen.HashEquiv": {
716               "level": "VERBOSE",
717               "handlers": ["autobuilderlog"]
718           },
719           "BitBake.RunQueue.HashEquiv": {
720               "level": "VERBOSE",
721               "handlers": ["autobuilderlog"]
722           }
723       }
724   }
725
726Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``::
727
728   BB_LOGCONFIG = "hashequiv.json"
729
730Another example is this ``warn.json`` file to log all ``WARNING`` and
731higher priority messages to a ``warn.log`` file::
732
733  {
734      "version": 1,
735      "formatters": {
736          "warnlogFormatter": {
737              "()": "bb.msg.BBLogFormatter",
738              "format": "%(levelname)s: %(message)s"
739          }
740      },
741
742      "handlers": {
743          "warnlog": {
744              "class": "logging.FileHandler",
745              "formatter": "warnlogFormatter",
746              "level": "WARNING",
747              "filename": "warn.log"
748          }
749      },
750
751      "loggers": {
752          "BitBake": {
753              "handlers": ["warnlog"]
754          }
755      },
756
757      "@disable_existing_loggers": false
758  }
759
760Note that BitBake's helper classes for structured logging are implemented in
761``lib/bb/msg.py``.
762