1.. SPDX-License-Identifier: CC-BY-2.5 2 3======== 4Overview 5======== 6 7| 8 9Welcome to the BitBake User Manual. This manual provides information on 10the BitBake tool. The information attempts to be as independent as 11possible regarding systems that use BitBake, such as OpenEmbedded and 12the Yocto Project. In some cases, scenarios or examples within the 13context of a build system are used in the manual to help with 14understanding. For these cases, the manual clearly states the context. 15 16.. _intro: 17 18Introduction 19============ 20 21Fundamentally, BitBake is a generic task execution engine that allows 22shell and Python tasks to be run efficiently and in parallel while 23working within complex inter-task dependency constraints. One of 24BitBake's main users, OpenEmbedded, takes this core and builds embedded 25Linux software stacks using a task-oriented approach. 26 27Conceptually, BitBake is similar to GNU Make in some regards but has 28significant differences: 29 30- BitBake executes tasks according to the provided metadata that builds up 31 the tasks. Metadata is stored in recipe (``.bb``) and related recipe 32 "append" (``.bbappend``) files, configuration (``.conf``) and 33 underlying include (``.inc``) files, and in class (``.bbclass``) 34 files. The metadata provides BitBake with instructions on what tasks 35 to run and the dependencies between those tasks. 36 37- BitBake includes a fetcher library for obtaining source code from 38 various places such as local files, source control systems, or 39 websites. 40 41- The instructions for each unit to be built (e.g. a piece of software) 42 are known as "recipe" files and contain all the information about the 43 unit (dependencies, source file locations, checksums, description and 44 so on). 45 46- BitBake includes a client/server abstraction and can be used from a 47 command line or used as a service over XML-RPC and has several 48 different user interfaces. 49 50History and Goals 51================= 52 53BitBake was originally a part of the OpenEmbedded project. It was 54inspired by the Portage package management system used by the Gentoo 55Linux distribution. On December 7, 2004, OpenEmbedded project team 56member Chris Larson split the project into two distinct pieces: 57 58- BitBake, a generic task executor 59 60- OpenEmbedded, a metadata set utilized by BitBake 61 62Today, BitBake is the primary basis of the 63`OpenEmbedded <https://www.openembedded.org/>`__ project, which is being 64used to build and maintain Linux distributions such as the `Poky 65Reference Distribution <https://www.yoctoproject.org/software-item/poky/>`__, 66developed under the umbrella of the `Yocto Project <https://www.yoctoproject.org>`__. 67 68Prior to BitBake, no other build tool adequately met the needs of an 69aspiring embedded Linux distribution. All of the build systems used by 70traditional desktop Linux distributions lacked important functionality, 71and none of the ad hoc Buildroot-based systems, prevalent in the 72embedded space, were scalable or maintainable. 73 74Some important original goals for BitBake were: 75 76- Handle cross-compilation. 77 78- Handle inter-package dependencies (build time on target architecture, 79 build time on native architecture, and runtime). 80 81- Support running any number of tasks within a given package, 82 including, but not limited to, fetching upstream sources, unpacking 83 them, patching them, configuring them, and so forth. 84 85- Be Linux distribution agnostic for both build and target systems. 86 87- Be architecture agnostic. 88 89- Support multiple build and target operating systems (e.g. Cygwin, the 90 BSDs, and so forth). 91 92- Be self-contained, rather than tightly integrated into the build 93 machine's root filesystem. 94 95- Handle conditional metadata on the target architecture, operating 96 system, distribution, and machine. 97 98- Be easy to use the tools to supply local metadata and packages 99 against which to operate. 100 101- Be easy to use BitBake to collaborate between multiple projects for 102 their builds. 103 104- Provide an inheritance mechanism to share common metadata between 105 many packages. 106 107Over time it became apparent that some further requirements were 108necessary: 109 110- Handle variants of a base recipe (e.g. native, sdk, and multilib). 111 112- Split metadata into layers and allow layers to enhance or override 113 other layers. 114 115- Allow representation of a given set of input variables to a task as a 116 checksum. Based on that checksum, allow acceleration of builds with 117 prebuilt components. 118 119BitBake satisfies all the original requirements and many more with 120extensions being made to the basic functionality to reflect the 121additional requirements. Flexibility and power have always been the 122priorities. BitBake is highly extensible and supports embedded Python 123code and execution of any arbitrary tasks. 124 125.. _Concepts: 126 127Concepts 128======== 129 130BitBake is a program written in the Python language. At the highest 131level, BitBake interprets metadata, decides what tasks are required to 132run, and executes those tasks. Similar to GNU Make, BitBake controls how 133software is built. GNU Make achieves its control through "makefiles", 134while BitBake uses "recipes". 135 136BitBake extends the capabilities of a simple tool like GNU Make by 137allowing for the definition of much more complex tasks, such as 138assembling entire embedded Linux distributions. 139 140The remainder of this section introduces several concepts that should be 141understood in order to better leverage the power of BitBake. 142 143Recipes 144------- 145 146BitBake Recipes, which are denoted by the file extension ``.bb``, are 147the most basic metadata files. These recipe files provide BitBake with 148the following: 149 150- Descriptive information about the package (author, homepage, license, 151 and so on) 152 153- The version of the recipe 154 155- Existing dependencies (both build and runtime dependencies) 156 157- Where the source code resides and how to fetch it 158 159- Whether the source code requires any patches, where to find them, and 160 how to apply them 161 162- How to configure and compile the source code 163 164- How to assemble the generated artifacts into one or more installable 165 packages 166 167- Where on the target machine to install the package or packages 168 created 169 170Within the context of BitBake, or any project utilizing BitBake as its 171build system, files with the ``.bb`` extension are referred to as 172recipes. 173 174.. note:: 175 176 The term "package" is also commonly used to describe recipes. 177 However, since the same word is used to describe packaged output from 178 a project, it is best to maintain a single descriptive term - 179 "recipes". Put another way, a single "recipe" file is quite capable 180 of generating a number of related but separately installable 181 "packages". In fact, that ability is fairly common. 182 183Configuration Files 184------------------- 185 186Configuration files, which are denoted by the ``.conf`` extension, 187define various configuration variables that govern the project's build 188process. These files fall into several areas that define machine 189configuration, distribution configuration, possible compiler tuning, 190general common configuration, and user configuration. The main 191configuration file is the sample ``bitbake.conf`` file, which is located 192within the BitBake source tree ``conf`` directory. 193 194Classes 195------- 196 197Class files, which are denoted by the ``.bbclass`` extension, contain 198information that is useful to share between metadata files. The BitBake 199source tree currently comes with one class metadata file called 200``base.bbclass``. You can find this file in the ``classes`` directory. 201The ``base.bbclass`` class files is special since it is always included 202automatically for all recipes and classes. This class contains 203definitions for standard basic tasks such as fetching, unpacking, 204configuring (empty by default), compiling (runs any Makefile present), 205installing (empty by default) and packaging (empty by default). These 206tasks are often overridden or extended by other classes added during the 207project development process. 208 209Layers 210------ 211 212Layers allow you to isolate different types of customizations from each 213other. While you might find it tempting to keep everything in one layer 214when working on a single project, the more modular your metadata, the 215easier it is to cope with future changes. 216 217To illustrate how you can use layers to keep things modular, consider 218customizations you might make to support a specific target machine. 219These types of customizations typically reside in a special layer, 220rather than a general layer, called a Board Support Package (BSP) layer. 221Furthermore, the machine customizations should be isolated from recipes 222and metadata that support a new GUI environment, for example. This 223situation gives you a couple of layers: one for the machine 224configurations and one for the GUI environment. It is important to 225understand, however, that the BSP layer can still make machine-specific 226additions to recipes within the GUI environment layer without polluting 227the GUI layer itself with those machine-specific changes. You can 228accomplish this through a recipe that is a BitBake append 229(``.bbappend``) file. 230 231.. _append-bbappend-files: 232 233Append Files 234------------ 235 236Append files, which are files that have the ``.bbappend`` file 237extension, extend or override information in an existing recipe file. 238 239BitBake expects every append file to have a corresponding recipe file. 240Furthermore, the append file and corresponding recipe file must use the 241same root filename. The filenames can differ only in the file type 242suffix used (e.g. ``formfactor_0.0.bb`` and 243``formfactor_0.0.bbappend``). 244 245Information in append files extends or overrides the information in the 246underlying, similarly-named recipe files. 247 248When you name an append file, you can use the "``%``" wildcard character 249to allow for matching recipe names. For example, suppose you have an 250append file named as follows:: 251 252 busybox_1.21.%.bbappend 253 254That append file 255would match any ``busybox_1.21.``\ x\ ``.bb`` version of the recipe. So, 256the append file would match the following recipe names:: 257 258 busybox_1.21.1.bb 259 busybox_1.21.2.bb 260 busybox_1.21.3.bb 261 262.. note:: 263 264 The use of the " % " character is limited in that it only works directly in 265 front of the .bbappend portion of the append file's name. You cannot use the 266 wildcard character in any other location of the name. 267 268If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the 269append name would not match. However, if you named the append file 270``busybox_1.%.bbappend``, then you would have a match. 271 272In the most general case, you could name the append file something as 273simple as ``busybox_%.bbappend`` to be entirely version independent. 274 275Obtaining BitBake 276================= 277 278You can obtain BitBake several different ways: 279 280- **Cloning BitBake:** Using Git to clone the BitBake source code 281 repository is the recommended method for obtaining BitBake. Cloning 282 the repository makes it easy to get bug fixes and have access to 283 stable branches and the master branch. Once you have cloned BitBake, 284 you should use the latest stable branch for development since the 285 master branch is for BitBake development and might contain less 286 stable changes. 287 288 You usually need a version of BitBake that matches the metadata you 289 are using. The metadata is generally backwards compatible but not 290 forward compatible. 291 292 Here is an example that clones the BitBake repository:: 293 294 $ git clone git://git.openembedded.org/bitbake 295 296 This command clones the BitBake 297 Git repository into a directory called ``bitbake``. Alternatively, 298 you can designate a directory after the ``git clone`` command if you 299 want to call the new directory something other than ``bitbake``. Here 300 is an example that names the directory ``bbdev``:: 301 302 $ git clone git://git.openembedded.org/bitbake bbdev 303 304- **Installation using your Distribution Package Management System:** 305 This method is not recommended because the BitBake version that is 306 provided by your distribution, in most cases, is several releases 307 behind a snapshot of the BitBake repository. 308 309- **Taking a snapshot of BitBake:** Downloading a snapshot of BitBake 310 from the source code repository gives you access to a known branch or 311 release of BitBake. 312 313 .. note:: 314 315 Cloning the Git repository, as described earlier, is the preferred 316 method for getting BitBake. Cloning the repository makes it easier 317 to update as patches are added to the stable branches. 318 319 The following example downloads a snapshot of BitBake version 1.17.0:: 320 321 $ wget https://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz 322 $ tar zxpvf bitbake-1.17.0.tar.gz 323 324 After extraction of the tarball using 325 the tar utility, you have a directory entitled ``bitbake-1.17.0``. 326 327- **Using the BitBake that Comes With Your Build Checkout:** A final 328 possibility for getting a copy of BitBake is that it already comes 329 with your checkout of a larger BitBake-based build system, such as 330 Poky. Rather than manually checking out individual layers and gluing 331 them together yourself, you can check out an entire build system. The 332 checkout will already include a version of BitBake that has been 333 thoroughly tested for compatibility with the other components. For 334 information on how to check out a particular BitBake-based build 335 system, consult that build system's supporting documentation. 336 337.. _bitbake-user-manual-command: 338 339The BitBake Command 340=================== 341 342The ``bitbake`` command is the primary interface to the BitBake tool. 343This section presents the BitBake command syntax and provides several 344execution examples. 345 346Usage and syntax 347---------------- 348 349Following is the usage and syntax for BitBake:: 350 351 $ bitbake -h 352 Usage: bitbake [options] [recipename/target recipe:do_task ...] 353 354 Executes the specified task (default is 'build') for a given set of target recipes (.bb files). 355 It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which 356 will provide the layer, BBFILES and other configuration information. 357 358 Options: 359 --version show program's version number and exit 360 -h, --help show this help message and exit 361 -b BUILDFILE, --buildfile=BUILDFILE 362 Execute tasks from a specific .bb recipe directly. 363 WARNING: Does not handle any dependencies from other 364 recipes. 365 -k, --continue Continue as much as possible after an error. While the 366 target that failed and anything depending on it cannot 367 be built, as much as possible will be built before 368 stopping. 369 -f, --force Force the specified targets/task to run (invalidating 370 any existing stamp file). 371 -c CMD, --cmd=CMD Specify the task to execute. The exact options 372 available depend on the metadata. Some examples might 373 be 'compile' or 'populate_sysroot' or 'listtasks' may 374 give a list of the tasks available. 375 -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP 376 Invalidate the stamp for the specified task such as 377 'compile' and then run the default task for the 378 specified target(s). 379 -r PREFILE, --read=PREFILE 380 Read the specified file before bitbake.conf. 381 -R POSTFILE, --postread=POSTFILE 382 Read the specified file after bitbake.conf. 383 -v, --verbose Enable tracing of shell tasks (with 'set -x'). Also 384 print bb.note(...) messages to stdout (in addition to 385 writing them to ${T}/log.do_<task>). 386 -D, --debug Increase the debug level. You can specify this more 387 than once. -D sets the debug level to 1, where only 388 bb.debug(1, ...) messages are printed to stdout; -DD 389 sets the debug level to 2, where both bb.debug(1, ...) 390 and bb.debug(2, ...) messages are printed; etc. 391 Without -D, no debug messages are printed. Note that 392 -D only affects output to stdout. All debug messages 393 are written to ${T}/log.do_taskname, regardless of the 394 debug level. 395 -q, --quiet Output less log message data to the terminal. You can 396 specify this more than once. 397 -n, --dry-run Don't execute, just go through the motions. 398 -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER 399 Dump out the signature construction information, with 400 no task execution. The SIGNATURE_HANDLER parameter is 401 passed to the handler. Two common values are none and 402 printdiff but the handler may define more/less. none 403 means only dump the signature, printdiff means compare 404 the dumped signature with the cached one. 405 -p, --parse-only Quit after parsing the BB recipes. 406 -s, --show-versions Show current and preferred versions of all recipes. 407 -e, --environment Show the global or per-recipe environment complete 408 with information about where variables were 409 set/changed. 410 -g, --graphviz Save dependency tree information for the specified 411 targets in the dot syntax. 412 -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED 413 Assume these dependencies don't exist and are already 414 provided (equivalent to ASSUME_PROVIDED). Useful to 415 make dependency graphs more appealing 416 -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS 417 Show debug logging for the specified logging domains 418 -P, --profile Profile the command and save reports. 419 -u UI, --ui=UI The user interface to use (knotty, ncurses, taskexp or 420 teamcity - default knotty). 421 --token=XMLRPCTOKEN Specify the connection token to be used when 422 connecting to a remote server. 423 --revisions-changed Set the exit code depending on whether upstream 424 floating revisions have changed or not. 425 --server-only Run bitbake without a UI, only starting a server 426 (cooker) process. 427 -B BIND, --bind=BIND The name/address for the bitbake xmlrpc server to bind 428 to. 429 -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT 430 Set timeout to unload bitbake server due to 431 inactivity, set to -1 means no unload, default: 432 Environment variable BB_SERVER_TIMEOUT. 433 --no-setscene Do not run any setscene tasks. sstate will be ignored 434 and everything needed, built. 435 --skip-setscene Skip setscene tasks if they would be executed. Tasks 436 previously restored from sstate will be kept, unlike 437 --no-setscene 438 --setscene-only Only run setscene tasks, don't run any real tasks. 439 --remote-server=REMOTE_SERVER 440 Connect to the specified server. 441 -m, --kill-server Terminate any running bitbake server. 442 --observe-only Connect to a server as an observing-only client. 443 --status-only Check the status of the remote bitbake server. 444 -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG 445 Writes the event log of the build to a bitbake event 446 json file. Use '' (empty string) to assign the name 447 automatically. 448 --runall=RUNALL Run the specified task for any recipe in the taskgraph 449 of the specified target (even if it wouldn't otherwise 450 have run). 451 --runonly=RUNONLY Run only the specified task within the taskgraph of 452 the specified targets (and any task dependencies those 453 tasks may have). 454 455.. _bitbake-examples: 456 457Examples 458-------- 459 460This section presents some examples showing how to use BitBake. 461 462.. _example-executing-a-task-against-a-single-recipe: 463 464Executing a Task Against a Single Recipe 465~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 466 467Executing tasks for a single recipe file is relatively simple. You 468specify the file in question, and BitBake parses it and executes the 469specified task. If you do not specify a task, BitBake executes the 470default task, which is "build". BitBake obeys inter-task dependencies 471when doing so. 472 473The following command runs the build task, which is the default task, on 474the ``foo_1.0.bb`` recipe file:: 475 476 $ bitbake -b foo_1.0.bb 477 478The following command runs the clean task on the ``foo.bb`` recipe file:: 479 480 $ bitbake -b foo.bb -c clean 481 482.. note:: 483 484 The "-b" option explicitly does not handle recipe dependencies. Other 485 than for debugging purposes, it is instead recommended that you use 486 the syntax presented in the next section. 487 488Executing Tasks Against a Set of Recipe Files 489~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 490 491There are a number of additional complexities introduced when one wants 492to manage multiple ``.bb`` files. Clearly there needs to be a way to 493tell BitBake what files are available and, of those, which you want to 494execute. There also needs to be a way for each recipe to express its 495dependencies, both for build-time and runtime. There must be a way for 496you to express recipe preferences when multiple recipes provide the same 497functionality, or when there are multiple versions of a recipe. 498 499The ``bitbake`` command, when not using "--buildfile" or "-b" only 500accepts a "PROVIDES". You cannot provide anything else. By default, a 501recipe file generally "PROVIDES" its "packagename" as shown in the 502following example:: 503 504 $ bitbake foo 505 506This next example "PROVIDES" the 507package name and also uses the "-c" option to tell BitBake to just 508execute the ``do_clean`` task:: 509 510 $ bitbake -c clean foo 511 512Executing a List of Task and Recipe Combinations 513~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 514 515The BitBake command line supports specifying different tasks for 516individual targets when you specify multiple targets. For example, 517suppose you had two targets (or recipes) ``myfirstrecipe`` and 518``mysecondrecipe`` and you needed BitBake to run ``taskA`` for the first 519recipe and ``taskB`` for the second recipe:: 520 521 $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB 522 523Generating Dependency Graphs 524~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 525 526BitBake is able to generate dependency graphs using the ``dot`` syntax. 527You can convert these graphs into images using the ``dot`` tool from 528`Graphviz <http://www.graphviz.org>`__. 529 530When you generate a dependency graph, BitBake writes two files to the 531current working directory: 532 533- ``task-depends.dot``: Shows dependencies between tasks. These 534 dependencies match BitBake's internal task execution list. 535 536- ``pn-buildlist``: Shows a simple list of targets that are to be 537 built. 538 539To stop depending on common depends, use the ``-I`` depend option and 540BitBake omits them from the graph. Leaving this information out can 541produce more readable graphs. This way, you can remove from the graph 542:term:`DEPENDS` from inherited classes such as ``base.bbclass``. 543 544Here are two examples that create dependency graphs. The second example 545omits depends common in OpenEmbedded from the graph:: 546 547 $ bitbake -g foo 548 549 $ bitbake -g -I virtual/kernel -I eglibc foo 550 551Executing a Multiple Configuration Build 552~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 553 554BitBake is able to build multiple images or packages using a single 555command where the different targets require different configurations 556(multiple configuration builds). Each target, in this scenario, is 557referred to as a "multiconfig". 558 559To accomplish a multiple configuration build, you must define each 560target's configuration separately using a parallel configuration file in 561the build directory. The location for these multiconfig configuration 562files is specific. They must reside in the current build directory in a 563sub-directory of ``conf`` named ``multiconfig``. Following is an example 564for two separate targets: 565 566.. image:: figures/bb_multiconfig_files.png 567 :align: center 568 569The reason for this required file hierarchy is because the :term:`BBPATH` 570variable is not constructed until the layers are parsed. Consequently, 571using the configuration file as a pre-configuration file is not possible 572unless it is located in the current working directory. 573 574Minimally, each configuration file must define the machine and the 575temporary directory BitBake uses for the build. Suggested practice 576dictates that you do not overlap the temporary directories used during 577the builds. 578 579Aside from separate configuration files for each target, you must also 580enable BitBake to perform multiple configuration builds. Enabling is 581accomplished by setting the 582:term:`BBMULTICONFIG` variable in the 583``local.conf`` configuration file. As an example, suppose you had 584configuration files for ``target1`` and ``target2`` defined in the build 585directory. The following statement in the ``local.conf`` file both 586enables BitBake to perform multiple configuration builds and specifies 587the two extra multiconfigs:: 588 589 BBMULTICONFIG = "target1 target2" 590 591Once the target configuration files are in place and BitBake has been 592enabled to perform multiple configuration builds, use the following 593command form to start the builds:: 594 595 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ] 596 597Here is an example for two extra multiconfigs: ``target1`` and ``target2``:: 598 599 $ bitbake mc::target mc:target1:target mc:target2:target 600 601.. _bb-enabling-multiple-configuration-build-dependencies: 602 603Enabling Multiple Configuration Build Dependencies 604~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 605 606Sometimes dependencies can exist between targets (multiconfigs) in a 607multiple configuration build. For example, suppose that in order to 608build an image for a particular architecture, the root filesystem of 609another build for a different architecture needs to exist. In other 610words, the image for the first multiconfig depends on the root 611filesystem of the second multiconfig. This dependency is essentially 612that the task in the recipe that builds one multiconfig is dependent on 613the completion of the task in the recipe that builds another 614multiconfig. 615 616To enable dependencies in a multiple configuration build, you must 617declare the dependencies in the recipe using the following statement 618form:: 619 620 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" 621 622To better show how to use this statement, consider an example with two 623multiconfigs: ``target1`` and ``target2``:: 624 625 image_task[mcdepends] = "mc:target1:target2:image2:rootfs_task" 626 627In this example, the 628``from_multiconfig`` is "target1" and the ``to_multiconfig`` is "target2". The 629task on which the image whose recipe contains image_task depends on the 630completion of the rootfs_task used to build out image2, which is 631associated with the "target2" multiconfig. 632 633Once you set up this dependency, you can build the "target1" multiconfig 634using a BitBake command as follows:: 635 636 $ bitbake mc:target1:image1 637 638This command executes all the tasks needed to create ``image1`` for the "target1" 639multiconfig. Because of the dependency, BitBake also executes through 640the ``rootfs_task`` for the "target2" multiconfig build. 641 642Having a recipe depend on the root filesystem of another build might not 643seem that useful. Consider this change to the statement in the image1 644recipe:: 645 646 image_task[mcdepends] = "mc:target1:target2:image2:image_task" 647 648In this case, BitBake must create ``image2`` for the "target2" build since 649the "target1" build depends on it. 650 651Because "target1" and "target2" are enabled for multiple configuration 652builds and have separate configuration files, BitBake places the 653artifacts for each build in the respective temporary build directories. 654
