1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3********************** 4Yocto Project Concepts 5********************** 6 7This chapter provides explanations for Yocto Project concepts that go 8beyond the surface of "how-to" information and reference (or look-up) 9material. Concepts such as components, the :term:`OpenEmbedded Build System` 10workflow, 11cross-development toolchains, shared state cache, and so forth are 12explained. 13 14Yocto Project Components 15======================== 16 17The :term:`BitBake` task executor 18together with various types of configuration files form the 19:term:`OpenEmbedded-Core (OE-Core)`. This section 20overviews these components by describing their use and how they 21interact. 22 23BitBake handles the parsing and execution of the data files. The data 24itself is of various types: 25 26- *Recipes:* Provides details about particular pieces of software. 27 28- *Class Data:* Abstracts common build information (e.g. how to build a 29 Linux kernel). 30 31- *Configuration Data:* Defines machine-specific settings, policy 32 decisions, and so forth. Configuration data acts as the glue to bind 33 everything together. 34 35BitBake knows how to combine multiple data sources together and refers 36to each data source as a layer. For information on layers, see the 37":ref:`dev-manual/layers:understanding and creating layers`" 38section of the Yocto Project Development Tasks Manual. 39 40Here are some brief details on these core components. For 41additional information on how these components interact during a build, 42see the 43":ref:`overview-manual/concepts:openembedded build system concepts`" 44section. 45 46BitBake 47------- 48 49BitBake is the tool at the heart of the :term:`OpenEmbedded Build System` 50and is responsible 51for parsing the :term:`Metadata`, generating 52a list of tasks from it, and then executing those tasks. 53 54This section briefly introduces BitBake. If you want more information on 55BitBake, see the :doc:`BitBake User Manual <bitbake:index>`. 56 57To see a list of the options BitBake supports, use either of the 58following commands:: 59 60 $ bitbake -h 61 $ bitbake --help 62 63The most common usage for BitBake is ``bitbake recipename``, where 64``recipename`` is the name of the recipe you want to build (referred 65to as the "target"). The target often equates to the first part of a 66recipe's filename (e.g. "foo" for a recipe named ``foo_1.3.0-r0.bb``). 67So, to process the ``matchbox-desktop_1.2.3.bb`` recipe file, you might 68type the following:: 69 70 $ bitbake matchbox-desktop 71 72Several different versions of ``matchbox-desktop`` might exist. BitBake chooses 73the one selected by the distribution configuration. You can get more details 74about how BitBake chooses between different target versions and providers in the 75":ref:`bitbake-user-manual/bitbake-user-manual-execution:preferences`" section 76of the BitBake User Manual. 77 78BitBake also tries to execute any dependent tasks first. So for example, 79before building ``matchbox-desktop``, BitBake would build a cross 80compiler and ``glibc`` if they had not already been built. 81 82A useful BitBake option to consider is the ``-k`` or ``--continue`` 83option. This option instructs BitBake to try and continue processing the 84job as long as possible even after encountering an error. When an error 85occurs, the target that failed and those that depend on it cannot be 86remade. However, when you use this option other dependencies can still 87be processed. 88 89Recipes 90------- 91 92Files that have the ``.bb`` suffix are "recipes" files. In general, a 93recipe contains information about a single piece of software. This 94information includes the location from which to download the unaltered 95source, any source patches to be applied to that source (if needed), 96which special configuration options to apply, how to compile the source 97files, and how to package the compiled output. 98 99The term "package" is sometimes used to refer to recipes. However, since 100the word "package" is used for the packaged output from the OpenEmbedded 101build system (i.e. ``.ipk`` or ``.deb`` files), this document avoids 102using the term "package" when referring to recipes. 103 104Classes 105------- 106 107Class files (``.bbclass``) contain information that is useful to share 108between recipes files. An example is the :ref:`ref-classes-autotools` class, 109which contains common settings for any application that is built with 110the :wikipedia:`GNU Autotools <GNU_Autotools>`. 111The ":ref:`ref-manual/classes:Classes`" chapter in the Yocto Project 112Reference Manual provides details about classes and how to use them. 113 114Configurations 115-------------- 116 117The configuration files (``.conf``) define various configuration 118variables that govern the OpenEmbedded build process. These files fall 119into several areas that define machine configuration options, 120distribution configuration options, compiler tuning options, general 121common configuration options, and user configuration options in 122``conf/local.conf``, which is found in the :term:`Build Directory`. 123 124 125Layers 126====== 127 128Layers are repositories that contain related metadata (i.e. sets of 129instructions) that tell the OpenEmbedded build system how to build a 130target. :ref:`overview-manual/yp-intro:the yocto project layer model` 131facilitates collaboration, sharing, customization, and reuse within the 132Yocto Project development environment. Layers logically separate 133information for your project. For example, you can use a layer to hold 134all the configurations for a particular piece of hardware. Isolating 135hardware-specific configurations allows you to share other metadata by 136using a different layer where that metadata might be common across 137several pieces of hardware. 138 139There are many layers working in the Yocto Project development environment. The 140:yocto_home:`Yocto Project Compatible Layer Index </software-overview/layers/>` 141and :oe_layerindex:`OpenEmbedded Layer Index <>` both contain layers from 142which you can use or leverage. 143 144By convention, layers in the Yocto Project follow a specific form. 145Conforming to a known structure allows BitBake to make assumptions 146during builds on where to find types of metadata. You can find 147procedures and learn about tools (i.e. ``bitbake-layers``) for creating 148layers suitable for the Yocto Project in the 149":ref:`dev-manual/layers:understanding and creating layers`" 150section of the Yocto Project Development Tasks Manual. 151 152OpenEmbedded Build System Concepts 153================================== 154 155This section takes a more detailed look inside the build process used by 156the :term:`OpenEmbedded Build System`, 157which is the build 158system specific to the Yocto Project. At the heart of the build system 159is BitBake, the task executor. 160 161The following diagram represents the high-level workflow of a build. The 162remainder of this section expands on the fundamental input, output, 163process, and metadata logical blocks that make up the workflow. 164 165.. image:: figures/YP-flow-diagram.png 166 :width: 100% 167 168In general, the build's workflow consists of several functional areas: 169 170- *User Configuration:* metadata you can use to control the build 171 process. 172 173- *Metadata Layers:* Various layers that provide software, machine, and 174 distro metadata. 175 176- *Source Files:* Upstream releases, local projects, and SCMs. 177 178- *Build System:* Processes under the control of 179 :term:`BitBake`. This block expands 180 on how BitBake fetches source, applies patches, completes 181 compilation, analyzes output for package generation, creates and 182 tests packages, generates images, and generates cross-development 183 tools. 184 185- *Package Feeds:* Directories containing output packages (RPM, DEB or 186 IPK), which are subsequently used in the construction of an image or 187 Software Development Kit (SDK), produced by the build system. These 188 feeds can also be copied and shared using a web server or other means 189 to facilitate extending or updating existing images on devices at 190 runtime if runtime package management is enabled. 191 192- *Images:* Images produced by the workflow. 193 194- *Application Development SDK:* Cross-development tools that are 195 produced along with an image or separately with BitBake. 196 197User Configuration 198------------------ 199 200User configuration helps define the build. Through user configuration, 201you can tell BitBake the target architecture for which you are building 202the image, where to store downloaded source, and other build properties. 203 204The following figure shows an expanded representation of the "User 205Configuration" box of the :ref:`general workflow 206figure <overview-manual/concepts:openembedded build system concepts>`: 207 208.. image:: figures/user-configuration.png 209 :width: 100% 210 211BitBake needs some basic configuration files in order to complete a 212build. These files are ``*.conf`` files. The minimally necessary ones 213reside as example files in the ``build/conf`` directory of the 214:term:`Source Directory`. For simplicity, 215this section refers to the Source Directory as the "Poky Directory." 216 217When you clone the :term:`Poky` Git repository 218or you download and unpack a Yocto Project release, you can set up the 219Source Directory to be named anything you want. For this discussion, the 220cloned repository uses the default name ``poky``. 221 222.. note:: 223 224 The Poky repository is primarily an aggregation of existing 225 repositories. It is not a canonical upstream source. 226 227The ``meta-poky`` layer inside Poky contains a ``conf`` directory that 228has example configuration files. These example files are used as a basis 229for creating actual configuration files when you source 230:ref:`structure-core-script`, which is the 231build environment script. 232 233Sourcing the build environment script creates a :term:`Build Directory` 234if one does not already exist. BitBake uses the :term:`Build Directory` 235for all its work during builds. The Build Directory has a ``conf`` directory 236that contains default versions of your ``local.conf`` and ``bblayers.conf`` 237configuration files. These default configuration files are created only 238if versions do not already exist in the :term:`Build Directory` at the time you 239source the build environment setup script. 240 241Because the Poky repository is fundamentally an aggregation of existing 242repositories, some users might be familiar with running the 243:ref:`structure-core-script` script in the context of separate 244:term:`OpenEmbedded-Core (OE-Core)` and BitBake 245repositories rather than a single Poky repository. This discussion 246assumes the script is executed from within a cloned or unpacked version 247of Poky. 248 249Depending on where the script is sourced, different sub-scripts are 250called to set up the :term:`Build Directory` (Yocto or OpenEmbedded). 251Specifically, the script ``scripts/oe-setup-builddir`` inside the poky 252directory sets up the :term:`Build Directory` and seeds the directory (if 253necessary) with configuration files appropriate for the Yocto Project 254development environment. 255 256.. note:: 257 258 The 259 scripts/oe-setup-builddir 260 script uses the 261 ``$TEMPLATECONF`` 262 variable to determine which sample configuration files to locate. 263 264The ``local.conf`` file provides many basic variables that define a 265build environment. Here is a list of a few. To see the default 266configurations in a ``local.conf`` file created by the build environment 267script, see the 268:yocto_git:`local.conf.sample </poky/tree/meta-poky/conf/templates/default/local.conf.sample>` 269in the ``meta-poky`` layer: 270 271- *Target Machine Selection:* Controlled by the 272 :term:`MACHINE` variable. 273 274- *Download Directory:* Controlled by the 275 :term:`DL_DIR` variable. 276 277- *Shared State Directory:* Controlled by the 278 :term:`SSTATE_DIR` variable. 279 280- *Build Output:* Controlled by the 281 :term:`TMPDIR` variable. 282 283- *Distribution Policy:* Controlled by the 284 :term:`DISTRO` variable. 285 286- *Packaging Format:* Controlled by the 287 :term:`PACKAGE_CLASSES` 288 variable. 289 290- *SDK Target Architecture:* Controlled by the 291 :term:`SDKMACHINE` variable. 292 293- *Extra Image Packages:* Controlled by the 294 :term:`EXTRA_IMAGE_FEATURES` 295 variable. 296 297.. note:: 298 299 Configurations set in the ``conf/local.conf`` file can also be set 300 in the ``conf/site.conf`` and ``conf/auto.conf`` configuration files. 301 302The ``bblayers.conf`` file tells BitBake what layers you want considered 303during the build. By default, the layers listed in this file include 304layers minimally needed by the build system. However, you must manually 305add any custom layers you have created. You can find more information on 306working with the ``bblayers.conf`` file in the 307":ref:`dev-manual/layers:enabling your layer`" 308section in the Yocto Project Development Tasks Manual. 309 310The files ``site.conf`` and ``auto.conf`` are not created by the 311environment initialization script. If you want the ``site.conf`` file, 312you need to create it yourself. The ``auto.conf`` file is typically 313created by an autobuilder: 314 315- *site.conf:* You can use the ``conf/site.conf`` configuration 316 file to configure multiple build directories. For example, suppose 317 you had several build environments and they shared some common 318 features. You can set these default build properties here. A good 319 example is perhaps the packaging format to use through the 320 :term:`PACKAGE_CLASSES` variable. 321 322- *auto.conf:* The file is usually created and written to by an 323 autobuilder. The settings put into the file are typically the same as 324 you would find in the ``conf/local.conf`` or the ``conf/site.conf`` 325 files. 326 327You can edit all configuration files to further define any particular 328build environment. This process is represented by the "User 329Configuration Edits" box in the figure. 330 331When you launch your build with the ``bitbake target`` command, BitBake 332sorts out the configurations to ultimately define your build 333environment. It is important to understand that the 334:term:`OpenEmbedded Build System` reads the 335configuration files in a specific order: ``site.conf``, ``auto.conf``, 336and ``local.conf``. And, the build system applies the normal assignment 337statement rules as described in the 338":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter 339of the BitBake User Manual. Because the files are parsed in a specific 340order, variable assignments for the same variable could be affected. For 341example, if the ``auto.conf`` file and the ``local.conf`` set variable1 342to different values, because the build system parses ``local.conf`` 343after ``auto.conf``, variable1 is assigned the value from the 344``local.conf`` file. 345 346Metadata, Machine Configuration, and Policy Configuration 347--------------------------------------------------------- 348 349The previous section described the user configurations that define 350BitBake's global behavior. This section takes a closer look at the 351layers the build system uses to further control the build. These layers 352provide Metadata for the software, machine, and policies. 353 354In general, there are three types of layer input. You can see them below 355the "User Configuration" box in the `general workflow 356figure <overview-manual/concepts:openembedded build system concepts>`: 357 358- *Metadata (.bb + Patches):* Software layers containing 359 user-supplied recipe files, patches, and append files. A good example 360 of a software layer might be the :oe_layer:`meta-qt5 layer </meta-qt5>` 361 from the :oe_layerindex:`OpenEmbedded Layer Index <>`. This layer is for 362 version 5.0 of the popular `Qt <https://wiki.qt.io/About_Qt>`__ 363 cross-platform application development framework for desktop, embedded and 364 mobile. 365 366- *Machine BSP Configuration:* Board Support Package (BSP) layers (i.e. 367 "BSP Layer" in the following figure) providing machine-specific 368 configurations. This type of information is specific to a particular 369 target architecture. A good example of a BSP layer from the 370 :ref:`overview-manual/yp-intro:reference distribution (poky)` is the 371 :yocto_git:`meta-yocto-bsp </poky/tree/meta-yocto-bsp>` 372 layer. 373 374- *Policy Configuration:* Distribution Layers (i.e. "Distro Layer" in 375 the following figure) providing top-level or general policies for the 376 images or SDKs being built for a particular distribution. For 377 example, in the Poky Reference Distribution the distro layer is the 378 :yocto_git:`meta-poky </poky/tree/meta-poky>` 379 layer. Within the distro layer is a ``conf/distro`` directory that 380 contains distro configuration files (e.g. 381 :yocto_git:`poky.conf </poky/tree/meta-poky/conf/distro/poky.conf>` 382 that contain many policy configurations for the Poky distribution. 383 384The following figure shows an expanded representation of these three 385layers from the :ref:`general workflow figure 386<overview-manual/concepts:openembedded build system concepts>`: 387 388.. image:: figures/layer-input.png 389 :align: center 390 :width: 70% 391 392In general, all layers have a similar structure. They all contain a 393licensing file (e.g. ``COPYING.MIT``) if the layer is to be distributed, 394a ``README`` file as good practice and especially if the layer is to be 395distributed, a configuration directory, and recipe directories. You can 396learn about the general structure for layers used with the Yocto Project 397in the 398":ref:`dev-manual/layers:creating your own layer`" 399section in the 400Yocto Project Development Tasks Manual. For a general discussion on 401layers and the many layers from which you can draw, see the 402":ref:`overview-manual/concepts:layers`" and 403":ref:`overview-manual/yp-intro:the yocto project layer model`" sections both 404earlier in this manual. 405 406If you explored the previous links, you discovered some areas where many 407layers that work with the Yocto Project exist. The :yocto_git:`Source 408Repositories <>` also shows layers categorized under "Yocto Metadata Layers." 409 410.. note:: 411 412 There are layers in the Yocto Project Source Repositories that cannot be 413 found in the OpenEmbedded Layer Index. Such layers are either 414 deprecated or experimental in nature. 415 416BitBake uses the ``conf/bblayers.conf`` file, which is part of the user 417configuration, to find what layers it should be using as part of the 418build. 419 420Distro Layer 421~~~~~~~~~~~~ 422 423The distribution layer provides policy configurations for your 424distribution. Best practices dictate that you isolate these types of 425configurations into their own layer. Settings you provide in 426``conf/distro/distro.conf`` override similar settings that BitBake finds 427in your ``conf/local.conf`` file in the :term:`Build Directory`. 428 429The following list provides some explanation and references for what you 430typically find in the distribution layer: 431 432- *classes:* Class files (``.bbclass``) hold common functionality that 433 can be shared among recipes in the distribution. When your recipes 434 inherit a class, they take on the settings and functions for that 435 class. You can read more about class files in the 436 ":ref:`ref-manual/classes:Classes`" chapter of the Yocto 437 Reference Manual. 438 439- *conf:* This area holds configuration files for the layer 440 (``conf/layer.conf``), the distribution 441 (``conf/distro/distro.conf``), and any distribution-wide include 442 files. 443 444- *recipes-*:* Recipes and append files that affect common 445 functionality across the distribution. This area could include 446 recipes and append files to add distribution-specific configuration, 447 initialization scripts, custom image recipes, and so forth. Examples 448 of ``recipes-*`` directories are ``recipes-core`` and 449 ``recipes-extra``. Hierarchy and contents within a ``recipes-*`` 450 directory can vary. Generally, these directories contain recipe files 451 (``*.bb``), recipe append files (``*.bbappend``), directories that 452 are distro-specific for configuration files, and so forth. 453 454BSP Layer 455~~~~~~~~~ 456 457The BSP Layer provides machine configurations that target specific 458hardware. Everything in this layer is specific to the machine for which 459you are building the image or the SDK. A common structure or form is 460defined for BSP layers. You can learn more about this structure in the 461:doc:`/bsp-guide/index`. 462 463.. note:: 464 465 In order for a BSP layer to be considered compliant with the Yocto 466 Project, it must meet some structural requirements. 467 468The BSP Layer's configuration directory contains configuration files for 469the machine (``conf/machine/machine.conf``) and, of course, the layer 470(``conf/layer.conf``). 471 472The remainder of the layer is dedicated to specific recipes by function: 473``recipes-bsp``, ``recipes-core``, ``recipes-graphics``, 474``recipes-kernel``, and so forth. There can be metadata for multiple 475formfactors, graphics support systems, and so forth. 476 477.. note:: 478 479 While the figure shows several 480 recipes-\* 481 directories, not all these directories appear in all BSP layers. 482 483Software Layer 484~~~~~~~~~~~~~~ 485 486The software layer provides the Metadata for additional software 487packages used during the build. This layer does not include Metadata 488that is specific to the distribution or the machine, which are found in 489their respective layers. 490 491This layer contains any recipes, append files, and patches, that your 492project needs. 493 494Sources 495------- 496 497In order for the OpenEmbedded build system to create an image or any 498target, it must be able to access source files. The :ref:`general workflow 499figure <overview-manual/concepts:openembedded build system concepts>` 500represents source files using the "Upstream Project Releases", "Local 501Projects", and "SCMs (optional)" boxes. The figure represents mirrors, 502which also play a role in locating source files, with the "Source 503Materials" box. 504 505The method by which source files are ultimately organized is a function 506of the project. For example, for released software, projects tend to use 507tarballs or other archived files that can capture the state of a release 508guaranteeing that it is statically represented. On the other hand, for a 509project that is more dynamic or experimental in nature, a project might 510keep source files in a repository controlled by a Source Control Manager 511(SCM) such as Git. Pulling source from a repository allows you to 512control the point in the repository (the revision) from which you want 513to build software. A combination of the two is also possible. 514 515BitBake uses the :term:`SRC_URI` 516variable to point to source files regardless of their location. Each 517recipe must have a :term:`SRC_URI` variable that points to the source. 518 519Another area that plays a significant role in where source files come 520from is pointed to by the 521:term:`DL_DIR` variable. This area is 522a cache that can hold previously downloaded source. You can also 523instruct the OpenEmbedded build system to create tarballs from Git 524repositories, which is not the default behavior, and store them in the 525:term:`DL_DIR` by using the 526:term:`BB_GENERATE_MIRROR_TARBALLS` 527variable. 528 529Judicious use of a :term:`DL_DIR` directory can save the build system a trip 530across the Internet when looking for files. A good method for using a download 531directory is to have :term:`DL_DIR` point to an area outside of your 532:term:`Build Directory`. Doing so allows you to safely delete the 533:term:`Build Directory` if needed without fear of removing any downloaded 534source file. 535 536The remainder of this section provides a deeper look into the source 537files and the mirrors. Here is a more detailed look at the source file 538area of the :ref:`general workflow figure <overview-manual/concepts:openembedded build system concepts>`: 539 540.. image:: figures/source-input.png 541 :align: center 542 :width: 70% 543 544Upstream Project Releases 545~~~~~~~~~~~~~~~~~~~~~~~~~ 546 547Upstream project releases exist anywhere in the form of an archived file 548(e.g. tarball or zip file). These files correspond to individual 549recipes. For example, the figure uses specific releases each for 550BusyBox, Qt, and Dbus. An archive file can be for any released product 551that can be built using a recipe. 552 553Local Projects 554~~~~~~~~~~~~~~ 555 556Local projects are custom bits of software the user provides. These bits 557reside somewhere local to a project --- perhaps a directory into which the 558user checks in items (e.g. a local directory containing a development 559source tree used by the group). 560 561The canonical method through which to include a local project is to use the 562:ref:`ref-classes-externalsrc` class to include that local project. You use 563either the ``local.conf`` or a recipe's append file to override or set the 564recipe to point to the local directory on your disk to pull in the whole 565source tree. 566 567Source Control Managers (Optional) 568~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 569 570Another place from which the build system can get source files is with 571:ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` employing 572various Source Control Managers (SCMs) such as Git or Subversion. In such 573cases, a repository is cloned or checked out. The :ref:`ref-tasks-fetch` task 574inside BitBake uses the :term:`SRC_URI` variable and the argument's prefix to 575determine the correct fetcher module. 576 577.. note:: 578 579 For information on how to have the OpenEmbedded build system generate 580 tarballs for Git repositories and place them in the :term:`DL_DIR` 581 directory, see the :term:`BB_GENERATE_MIRROR_TARBALLS` 582 variable in the Yocto Project Reference Manual. 583 584When fetching a repository, BitBake uses the 585:term:`SRCREV` variable to determine 586the specific revision from which to build. 587 588Source Mirror(s) 589~~~~~~~~~~~~~~~~ 590 591There are two kinds of mirrors: pre-mirrors and regular mirrors. The 592:term:`PREMIRRORS` and 593:term:`MIRRORS` variables point to 594these, respectively. BitBake checks pre-mirrors before looking upstream 595for any source files. Pre-mirrors are appropriate when you have a shared 596directory that is not a directory defined by the 597:term:`DL_DIR` variable. A Pre-mirror 598typically points to a shared directory that is local to your 599organization. 600 601Regular mirrors can be any site across the Internet that is used as an 602alternative location for source code should the primary site not be 603functioning for some reason or another. 604 605Package Feeds 606------------- 607 608When the OpenEmbedded build system generates an image or an SDK, it gets 609the packages from a package feed area located in the 610:term:`Build Directory`. The :ref:`general workflow figure 611<overview-manual/concepts:openembedded build system concepts>` 612shows this package feeds area in the upper-right corner. 613 614This section looks a little closer into the package feeds area used by 615the build system. Here is a more detailed look at the area: 616 617.. image:: figures/package-feeds.png 618 :width: 100% 619 620Package feeds are an intermediary step in the build process. The 621OpenEmbedded build system provides classes to generate different package 622types, and you specify which classes to enable through the 623:term:`PACKAGE_CLASSES` 624variable. Before placing the packages into package feeds, the build 625process validates them with generated output quality assurance checks 626through the :ref:`ref-classes-insane` class. 627 628The package feed area resides in the :term:`Build Directory`. The directory the 629build system uses to temporarily store packages is determined by a 630combination of variables and the particular package manager in use. See 631the "Package Feeds" box in the illustration and note the information to 632the right of that area. In particular, the following defines where 633package files are kept: 634 635- :term:`DEPLOY_DIR`: Defined as ``tmp/deploy`` in the :term:`Build Directory`. 636 637- ``DEPLOY_DIR_*``: Depending on the package manager used, the package 638 type sub-folder. Given RPM, IPK, or DEB packaging and tarball 639 creation, the 640 :term:`DEPLOY_DIR_RPM`, 641 :term:`DEPLOY_DIR_IPK`, or 642 :term:`DEPLOY_DIR_DEB` 643 variables are used, respectively. 644 645- :term:`PACKAGE_ARCH`: Defines 646 architecture-specific sub-folders. For example, packages could be 647 available for the i586 or qemux86 architectures. 648 649BitBake uses the 650:ref:`do_package_write_* <ref-tasks-package_write_deb>` 651tasks to generate packages and place them into the package holding area 652(e.g. ``do_package_write_ipk`` for IPK packages). See the 653":ref:`ref-tasks-package_write_deb`", 654":ref:`ref-tasks-package_write_ipk`", 655and 656":ref:`ref-tasks-package_write_rpm`" 657sections in the Yocto Project Reference Manual for additional 658information. As an example, consider a scenario where an IPK packaging 659manager is being used and there is package architecture support for both 660i586 and qemux86. Packages for the i586 architecture are placed in 661``build/tmp/deploy/ipk/i586``, while packages for the qemux86 662architecture are placed in ``build/tmp/deploy/ipk/qemux86``. 663 664BitBake Tool 665------------ 666 667The OpenEmbedded build system uses 668:term:`BitBake` to produce images and 669Software Development Kits (SDKs). You can see from the :ref:`general workflow 670figure <overview-manual/concepts:openembedded build system concepts>`, 671the BitBake area consists of several functional areas. This section takes a 672closer look at each of those areas. 673 674.. note:: 675 676 Documentation for the BitBake tool is available separately. See the 677 :doc:`BitBake User Manual <bitbake:index>` 678 for reference material on BitBake. 679 680Source Fetching 681~~~~~~~~~~~~~~~ 682 683The first stages of building a recipe are to fetch and unpack the source 684code: 685 686.. image:: figures/source-fetching.png 687 :width: 100% 688 689The :ref:`ref-tasks-fetch` and :ref:`ref-tasks-unpack` tasks fetch 690the source files and unpack them into the :term:`Build Directory`. 691 692.. note:: 693 694 For every local file (e.g. ``file://``) that is part of a recipe's 695 :term:`SRC_URI` statement, the OpenEmbedded build system takes a 696 checksum of the file for the recipe and inserts the checksum into 697 the signature for the :ref:`ref-tasks-fetch` task. If any local 698 file has been modified, the :ref:`ref-tasks-fetch` task and all 699 tasks that depend on it are re-executed. 700 701By default, everything is accomplished in the :term:`Build Directory`, which has 702a defined structure. For additional general information on the 703:term:`Build Directory`, see the ":ref:`structure-core-build`" section in 704the Yocto Project Reference Manual. 705 706Each recipe has an area in the :term:`Build Directory` where the unpacked 707source code resides. The :term:`S` variable points to this area for a recipe's 708unpacked source code. The name of that directory for any given recipe is 709defined from several different variables. The preceding figure and the 710following list describe the :term:`Build Directory`'s hierarchy: 711 712- :term:`TMPDIR`: The base directory 713 where the OpenEmbedded build system performs all its work during the 714 build. The default base directory is the ``tmp`` directory. 715 716- :term:`PACKAGE_ARCH`: The 717 architecture of the built package or packages. Depending on the 718 eventual destination of the package or packages (i.e. machine 719 architecture, :term:`Build Host`, SDK, or 720 specific machine), :term:`PACKAGE_ARCH` varies. See the variable's 721 description for details. 722 723- :term:`TARGET_OS`: The operating 724 system of the target device. A typical value would be "linux" (e.g. 725 "qemux86-poky-linux"). 726 727- :term:`PN`: The name of the recipe used 728 to build the package. This variable can have multiple meanings. 729 However, when used in the context of input files, :term:`PN` represents 730 the name of the recipe. 731 732- :term:`WORKDIR`: The location 733 where the OpenEmbedded build system builds a recipe (i.e. does the 734 work to create the package). 735 736 - :term:`PV`: The version of the 737 recipe used to build the package. 738 739 - :term:`PR`: The revision of the 740 recipe used to build the package. 741 742- :term:`S`: Contains the unpacked source 743 files for a given recipe. 744 745 - :term:`BPN`: The name of the recipe 746 used to build the package. The :term:`BPN` variable is a version of 747 the :term:`PN` variable but with common prefixes and suffixes removed. 748 749 - :term:`PV`: The version of the 750 recipe used to build the package. 751 752.. note:: 753 754 In the previous figure, notice that there are two sample hierarchies: 755 one based on package architecture (i.e. :term:`PACKAGE_ARCH`) 756 and one based on a machine (i.e. :term:`MACHINE`). 757 The underlying structures are identical. The differentiator being 758 what the OpenEmbedded build system is using as a build target (e.g. 759 general architecture, a build host, an SDK, or a specific machine). 760 761Patching 762~~~~~~~~ 763 764Once source code is fetched and unpacked, BitBake locates patch files 765and applies them to the source files: 766 767.. image:: figures/patching.png 768 :width: 100% 769 770The :ref:`ref-tasks-patch` task uses a 771recipe's :term:`SRC_URI` statements 772and the :term:`FILESPATH` variable 773to locate applicable patch files. 774 775Default processing for patch files assumes the files have either 776``*.patch`` or ``*.diff`` file types. You can use :term:`SRC_URI` parameters 777to change the way the build system recognizes patch files. See the 778:ref:`ref-tasks-patch` task for more 779information. 780 781BitBake finds and applies multiple patches for a single recipe in the 782order in which it locates the patches. The :term:`FILESPATH` variable 783defines the default set of directories that the build system uses to 784search for patch files. Once found, patches are applied to the recipe's 785source files, which are located in the 786:term:`S` directory. 787 788For more information on how the source directories are created, see the 789":ref:`overview-manual/concepts:source fetching`" section. For 790more information on how to create patches and how the build system 791processes patches, see the 792":ref:`dev-manual/new-recipe:patching code`" 793section in the 794Yocto Project Development Tasks Manual. You can also see the 795":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" 796section in the Yocto Project Application Development and the Extensible 797Software Development Kit (SDK) manual and the 798":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`" 799section in the Yocto Project Linux Kernel Development Manual. 800 801Configuration, Compilation, and Staging 802~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 803 804After source code is patched, BitBake executes tasks that configure and 805compile the source code. Once compilation occurs, the files are copied 806to a holding area (staged) in preparation for packaging: 807 808.. image:: figures/configuration-compile-autoreconf.png 809 :width: 100% 810 811This step in the build process consists of the following tasks: 812 813- :ref:`ref-tasks-prepare_recipe_sysroot`: 814 This task sets up the two sysroots in 815 ``${``\ :term:`WORKDIR`\ ``}`` 816 (i.e. ``recipe-sysroot`` and ``recipe-sysroot-native``) so that 817 during the packaging phase the sysroots can contain the contents of 818 the 819 :ref:`ref-tasks-populate_sysroot` 820 tasks of the recipes on which the recipe containing the tasks 821 depends. A sysroot exists for both the target and for the native 822 binaries, which run on the host system. 823 824- *do_configure*: This task configures the source by enabling and 825 disabling any build-time and configuration options for the software 826 being built. Configurations can come from the recipe itself as well 827 as from an inherited class. Additionally, the software itself might 828 configure itself depending on the target for which it is being built. 829 830 The configurations handled by the 831 :ref:`ref-tasks-configure` task 832 are specific to configurations for the source code being built by the 833 recipe. 834 835 If you are using the :ref:`ref-classes-autotools` class, 836 you can add additional configuration options by using the 837 :term:`EXTRA_OECONF` or 838 :term:`PACKAGECONFIG_CONFARGS` 839 variables. For information on how this variable works within that 840 class, see the :ref:`ref-classes-autotools` class 841 :yocto_git:`here </poky/tree/meta/classes-recipe/autotools.bbclass>`. 842 843- *do_compile*: Once a configuration task has been satisfied, 844 BitBake compiles the source using the 845 :ref:`ref-tasks-compile` task. 846 Compilation occurs in the directory pointed to by the 847 :term:`B` variable. Realize that the 848 :term:`B` directory is, by default, the same as the 849 :term:`S` directory. 850 851- *do_install*: After compilation completes, BitBake executes the 852 :ref:`ref-tasks-install` task. 853 This task copies files from the :term:`B` directory and places them in a 854 holding area pointed to by the :term:`D` 855 variable. Packaging occurs later using files from this holding 856 directory. 857 858Package Splitting 859~~~~~~~~~~~~~~~~~ 860 861After source code is configured, compiled, and staged, the build system 862analyzes the results and splits the output into packages: 863 864.. image:: figures/analysis-for-package-splitting.png 865 :width: 100% 866 867The :ref:`ref-tasks-package` and 868:ref:`ref-tasks-packagedata` 869tasks combine to analyze the files found in the 870:term:`D` directory and split them into 871subsets based on available packages and files. Analysis involves the 872following as well as other items: splitting out debugging symbols, 873looking at shared library dependencies between packages, and looking at 874package relationships. 875 876The :ref:`ref-tasks-packagedata` task creates package metadata based on the 877analysis such that the build system can generate the final packages. The 878:ref:`ref-tasks-populate_sysroot` 879task stages (copies) a subset of the files installed by the 880:ref:`ref-tasks-install` task into 881the appropriate sysroot. Working, staged, and intermediate results of 882the analysis and package splitting process use several areas: 883 884- :term:`PKGD`: The destination 885 directory (i.e. ``package``) for packages before they are split into 886 individual packages. 887 888- :term:`PKGDESTWORK`: A 889 temporary work area (i.e. ``pkgdata``) used by the :ref:`ref-tasks-package` 890 task to save package metadata. 891 892- :term:`PKGDEST`: The parent 893 directory (i.e. ``packages-split``) for packages after they have been 894 split. 895 896- :term:`PKGDATA_DIR`: A shared, 897 global-state directory that holds packaging metadata generated during 898 the packaging process. The packaging process copies metadata from 899 :term:`PKGDESTWORK` to the :term:`PKGDATA_DIR` area where it becomes globally 900 available. 901 902- :term:`STAGING_DIR_HOST`: 903 The path for the sysroot for the system on which a component is built 904 to run (i.e. ``recipe-sysroot``). 905 906- :term:`STAGING_DIR_NATIVE`: 907 The path for the sysroot used when building components for the build 908 host (i.e. ``recipe-sysroot-native``). 909 910- :term:`STAGING_DIR_TARGET`: 911 The path for the sysroot used when a component that is built to 912 execute on a system and it generates code for yet another machine 913 (e.g. :ref:`ref-classes-cross-canadian` recipes). 914 915The :term:`FILES` variable defines the 916files that go into each package in 917:term:`PACKAGES`. If you want 918details on how this is accomplished, you can look at 919:yocto_git:`package.bbclass </poky/tree/meta/classes-global/package.bbclass>`. 920 921Depending on the type of packages being created (RPM, DEB, or IPK), the 922:ref:`do_package_write_* <ref-tasks-package_write_deb>` 923task creates the actual packages and places them in the Package Feed 924area, which is ``${TMPDIR}/deploy``. You can see the 925":ref:`overview-manual/concepts:package feeds`" section for more detail on 926that part of the build process. 927 928.. note:: 929 930 Support for creating feeds directly from the ``deploy/*`` 931 directories does not exist. Creating such feeds usually requires some 932 kind of feed maintenance mechanism that would upload the new packages 933 into an official package feed (e.g. the Ångström distribution). This 934 functionality is highly distribution-specific and thus is not 935 provided out of the box. 936 937Image Generation 938~~~~~~~~~~~~~~~~ 939 940Once packages are split and stored in the Package Feeds area, the build 941system uses BitBake to generate the root filesystem image: 942 943.. image:: figures/image-generation.png 944 :width: 100% 945 946The image generation process consists of several stages and depends on 947several tasks and variables. The 948:ref:`ref-tasks-rootfs` task creates 949the root filesystem (file and directory structure) for an image. This 950task uses several key variables to help create the list of packages to 951actually install: 952 953- :term:`IMAGE_INSTALL`: Lists 954 out the base set of packages from which to install from the Package 955 Feeds area. 956 957- :term:`PACKAGE_EXCLUDE`: 958 Specifies packages that should not be installed into the image. 959 960- :term:`IMAGE_FEATURES`: 961 Specifies features to include in the image. Most of these features 962 map to additional packages for installation. 963 964- :term:`PACKAGE_CLASSES`: 965 Specifies the package backend (e.g. RPM, DEB, or IPK) to use and 966 consequently helps determine where to locate packages within the 967 Package Feeds area. 968 969- :term:`IMAGE_LINGUAS`: 970 Determines the language(s) for which additional language support 971 packages are installed. 972 973- :term:`PACKAGE_INSTALL`: 974 The final list of packages passed to the package manager for 975 installation into the image. 976 977With :term:`IMAGE_ROOTFS` 978pointing to the location of the filesystem under construction and the 979:term:`PACKAGE_INSTALL` variable providing the final list of packages to 980install, the root file system is created. 981 982Package installation is under control of the package manager (e.g. 983dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package 984management is enabled for the target. At the end of the process, if 985package management is not enabled for the target, the package manager's 986data files are deleted from the root filesystem. As part of the final 987stage of package installation, post installation scripts that are part 988of the packages are run. Any scripts that fail to run on the build host 989are run on the target when the target system is first booted. If you are 990using a 991:ref:`read-only root filesystem <dev-manual/read-only-rootfs:creating a read-only root filesystem>`, 992all the post installation scripts must succeed on the build host during 993the package installation phase since the root filesystem on the target 994is read-only. 995 996The final stages of the :ref:`ref-tasks-rootfs` task handle post processing. Post 997processing includes creation of a manifest file and optimizations. 998 999The manifest file (``.manifest``) resides in the same directory as the root 1000filesystem image. This file lists out, line-by-line, the installed packages. 1001The manifest file is useful for the :ref:`ref-classes-testimage` class, 1002for example, to determine whether or not to run specific tests. See the 1003:term:`IMAGE_MANIFEST` variable for additional information. 1004 1005Optimizing processes that are run across the image include ``mklibs`` 1006and any other post-processing commands as defined by the 1007:term:`ROOTFS_POSTPROCESS_COMMAND` 1008variable. The ``mklibs`` process optimizes the size of the libraries. 1009 1010After the root filesystem is built, processing begins on the image 1011through the :ref:`ref-tasks-image` 1012task. The build system runs any pre-processing commands as defined by 1013the 1014:term:`IMAGE_PREPROCESS_COMMAND` 1015variable. This variable specifies a list of functions to call before the 1016build system creates the final image output files. 1017 1018The build system dynamically creates :ref:`do_image_* <ref-tasks-image>` tasks as needed, 1019based on the image types specified in the 1020:term:`IMAGE_FSTYPES` variable. 1021The process turns everything into an image file or a set of image files 1022and can compress the root filesystem image to reduce the overall size of 1023the image. The formats used for the root filesystem depend on the 1024:term:`IMAGE_FSTYPES` variable. Compression depends on whether the formats 1025support compression. 1026 1027As an example, a dynamically created task when creating a particular 1028image type would take the following form:: 1029 1030 do_image_type 1031 1032So, if the type 1033as specified by the :term:`IMAGE_FSTYPES` were ``ext4``, the dynamically 1034generated task would be as follows:: 1035 1036 do_image_ext4 1037 1038The final task involved in image creation is the 1039:ref:`do_image_complete <ref-tasks-image-complete>` 1040task. This task completes the image by applying any image post 1041processing as defined through the 1042:term:`IMAGE_POSTPROCESS_COMMAND` 1043variable. The variable specifies a list of functions to call once the 1044build system has created the final image output files. 1045 1046.. note:: 1047 1048 The entire image generation process is run under 1049 Pseudo. Running under Pseudo ensures that the files in the root filesystem 1050 have correct ownership. 1051 1052SDK Generation 1053~~~~~~~~~~~~~~ 1054 1055The OpenEmbedded build system uses BitBake to generate the Software 1056Development Kit (SDK) installer scripts for both the standard SDK and 1057the extensible SDK (eSDK): 1058 1059.. image:: figures/sdk-generation.png 1060 :width: 100% 1061 1062.. note:: 1063 1064 For more information on the cross-development toolchain generation, 1065 see the ":ref:`overview-manual/concepts:cross-development toolchain generation`" 1066 section. For information on advantages gained when building a 1067 cross-development toolchain using the :ref:`ref-tasks-populate_sdk` task, see the 1068 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" section in 1069 the Yocto Project Application Development and the Extensible Software 1070 Development Kit (eSDK) manual. 1071 1072Like image generation, the SDK script process consists of several stages 1073and depends on many variables. The 1074:ref:`ref-tasks-populate_sdk` 1075and 1076:ref:`ref-tasks-populate_sdk_ext` 1077tasks use these key variables to help create the list of packages to 1078actually install. For information on the variables listed in the figure, 1079see the ":ref:`overview-manual/concepts:application development sdk`" 1080section. 1081 1082The :ref:`ref-tasks-populate_sdk` task helps create the standard SDK and handles 1083two parts: a target part and a host part. The target part is the part 1084built for the target hardware and includes libraries and headers. The 1085host part is the part of the SDK that runs on the 1086:term:`SDKMACHINE`. 1087 1088The :ref:`ref-tasks-populate_sdk_ext` task helps create the extensible SDK and 1089handles host and target parts differently than its counter part does for 1090the standard SDK. For the extensible SDK, the task encapsulates the 1091build system, which includes everything needed (host and target) for the 1092SDK. 1093 1094Regardless of the type of SDK being constructed, the tasks perform some 1095cleanup after which a cross-development environment setup script and any 1096needed configuration files are created. The final output is the 1097Cross-development toolchain installation script (``.sh`` file), which 1098includes the environment setup script. 1099 1100Stamp Files and the Rerunning of Tasks 1101~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1102 1103For each task that completes successfully, BitBake writes a stamp file 1104into the :term:`STAMPS_DIR` 1105directory. The beginning of the stamp file's filename is determined by 1106the :term:`STAMP` variable, and the end 1107of the name consists of the task's name and current :ref:`input 1108checksum <overview-manual/concepts:checksums (signatures)>`. 1109 1110.. note:: 1111 1112 This naming scheme assumes that :term:`BB_SIGNATURE_HANDLER` 1113 is "OEBasicHash", which is almost always the case in current 1114 OpenEmbedded. 1115 1116To determine if a task needs to be rerun, BitBake checks if a stamp file 1117with a matching input checksum exists for the task. In this case, 1118the task's output is assumed to exist and still be valid. Otherwise, 1119the task is rerun. 1120 1121.. note:: 1122 1123 The stamp mechanism is more general than the shared state (sstate) 1124 cache mechanism described in the 1125 ":ref:`overview-manual/concepts:setscene tasks and shared state`" section. 1126 BitBake avoids rerunning any task that has a valid stamp file, not just 1127 tasks that can be accelerated through the sstate cache. 1128 1129 However, you should realize that stamp files only serve as a marker 1130 that some work has been done and that these files do not record task 1131 output. The actual task output would usually be somewhere in 1132 :term:`TMPDIR` (e.g. in some 1133 recipe's :term:`WORKDIR`.) What 1134 the sstate cache mechanism adds is a way to cache task output that 1135 can then be shared between build machines. 1136 1137Since :term:`STAMPS_DIR` is usually a subdirectory of :term:`TMPDIR`, removing 1138:term:`TMPDIR` will also remove :term:`STAMPS_DIR`, which means tasks will 1139properly be rerun to repopulate :term:`TMPDIR`. 1140 1141If you want some task to always be considered "out of date", you can 1142mark it with the :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>` 1143varflag. If some other task depends on such a task, then that task will 1144also always be considered out of date, which might not be what you want. 1145 1146For details on how to view information about a task's signature, see the 1147":ref:`dev-manual/debugging:viewing task variable dependencies`" 1148section in the Yocto Project Development Tasks Manual. 1149 1150Setscene Tasks and Shared State 1151~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1152 1153The description of tasks so far assumes that BitBake needs to build 1154everything and no available prebuilt objects exist. BitBake does support 1155skipping tasks if prebuilt objects are available. These objects are 1156usually made available in the form of a shared state (sstate) cache. 1157 1158.. note:: 1159 1160 For information on variables affecting sstate, see the 1161 :term:`SSTATE_DIR` 1162 and 1163 :term:`SSTATE_MIRRORS` 1164 variables. 1165 1166The idea of a setscene task (i.e ``do_taskname_setscene``) is a 1167version of the task where instead of building something, BitBake can 1168skip to the end result and simply place a set of files into specific 1169locations as needed. In some cases, it makes sense to have a setscene 1170task variant (e.g. generating package files in the 1171:ref:`do_package_write_* <ref-tasks-package_write_deb>` 1172task). In other cases, it does not make sense (e.g. a 1173:ref:`ref-tasks-patch` task or a 1174:ref:`ref-tasks-unpack` task) since 1175the work involved would be equal to or greater than the underlying task. 1176 1177In the build system, the common tasks that have setscene variants are 1178:ref:`ref-tasks-package`, 1179:ref:`do_package_write_* <ref-tasks-package_write_deb>`, 1180:ref:`ref-tasks-deploy`, 1181:ref:`ref-tasks-packagedata`, and 1182:ref:`ref-tasks-populate_sysroot`. 1183Notice that these tasks represent most of the tasks whose output is an 1184end result. 1185 1186The build system has knowledge of the relationship between these tasks 1187and other preceding tasks. For example, if BitBake runs 1188``do_populate_sysroot_setscene`` for something, it does not make sense 1189to run any of the :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`, 1190:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, and :ref:`ref-tasks-install` tasks. However, if 1191:ref:`ref-tasks-package` needs to be run, BitBake needs to run those other tasks. 1192 1193It becomes more complicated if everything can come from an sstate cache 1194because some objects are simply not required at all. For example, you do 1195not need a compiler or native tools, such as quilt, if there isn't anything 1196to compile or patch. If the :ref:`do_package_write_* <ref-tasks-package_write_deb>` packages are available 1197from sstate, BitBake does not need the :ref:`ref-tasks-package` task data. 1198 1199To handle all these complexities, BitBake runs in two phases. The first 1200is the "setscene" stage. During this stage, BitBake first checks the 1201sstate cache for any targets it is planning to build. BitBake does a 1202fast check to see if the object exists rather than doing a complete download. 1203If nothing exists, the second phase, which is the setscene stage, 1204completes and the main build proceeds. 1205 1206If objects are found in the sstate cache, the build system works 1207backwards from the end targets specified by the user. For example, if an 1208image is being built, the build system first looks for the packages 1209needed for that image and the tools needed to construct an image. If 1210those are available, the compiler is not needed. Thus, the compiler is 1211not even downloaded. If something was found to be unavailable, or the 1212download or setscene task fails, the build system then tries to install 1213dependencies, such as the compiler, from the cache. 1214 1215The availability of objects in the sstate cache is handled by the 1216function specified by the :term:`BB_HASHCHECK_FUNCTION` 1217variable and returns a list of available objects. The function specified 1218by the :term:`BB_SETSCENE_DEPVALID` 1219variable is the function that determines whether a given dependency 1220needs to be followed, and whether for any given relationship the 1221function needs to be passed. The function returns a True or False value. 1222 1223Images 1224------ 1225 1226The images produced by the build system are compressed forms of the root 1227filesystem and are ready to boot on a target device. You can see from 1228the :ref:`general workflow figure 1229<overview-manual/concepts:openembedded build system concepts>` that BitBake 1230output, in part, consists of images. This section takes a closer look at 1231this output: 1232 1233.. image:: figures/images.png 1234 :align: center 1235 :width: 75% 1236 1237.. note:: 1238 1239 For a list of example images that the Yocto Project provides, see the 1240 ":doc:`/ref-manual/images`" chapter in the Yocto Project Reference 1241 Manual. 1242 1243The build process writes images out to the :term:`Build Directory` inside 1244the ``tmp/deploy/images/machine/`` folder as shown in the figure. This 1245folder contains any files expected to be loaded on the target device. 1246The :term:`DEPLOY_DIR` variable points to the ``deploy`` directory, while the 1247:term:`DEPLOY_DIR_IMAGE` variable points to the appropriate directory 1248containing images for the current configuration. 1249 1250- kernel-image: A kernel binary file. The 1251 :term:`KERNEL_IMAGETYPE` 1252 variable determines the naming scheme for the kernel image file. 1253 Depending on this variable, the file could begin with a variety of 1254 naming strings. The ``deploy/images/``\ machine directory can contain 1255 multiple image files for the machine. 1256 1257- root-filesystem-image: Root filesystems for the target device (e.g. 1258 ``*.ext3`` or ``*.bz2`` files). The 1259 :term:`IMAGE_FSTYPES` 1260 variable determines the root filesystem image type. The 1261 ``deploy/images/``\ machine directory can contain multiple root 1262 filesystems for the machine. 1263 1264- kernel-modules: Tarballs that contain all the modules built for the 1265 kernel. Kernel module tarballs exist for legacy purposes and can be 1266 suppressed by setting the 1267 :term:`MODULE_TARBALL_DEPLOY` 1268 variable to "0". The ``deploy/images/``\ machine directory can 1269 contain multiple kernel module tarballs for the machine. 1270 1271- bootloaders: If applicable to the target machine, bootloaders 1272 supporting the image. The ``deploy/images/``\ machine directory can 1273 contain multiple bootloaders for the machine. 1274 1275- symlinks: The ``deploy/images/``\ machine folder contains a symbolic 1276 link that points to the most recently built file for each machine. 1277 These links might be useful for external scripts that need to obtain 1278 the latest version of each file. 1279 1280Application Development SDK 1281--------------------------- 1282 1283In the :ref:`general workflow figure 1284<overview-manual/concepts:openembedded build system concepts>`, the 1285output labeled "Application Development SDK" represents an SDK. The SDK 1286generation process differs depending on whether you build an extensible 1287SDK (e.g. ``bitbake -c populate_sdk_ext`` imagename) or a standard SDK 1288(e.g. ``bitbake -c populate_sdk`` imagename). This section takes a 1289closer look at this output: 1290 1291.. image:: figures/sdk.png 1292 :width: 100% 1293 1294The specific form of this output is a set of files that includes a 1295self-extracting SDK installer (``*.sh``), host and target manifest 1296files, and files used for SDK testing. When the SDK installer file is 1297run, it installs the SDK. The SDK consists of a cross-development 1298toolchain, a set of libraries and headers, and an SDK environment setup 1299script. Running this installer essentially sets up your 1300cross-development environment. You can think of the cross-toolchain as 1301the "host" part because it runs on the SDK machine. You can think of the 1302libraries and headers as the "target" part because they are built for 1303the target hardware. The environment setup script is added so that you 1304can initialize the environment before using the tools. 1305 1306.. note:: 1307 1308 - The Yocto Project supports several methods by which you can set up 1309 this cross-development environment. These methods include 1310 downloading pre-built SDK installers or building and installing 1311 your own SDK installer. 1312 1313 - For background information on cross-development toolchains in the 1314 Yocto Project development environment, see the 1315 ":ref:`overview-manual/concepts:cross-development toolchain generation`" 1316 section. 1317 1318 - For information on setting up a cross-development environment, see 1319 the :doc:`/sdk-manual/index` manual. 1320 1321All the output files for an SDK are written to the ``deploy/sdk`` folder 1322inside the :term:`Build Directory` as shown in the previous figure. Depending 1323on the type of SDK, there are several variables to configure these files. 1324The variables associated with an extensible SDK are: 1325 1326- :term:`DEPLOY_DIR`: Points to 1327 the ``deploy`` directory. 1328 1329- :term:`SDK_EXT_TYPE`: 1330 Controls whether or not shared state artifacts are copied into the 1331 extensible SDK. By default, all required shared state artifacts are 1332 copied into the SDK. 1333 1334- :term:`SDK_INCLUDE_PKGDATA`: 1335 Specifies whether or not packagedata is included in the extensible 1336 SDK for all recipes in the "world" target. 1337 1338- :term:`SDK_INCLUDE_TOOLCHAIN`: 1339 Specifies whether or not the toolchain is included when building the 1340 extensible SDK. 1341 1342- :term:`ESDK_LOCALCONF_ALLOW`: 1343 A list of variables allowed through from the build system 1344 configuration into the extensible SDK configuration. 1345 1346- :term:`ESDK_LOCALCONF_REMOVE`: 1347 A list of variables not allowed through from the build system 1348 configuration into the extensible SDK configuration. 1349 1350- :term:`ESDK_CLASS_INHERIT_DISABLE`: 1351 A list of classes to remove from the 1352 :term:`INHERIT` value globally 1353 within the extensible SDK configuration. 1354 1355This next list, shows the variables associated with a standard SDK: 1356 1357- :term:`DEPLOY_DIR`: Points to 1358 the ``deploy`` directory. 1359 1360- :term:`SDKMACHINE`: Specifies 1361 the architecture of the machine on which the cross-development tools 1362 are run to create packages for the target hardware. 1363 1364- :term:`SDKIMAGE_FEATURES`: 1365 Lists the features to include in the "target" part of the SDK. 1366 1367- :term:`TOOLCHAIN_HOST_TASK`: 1368 Lists packages that make up the host part of the SDK (i.e. the part 1369 that runs on the :term:`SDKMACHINE`). When you use 1370 ``bitbake -c populate_sdk imagename`` to create the SDK, a set of 1371 default packages apply. This variable allows you to add more 1372 packages. 1373 1374- :term:`TOOLCHAIN_TARGET_TASK`: 1375 Lists packages that make up the target part of the SDK (i.e. the part 1376 built for the target hardware). 1377 1378- :term:`SDKPATHINSTALL`: Defines the 1379 default SDK installation path offered by the installation script. 1380 1381- :term:`SDK_HOST_MANIFEST`: 1382 Lists all the installed packages that make up the host part of the 1383 SDK. This variable also plays a minor role for extensible SDK 1384 development as well. However, it is mainly used for the standard SDK. 1385 1386- :term:`SDK_TARGET_MANIFEST`: 1387 Lists all the installed packages that make up the target part of the 1388 SDK. This variable also plays a minor role for extensible SDK 1389 development as well. However, it is mainly used for the standard SDK. 1390 1391Cross-Development Toolchain Generation 1392====================================== 1393 1394The Yocto Project does most of the work for you when it comes to 1395creating :ref:`sdk-manual/intro:the cross-development toolchain`. This 1396section provides some technical background on how cross-development 1397toolchains are created and used. For more information on toolchains, you 1398can also see the :doc:`/sdk-manual/index` manual. 1399 1400In the Yocto Project development environment, cross-development 1401toolchains are used to build images and applications that run on the 1402target hardware. With just a few commands, the OpenEmbedded build system 1403creates these necessary toolchains for you. 1404 1405The following figure shows a high-level build environment regarding 1406toolchain construction and use. 1407 1408.. image:: figures/cross-development-toolchains.png 1409 :width: 100% 1410 1411Most of the work occurs on the Build Host. This is the machine used to 1412build images and generally work within the Yocto Project 1413environment. When you run 1414:term:`BitBake` to create an image, the 1415OpenEmbedded build system uses the host ``gcc`` compiler to bootstrap a 1416cross-compiler named ``gcc-cross``. The ``gcc-cross`` compiler is what 1417BitBake uses to compile source files when creating the target image. You 1418can think of ``gcc-cross`` simply as an automatically generated 1419cross-compiler that is used internally within BitBake only. 1420 1421.. note:: 1422 1423 The extensible SDK does not use ``gcc-cross-canadian`` 1424 since this SDK ships a copy of the OpenEmbedded build system and the 1425 sysroot within it contains ``gcc-cross``. 1426 1427The chain of events that occurs when the standard toolchain is bootstrapped:: 1428 1429 binutils-cross -> linux-libc-headers -> gcc-cross -> libgcc-initial -> glibc -> libgcc -> gcc-runtime 1430 1431- ``gcc``: The compiler, GNU Compiler Collection (GCC). 1432 1433- ``binutils-cross``: The binary utilities needed in order 1434 to run the ``gcc-cross`` phase of the bootstrap operation and build the 1435 headers for the C library. 1436 1437- ``linux-libc-headers``: Headers needed for the cross-compiler and C library build. 1438 1439- ``libgcc-initial``: An initial version of the gcc support library needed 1440 to bootstrap ``glibc``. 1441 1442- ``libgcc``: The final version of the gcc support library which 1443 can only be built once there is a C library to link against. 1444 1445- ``glibc``: The GNU C Library. 1446 1447- ``gcc-cross``: The final stage of the bootstrap process for the 1448 cross-compiler. This stage results in the actual cross-compiler that 1449 BitBake uses when it builds an image for a targeted device. 1450 1451 This tool is a "native" tool (i.e. it is designed to run on 1452 the build host). 1453 1454- ``gcc-runtime``: Runtime libraries resulting from the toolchain 1455 bootstrapping process. This tool produces a binary that consists of 1456 the runtime libraries need for the targeted device. 1457 1458You can use the OpenEmbedded build system to build an installer for the 1459relocatable SDK used to develop applications. When you run the 1460installer, it installs the toolchain, which contains the development 1461tools (e.g., ``gcc-cross-canadian``, ``binutils-cross-canadian``, and 1462other ``nativesdk-*`` tools), which are tools native to the SDK (i.e. 1463native to :term:`SDK_ARCH`), you need to cross-compile and test your 1464software. The figure shows the commands you use to easily build out 1465this toolchain. This cross-development toolchain is built to execute on the 1466:term:`SDKMACHINE`, which might or might not be the same machine as 1467the Build Host. 1468 1469.. note:: 1470 1471 If your target architecture is supported by the Yocto Project, you 1472 can take advantage of pre-built images that ship with the Yocto 1473 Project and already contain cross-development toolchain installers. 1474 1475Here is the bootstrap process for the relocatable toolchain:: 1476 1477 gcc -> binutils-crosssdk -> gcc-crosssdk-initial -> linux-libc-headers -> glibc-initial -> nativesdk-glibc -> gcc-crosssdk -> gcc-cross-canadian 1478 1479- ``gcc``: The build host's GNU Compiler Collection (GCC). 1480 1481- ``binutils-crosssdk``: The bare minimum binary utilities needed in 1482 order to run the ``gcc-crosssdk-initial`` phase of the bootstrap 1483 operation. 1484 1485- ``gcc-crosssdk-initial``: An early stage of the bootstrap process for 1486 creating the cross-compiler. This stage builds enough of the 1487 ``gcc-crosssdk`` and supporting pieces so that the final stage of the 1488 bootstrap process can produce the finished cross-compiler. This tool 1489 is a "native" binary that runs on the build host. 1490 1491- ``linux-libc-headers``: Headers needed for the cross-compiler. 1492 1493- ``glibc-initial``: An initial version of the Embedded GLIBC needed to 1494 bootstrap ``nativesdk-glibc``. 1495 1496- ``nativesdk-glibc``: The Embedded GLIBC needed to bootstrap the 1497 ``gcc-crosssdk``. 1498 1499- ``gcc-crosssdk``: The final stage of the bootstrap process for the 1500 relocatable cross-compiler. The ``gcc-crosssdk`` is a transitory 1501 compiler and never leaves the build host. Its purpose is to help in 1502 the bootstrap process to create the eventual ``gcc-cross-canadian`` 1503 compiler, which is relocatable. This tool is also a "native" package 1504 (i.e. it is designed to run on the build host). 1505 1506- ``gcc-cross-canadian``: The final relocatable cross-compiler. When 1507 run on the :term:`SDKMACHINE`, 1508 this tool produces executable code that runs on the target device. 1509 Only one cross-canadian compiler is produced per architecture since 1510 they can be targeted at different processor optimizations using 1511 configurations passed to the compiler through the compile commands. 1512 This circumvents the need for multiple compilers and thus reduces the 1513 size of the toolchains. 1514 1515.. note:: 1516 1517 For information on advantages gained when building a 1518 cross-development toolchain installer, see the 1519 ":ref:`sdk-manual/appendix-obtain:building an sdk installer`" appendix 1520 in the Yocto Project Application Development and the 1521 Extensible Software Development Kit (eSDK) manual. 1522 1523Shared State Cache 1524================== 1525 1526By design, the OpenEmbedded build system builds everything from scratch 1527unless :term:`BitBake` can determine 1528that parts do not need to be rebuilt. Fundamentally, building from 1529scratch is attractive as it means all parts are built fresh and there is 1530no possibility of stale data that can cause problems. When 1531developers hit problems, they typically default back to building from 1532scratch so they have a known state from the start. 1533 1534Building an image from scratch is both an advantage and a disadvantage 1535to the process. As mentioned in the previous paragraph, building from 1536scratch ensures that everything is current and starts from a known 1537state. However, building from scratch also takes much longer as it 1538generally means rebuilding things that do not necessarily need to be 1539rebuilt. 1540 1541The Yocto Project implements shared state code that supports incremental 1542builds. The implementation of the shared state code answers the 1543following questions that were fundamental roadblocks within the 1544OpenEmbedded incremental build support system: 1545 1546- What pieces of the system have changed and what pieces have not 1547 changed? 1548 1549- How are changed pieces of software removed and replaced? 1550 1551- How are pre-built components that do not need to be rebuilt from 1552 scratch used when they are available? 1553 1554For the first question, the build system detects changes in the "inputs" 1555to a given task by creating a checksum (or signature) of the task's 1556inputs. If the checksum changes, the system assumes the inputs have 1557changed and the task needs to be rerun. For the second question, the 1558shared state (sstate) code tracks which tasks add which output to the 1559build process. This means the output from a given task can be removed, 1560upgraded or otherwise manipulated. The third question is partly 1561addressed by the solution for the second question assuming the build 1562system can fetch the sstate objects from remote locations and install 1563them if they are deemed to be valid. 1564 1565.. note:: 1566 1567 - The build system does not maintain 1568 :term:`PR` information as part of 1569 the shared state packages. Consequently, there are considerations that 1570 affect maintaining shared state feeds. For information on how the 1571 build system works with packages and can track incrementing :term:`PR` 1572 information, see the ":ref:`dev-manual/packages:automatically incrementing a package version number`" 1573 section in the Yocto Project Development Tasks Manual. 1574 1575 - The code in the build system that supports incremental builds is 1576 complex. For techniques that help you work around issues 1577 related to shared state code, see the 1578 ":ref:`dev-manual/debugging:viewing metadata used to create the input signature of a shared state task`" 1579 and 1580 ":ref:`dev-manual/debugging:invalidating shared state to force a task to run`" 1581 sections both in the Yocto Project Development Tasks Manual. 1582 1583The rest of this section goes into detail about the overall incremental 1584build architecture, the checksums (signatures), and shared state. 1585 1586Overall Architecture 1587-------------------- 1588 1589When determining what parts of the system need to be built, BitBake 1590works on a per-task basis rather than a per-recipe basis. You might 1591wonder why using a per-task basis is preferred over a per-recipe basis. 1592To help explain, consider having the IPK packaging backend enabled and 1593then switching to DEB. In this case, the 1594:ref:`ref-tasks-install` and 1595:ref:`ref-tasks-package` task outputs 1596are still valid. However, with a per-recipe approach, the build would 1597not include the ``.deb`` files. Consequently, you would have to 1598invalidate the whole build and rerun it. Rerunning everything is not the 1599best solution. Also, in this case, the core must be "taught" much about 1600specific tasks. This methodology does not scale well and does not allow 1601users to easily add new tasks in layers or as external recipes without 1602touching the packaged-staging core. 1603 1604Checksums (Signatures) 1605---------------------- 1606 1607The shared state code uses a checksum, which is a unique signature of a 1608task's inputs, to determine if a task needs to be run again. Because it 1609is a change in a task's inputs that triggers a rerun, the process needs 1610to detect all the inputs to a given task. For shell tasks, this turns 1611out to be fairly easy because the build process generates a "run" shell 1612script for each task and it is possible to create a checksum that gives 1613you a good idea of when the task's data changes. 1614 1615To complicate the problem, there are things that should not be included 1616in the checksum. First, there is the actual specific build path of a 1617given task --- the :term:`WORKDIR`. It 1618does not matter if the work directory changes because it should not 1619affect the output for target packages. Also, the build process has the 1620objective of making native or cross packages relocatable. 1621 1622.. note:: 1623 1624 Both native and cross packages run on the 1625 build host. However, cross packages generate output for the target 1626 architecture. 1627 1628The checksum therefore needs to exclude :term:`WORKDIR`. The simplistic 1629approach for excluding the work directory is to set :term:`WORKDIR` to some 1630fixed value and create the checksum for the "run" script. 1631 1632Another problem results from the "run" scripts containing functions that 1633might or might not get called. The incremental build solution contains 1634code that figures out dependencies between shell functions. This code is 1635used to prune the "run" scripts down to the minimum set, thereby 1636alleviating this problem and making the "run" scripts much more readable 1637as a bonus. 1638 1639So far, there are solutions for shell scripts. What about Python tasks? The 1640same approach applies even though these tasks are more difficult. The 1641process needs to figure out what variables a Python function accesses 1642and what functions it calls. Again, the incremental build solution 1643contains code that first figures out the variable and function 1644dependencies, and then creates a checksum for the data used as the input 1645to the task. 1646 1647Like the :term:`WORKDIR` case, there can be situations where dependencies should be 1648ignored. For these situations, you can instruct the build process to 1649ignore a dependency by using a line like the following:: 1650 1651 PACKAGE_ARCHS[vardepsexclude] = "MACHINE" 1652 1653This example ensures that the :term:`PACKAGE_ARCHS` variable 1654does not depend on the value of :term:`MACHINE`, even if it does 1655reference it. 1656 1657Equally, there are cases where you need to add dependencies BitBake is 1658not able to find. You can accomplish this by using a line like the 1659following:: 1660 1661 PACKAGE_ARCHS[vardeps] = "MACHINE" 1662 1663This example explicitly 1664adds the :term:`MACHINE` variable as a dependency for :term:`PACKAGE_ARCHS`. 1665 1666As an example, consider a case with in-line Python where BitBake is not 1667able to figure out dependencies. When running in debug mode (i.e. using 1668``-DDD``), BitBake produces output when it discovers something for which 1669it cannot figure out dependencies. The Yocto Project team has currently 1670not managed to cover those dependencies in detail and is aware of the 1671need to fix this situation. 1672 1673Thus far, this section has limited discussion to the direct inputs into 1674a task. Information based on direct inputs is referred to as the 1675"basehash" in the code. However, the question of a task's indirect 1676inputs still exits --- items already built and present in the 1677:term:`Build Directory`. The checksum (or 1678signature) for a particular task needs to add the hashes of all the 1679tasks on which the particular task depends. Choosing which dependencies 1680to add is a policy decision. However, the effect is to generate a 1681checksum that combines the basehash and the hashes of the task's 1682dependencies. 1683 1684At the code level, there are multiple ways by which both the basehash 1685and the dependent task hashes can be influenced. Within the BitBake 1686configuration file, you can give BitBake some extra information to help 1687it construct the basehash. The following statement effectively results 1688in a list of global variable dependency excludes (i.e. variables never 1689included in any checksum):: 1690 1691 BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \\ 1692 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \\ 1693 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \\ 1694 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \\ 1695 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" 1696 1697The previous example does not include :term:`WORKDIR` since that variable is 1698actually constructed as a path within :term:`TMPDIR`, which is included above. 1699 1700The rules for deciding which hashes of dependent tasks to include 1701through dependency chains are more complex and are generally 1702accomplished with a Python function. The code in 1703``meta/lib/oe/sstatesig.py`` shows two examples of this and also 1704illustrates how you can insert your own policy into the system if so 1705desired. This file defines the two basic signature generators 1706:term:`OpenEmbedded-Core (OE-Core)` uses: "OEBasic" and 1707"OEBasicHash". By default, a dummy "noop" signature handler is enabled 1708in BitBake. This means that behavior is unchanged from previous 1709versions. OE-Core uses the "OEBasicHash" signature handler by default 1710through this setting in the ``bitbake.conf`` file:: 1711 1712 BB_SIGNATURE_HANDLER ?= "OEBasicHash" 1713 1714The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same 1715as the "OEBasic" version but adds the task hash to the :ref:`stamp 1716files <overview-manual/concepts:stamp files and the rerunning of tasks>`. This 1717results in any metadata change that changes the task hash, automatically causing 1718the task to be run again. This removes the need to bump 1719:term:`PR` values, and changes to metadata 1720automatically ripple across the build. 1721 1722It is also worth noting that the end result of these signature 1723generators is to make some dependency and hash information available to 1724the build. This information includes: 1725 1726- ``BB_BASEHASH:task-``\ taskname: The base hashes for each task in the 1727 recipe. 1728 1729- ``BB_BASEHASH_``\ filename\ ``:``\ taskname: The base hashes for each 1730 dependent task. 1731 1732- :term:`BB_TASKHASH`: The hash of the currently running task. 1733 1734Shared State 1735------------ 1736 1737Checksums and dependencies, as discussed in the previous section, solve 1738half the problem of supporting a shared state. The other half of the 1739problem is being able to use checksum information during the build and 1740being able to reuse or rebuild specific components. 1741 1742The :ref:`ref-classes-sstate` class is a relatively generic implementation of 1743how to "capture" a snapshot of a given task. The idea is that the build process 1744does not care about the source of a task's output. Output could be freshly 1745built or it could be downloaded and unpacked from somewhere. In other words, 1746the build process does not need to worry about its origin. 1747 1748Two types of output exist. One type is just about creating a directory 1749in :term:`WORKDIR`. A good example is 1750the output of either 1751:ref:`ref-tasks-install` or 1752:ref:`ref-tasks-package`. The other 1753type of output occurs when a set of data is merged into a shared 1754directory tree such as the sysroot. 1755 1756The Yocto Project team has tried to keep the details of the 1757implementation hidden in the :ref:`ref-classes-sstate` class. From a user's perspective, 1758adding shared state wrapping to a task is as simple as this 1759:ref:`ref-tasks-deploy` example taken from the :ref:`ref-classes-deploy` class:: 1760 1761 DEPLOYDIR = "${WORKDIR}/deploy-${PN}" 1762 SSTATETASKS += "do_deploy" 1763 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" 1764 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" 1765 1766 python do_deploy_setscene () { 1767 sstate_setscene(d) 1768 } 1769 addtask do_deploy_setscene 1770 do_deploy[dirs] = "${DEPLOYDIR} ${B}" 1771 do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" 1772 1773The following list explains the previous example: 1774 1775- Adding ``do_deploy`` to ``SSTATETASKS`` adds some required sstate-related 1776 processing, which is implemented in the :ref:`ref-classes-sstate` class, to 1777 before and after the :ref:`ref-tasks-deploy` task. 1778 1779- The ``do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"`` declares that 1780 :ref:`ref-tasks-deploy` places its output in ``${DEPLOYDIR}`` when run normally 1781 (i.e. when not using the sstate cache). This output becomes the input 1782 to the shared state cache. 1783 1784- The ``do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"`` line 1785 causes the contents of the shared state cache to be copied to 1786 ``${DEPLOY_DIR_IMAGE}``. 1787 1788 .. note:: 1789 1790 If :ref:`ref-tasks-deploy` is not already in the shared state cache or if its input 1791 checksum (signature) has changed from when the output was cached, the task 1792 runs to populate the shared state cache, after which the contents of the 1793 shared state cache is copied to ${:term:`DEPLOY_DIR_IMAGE`}. If 1794 :ref:`ref-tasks-deploy` is in the shared state cache and its signature indicates 1795 that the cached output is still valid (i.e. if no relevant task inputs 1796 have changed), then the contents of the shared state cache copies 1797 directly to ${:term:`DEPLOY_DIR_IMAGE`} by the ``do_deploy_setscene`` task 1798 instead, skipping the :ref:`ref-tasks-deploy` task. 1799 1800- The following task definition is glue logic needed to make the 1801 previous settings effective:: 1802 1803 python do_deploy_setscene () { 1804 sstate_setscene(d) 1805 } 1806 addtask do_deploy_setscene 1807 1808 ``sstate_setscene()`` takes the flags above as input and accelerates the 1809 :ref:`ref-tasks-deploy` task through the shared state cache if possible. If 1810 the task was accelerated, ``sstate_setscene()`` returns True. Otherwise, it 1811 returns False, and the normal :ref:`ref-tasks-deploy` task runs. For more 1812 information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:setscene`" 1813 section in the BitBake User Manual. 1814 1815- The ``do_deploy[dirs] = "${DEPLOYDIR} ${B}"`` line creates ``${DEPLOYDIR}`` 1816 and ``${B}`` before the :ref:`ref-tasks-deploy` task runs, and also sets the 1817 current working directory of :ref:`ref-tasks-deploy` to ``${B}``. For more 1818 information, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" 1819 section in the BitBake User Manual. 1820 1821 .. note:: 1822 1823 In cases where ``sstate-inputdirs`` and ``sstate-outputdirs`` would be 1824 the same, you can use ``sstate-plaindirs``. For example, to preserve the 1825 ${:term:`PKGD`} and ${:term:`PKGDEST`} output from the :ref:`ref-tasks-package` 1826 task, use the following:: 1827 1828 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" 1829 1830 1831- The ``do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"`` line appends extra 1832 metadata to the :ref:`stamp file <overview-manual/concepts:stamp files and the rerunning of tasks>`. 1833 In this case, the metadata makes the task specific to a machine's architecture. 1834 See the ":ref:`bitbake-user-manual/bitbake-user-manual-execution:the task list`" 1835 section in the BitBake User Manual for more information on the 1836 ``stamp-extra-info`` flag. 1837 1838- ``sstate-inputdirs`` and ``sstate-outputdirs`` can also be used with 1839 multiple directories. For example, the following declares 1840 :term:`PKGDESTWORK` and ``SHLIBWORK`` as shared state input directories, 1841 which populates the shared state cache, and :term:`PKGDATA_DIR` and 1842 ``SHLIBSDIR`` as the corresponding shared state output directories:: 1843 1844 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" 1845 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" 1846 1847- These methods also include the ability to take a lockfile when 1848 manipulating shared state directory structures, for cases where file 1849 additions or removals are sensitive:: 1850 1851 do_package[sstate-lockfile] = "${PACKAGELOCK}" 1852 1853Behind the scenes, the shared state code works by looking in 1854:term:`SSTATE_DIR` and 1855:term:`SSTATE_MIRRORS` for 1856shared state files. Here is an example:: 1857 1858 SSTATE_MIRRORS ?= "\ 1859 file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \ 1860 file://.* file:///some/local/dir/sstate/PATH" 1861 1862.. note:: 1863 1864 The shared state directory (:term:`SSTATE_DIR`) is organized into two-character 1865 subdirectories, where the subdirectory names are based on the first two 1866 characters of the hash. 1867 If the shared state directory structure for a mirror has the same structure 1868 as :term:`SSTATE_DIR`, you must specify "PATH" as part of the URI to enable the build 1869 system to map to the appropriate subdirectory. 1870 1871The shared state package validity can be detected just by looking at the 1872filename since the filename contains the task checksum (or signature) as 1873described earlier in this section. If a valid shared state package is 1874found, the build process downloads it and uses it to accelerate the 1875task. 1876 1877The build processes use the ``*_setscene`` tasks for the task 1878acceleration phase. BitBake goes through this phase before the main 1879execution code and tries to accelerate any tasks for which it can find 1880shared state packages. If a shared state package for a task is 1881available, the shared state package is used. This means the task and any 1882tasks on which it is dependent are not executed. 1883 1884As a real world example, the aim is when building an IPK-based image, 1885only the 1886:ref:`ref-tasks-package_write_ipk` 1887tasks would have their shared state packages fetched and extracted. 1888Since the sysroot is not used, it would never get extracted. This is 1889another reason why a task-based approach is preferred over a 1890recipe-based approach, which would have to install the output from every 1891task. 1892 1893Hash Equivalence 1894---------------- 1895 1896The above section explained how BitBake skips the execution of tasks 1897whose output can already be found in the Shared State cache. 1898 1899During a build, it may often be the case that the output / result of a task might 1900be unchanged despite changes in the task's input values. An example might be 1901whitespace changes in some input C code. In project terms, this is what we define 1902as "equivalence". 1903 1904To keep track of such equivalence, BitBake has to manage three hashes 1905for each task: 1906 1907- The *task hash* explained earlier: computed from the recipe metadata, 1908 the task code and the task hash values from its dependencies. 1909 When changes are made, these task hashes are therefore modified, 1910 causing the task to re-execute. The task hashes of tasks depending on this 1911 task are therefore modified too, causing the whole dependency 1912 chain to re-execute. 1913 1914- The *output hash*, a new hash computed from the output of Shared State tasks, 1915 tasks that save their resulting output to a Shared State tarball. 1916 The mapping between the task hash and its output hash is reported 1917 to a new *Hash Equivalence* server. This mapping is stored in a database 1918 by the server for future reference. 1919 1920- The *unihash*, a new hash, initially set to the task hash for the task. 1921 This is used to track the *unicity* of task output, and we will explain 1922 how its value is maintained. 1923 1924When Hash Equivalence is enabled, BitBake computes the task hash 1925for each task by using the unihash of its dependencies, instead 1926of their task hash. 1927 1928Now, imagine that a Shared State task is modified because of a change in 1929its code or metadata, or because of a change in its dependencies. 1930Since this modifies its task hash, this task will need re-executing. 1931Its output hash will therefore be computed again. 1932 1933Then, the new mapping between the new task hash and its output hash 1934will be reported to the Hash Equivalence server. The server will 1935let BitBake know whether this output hash is the same as a previously 1936reported output hash, for a different task hash. 1937 1938If the output hash is already known, BitBake will update the task's 1939unihash to match the original task hash that generated that output. 1940Thanks to this, the depending tasks will keep a previously recorded 1941task hash, and BitBake will be able to retrieve their output from 1942the Shared State cache, instead of re-executing them. Similarly, the 1943output of further downstream tasks can also be retrieved from Shared 1944State. 1945 1946If the output hash is unknown, a new entry will be created on the Hash 1947Equivalence server, matching the task hash to that output. 1948The depending tasks, still having a new task hash because of the 1949change, will need to re-execute as expected. The change propagates 1950to the depending tasks. 1951 1952To summarize, when Hash Equivalence is enabled, a change in one of the 1953tasks in BitBake's run queue doesn't have to propagate to all the 1954downstream tasks that depend on the output of this task, causing a 1955full rebuild of such tasks, and so on with the next depending tasks. 1956Instead, when the output of this task remains identical to previously 1957recorded output, BitBake can safely retrieve all the downstream 1958task output from the Shared State cache. 1959 1960.. note:: 1961 1962 Having :doc:`/test-manual/reproducible-builds` is a key ingredient for 1963 the stability of the task's output hash. Therefore, the effectiveness 1964 of Hash Equivalence strongly depends on it. 1965 1966 Recipes that are not reproducible may have undesired behavior if hash 1967 equivalence is enabled, since the non-reproducible diverging output maybe be 1968 remapped to an older sstate object in the cache by the server. If a recipe 1969 is non-reproducible in trivial ways, such as different timestamps, this is 1970 likely not a problem. However recipes that have more dramatic changes (such 1971 as completely different file names) will likely outright fail since the 1972 downstream sstate objects are not actually equivalent to what was just 1973 built. 1974 1975This applies to multiple scenarios: 1976 1977- A "trivial" change to a recipe that doesn't impact its generated output, 1978 such as whitespace changes, modifications to unused code paths or 1979 in the ordering of variables. 1980 1981- Shared library updates, for example to fix a security vulnerability. 1982 For sure, the programs using such a library should be rebuilt, but 1983 their new binaries should remain identical. The corresponding tasks should 1984 have a different output hash because of the change in the hash of their 1985 library dependency, but thanks to their output being identical, Hash 1986 Equivalence will stop the propagation down the dependency chain. 1987 1988- Native tool updates. Though the depending tasks should be rebuilt, 1989 it's likely that they will generate the same output and be marked 1990 as equivalent. 1991 1992This mechanism is enabled by default in Poky, and is controlled by three 1993variables: 1994 1995- :term:`bitbake:BB_HASHSERVE`, specifying a local or remote Hash 1996 Equivalence server to use. 1997 1998- :term:`BB_HASHSERVE_UPSTREAM`, when ``BB_HASHSERVE = "auto"``, 1999 allowing to connect the local server to an upstream one. 2000 2001- :term:`bitbake:BB_SIGNATURE_HANDLER`, which must be set to ``OEEquivHash``. 2002 2003Therefore, the default configuration in Poky corresponds to the 2004below settings:: 2005 2006 BB_HASHSERVE = "auto" 2007 BB_SIGNATURE_HANDLER = "OEEquivHash" 2008 2009Rather than starting a local server, another possibility is to rely 2010on a Hash Equivalence server on a network, by setting:: 2011 2012 BB_HASHSERVE = "<HOSTNAME>:<PORT>" 2013 2014.. note:: 2015 2016 The shared Hash Equivalence server needs to be maintained together with the 2017 Shared State cache. Otherwise, the server could report Shared State hashes 2018 that only exist on specific clients. 2019 2020 We therefore recommend that one Hash Equivalence server be set up to 2021 correspond with a given Shared State cache, and to start this server 2022 in *read-only mode*, so that it doesn't store equivalences for 2023 Shared State caches that are local to clients. 2024 2025 See the :term:`BB_HASHSERVE` reference for details about starting 2026 a Hash Equivalence server. 2027 2028See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ 2029of Joshua Watt's `Hash Equivalence and Reproducible Builds 2030<https://elinux.org/images/3/37/Hash_Equivalence_and_Reproducible_Builds.pdf>`__ 2031presentation at ELC 2020 for a very synthetic introduction to the 2032Hash Equivalence implementation in the Yocto Project. 2033 2034Automatically Added Runtime Dependencies 2035======================================== 2036 2037The OpenEmbedded build system automatically adds common types of runtime 2038dependencies between packages, which means that you do not need to 2039explicitly declare the packages using 2040:term:`RDEPENDS`. There are three automatic 2041mechanisms (``shlibdeps``, ``pcdeps``, and ``depchains``) that 2042handle shared libraries, package configuration (pkg-config) modules, and 2043``-dev`` and ``-dbg`` packages, respectively. For other types of runtime 2044dependencies, you must manually declare the dependencies. 2045 2046- ``shlibdeps``: During the 2047 :ref:`ref-tasks-package` task of 2048 each recipe, all shared libraries installed by the recipe are 2049 located. For each shared library, the package that contains the 2050 shared library is registered as providing the shared library. More 2051 specifically, the package is registered as providing the 2052 :wikipedia:`soname <Soname>` of the library. The 2053 resulting shared-library-to-package mapping is saved globally in 2054 :term:`PKGDATA_DIR` by the 2055 :ref:`ref-tasks-packagedata` 2056 task. 2057 2058 Simultaneously, all executables and shared libraries installed by the 2059 recipe are inspected to see what shared libraries they link against. 2060 For each shared library dependency that is found, :term:`PKGDATA_DIR` is 2061 queried to see if some package (likely from a different recipe) 2062 contains the shared library. If such a package is found, a runtime 2063 dependency is added from the package that depends on the shared 2064 library to the package that contains the library. 2065 2066 The automatically added runtime dependency also includes a version 2067 restriction. This version restriction specifies that at least the 2068 current version of the package that provides the shared library must 2069 be used, as if "package (>= version)" had been added to :term:`RDEPENDS`. 2070 This forces an upgrade of the package containing the shared library 2071 when installing the package that depends on the library, if needed. 2072 2073 If you want to avoid a package being registered as providing a 2074 particular shared library (e.g. because the library is for internal 2075 use only), then add the library to 2076 :term:`PRIVATE_LIBS` inside 2077 the package's recipe. 2078 2079- ``pcdeps``: During the :ref:`ref-tasks-package` task of each recipe, all 2080 pkg-config modules (``*.pc`` files) installed by the recipe are 2081 located. For each module, the package that contains the module is 2082 registered as providing the module. The resulting module-to-package 2083 mapping is saved globally in :term:`PKGDATA_DIR` by the 2084 :ref:`ref-tasks-packagedata` task. 2085 2086 Simultaneously, all pkg-config modules installed by the recipe are 2087 inspected to see what other pkg-config modules they depend on. A 2088 module is seen as depending on another module if it contains a 2089 "Requires:" line that specifies the other module. For each module 2090 dependency, :term:`PKGDATA_DIR` is queried to see if some package 2091 contains the module. If such a package is found, a runtime dependency 2092 is added from the package that depends on the module to the package 2093 that contains the module. 2094 2095 .. note:: 2096 2097 The 2098 pcdeps 2099 mechanism most often infers dependencies between 2100 -dev 2101 packages. 2102 2103- ``depchains``: If a package ``foo`` depends on a package ``bar``, 2104 then ``foo-dev`` and ``foo-dbg`` are also made to depend on 2105 ``bar-dev`` and ``bar-dbg``, respectively. Taking the ``-dev`` 2106 packages as an example, the ``bar-dev`` package might provide headers 2107 and shared library symlinks needed by ``foo-dev``, which shows the 2108 need for a dependency between the packages. 2109 2110 The dependencies added by ``depchains`` are in the form of 2111 :term:`RRECOMMENDS`. 2112 2113 .. note:: 2114 2115 By default, ``foo-dev`` also has an :term:`RDEPENDS`-style dependency on 2116 ``foo``, because the default value of ``RDEPENDS:${PN}-dev`` (set in 2117 ``bitbake.conf``) includes "${PN}". 2118 2119 To ensure that the dependency chain is never broken, ``-dev`` and 2120 ``-dbg`` packages are always generated by default, even if the 2121 packages turn out to be empty. See the 2122 :term:`ALLOW_EMPTY` variable 2123 for more information. 2124 2125The :ref:`ref-tasks-package` task depends on the :ref:`ref-tasks-packagedata` 2126task of each recipe in :term:`DEPENDS` through use of a 2127``[``\ :ref:`deptask <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]`` 2128declaration, which guarantees that the required shared-library / 2129module-to-package mapping information will be available when needed as long as 2130:term:`DEPENDS` has been correctly set. 2131 2132Fakeroot and Pseudo 2133=================== 2134 2135Some tasks are easier to implement when allowed to perform certain 2136operations that are normally reserved for the root user (e.g. 2137:ref:`ref-tasks-install`, 2138:ref:`do_package_write* <ref-tasks-package_write_deb>`, 2139:ref:`ref-tasks-rootfs`, and 2140:ref:`do_image_* <ref-tasks-image>`). For example, 2141the :ref:`ref-tasks-install` task benefits from being able to set the UID and GID 2142of installed files to arbitrary values. 2143 2144One approach to allowing tasks to perform root-only operations would be 2145to require :term:`BitBake` to run as 2146root. However, this method is cumbersome and has security issues. The 2147approach that is actually used is to run tasks that benefit from root 2148privileges in a "fake" root environment. Within this environment, the 2149task and its child processes believe that they are running as the root 2150user, and see an internally consistent view of the filesystem. As long 2151as generating the final output (e.g. a package or an image) does not 2152require root privileges, the fact that some earlier steps ran in a fake 2153root environment does not cause problems. 2154 2155The capability to run tasks in a fake root environment is known as 2156"`fakeroot <http://man.he.net/man1/fakeroot>`__", which is derived from 2157the BitBake keyword/variable flag that requests a fake root environment 2158for a task. 2159 2160In the :term:`OpenEmbedded Build System`, the program that implements 2161fakeroot is known as :yocto_home:`Pseudo </software-item/pseudo/>`. Pseudo 2162overrides system calls by using the environment variable ``LD_PRELOAD``, 2163which results in the illusion of running as root. To keep track of 2164"fake" file ownership and permissions resulting from operations that 2165require root permissions, Pseudo uses an SQLite 3 database. This 2166database is stored in 2167``${``\ :term:`WORKDIR`\ ``}/pseudo/files.db`` 2168for individual recipes. Storing the database in a file as opposed to in 2169memory gives persistence between tasks and builds, which is not 2170accomplished using fakeroot. 2171 2172.. note:: 2173 2174 If you add your own task that manipulates the same files or 2175 directories as a fakeroot task, then that task also needs to run 2176 under fakeroot. Otherwise, the task cannot run root-only operations, 2177 and cannot see the fake file ownership and permissions set by the 2178 other task. You need to also add a dependency on 2179 ``virtual/fakeroot-native:do_populate_sysroot``, giving the following:: 2180 2181 fakeroot do_mytask () { 2182 ... 2183 } 2184 do_mytask[depends] += "virtual/fakeroot-native:do_populate_sysroot" 2185 2186 2187For more information, see the 2188:term:`FAKEROOT* <bitbake:FAKEROOT>` variables in the 2189BitBake User Manual. You can also reference the "`Why Not 2190Fakeroot? <https://github.com/wrpseudo/pseudo/wiki/WhyNotFakeroot>`__" 2191article for background information on Fakeroot and Pseudo. 2192 2193BitBake Tasks Map 2194================= 2195 2196To understand how BitBake operates in the build directory and environment 2197we can consider the following recipes and diagram, to have full picture 2198about the tasks that BitBake runs to generate the final package file 2199for the recipe. 2200 2201We will have two recipes as an example: 2202 2203- ``libhello``: A recipe that provides a shared library 2204- ``sayhello``: A recipe that uses ``libhello`` library to do its job 2205 2206.. note:: 2207 2208 ``sayhello`` depends on ``libhello`` at compile time as it needs the shared 2209 library to do the dynamic linking process. It also depends on it at runtime 2210 as the shared library loader needs to find the library. 2211 For more details about dependencies check :ref:`ref-varlocality-recipe-dependencies`. 2212 2213``libhello`` sources are as follows: 2214 2215- ``LICENSE``: This is the license associated with this library 2216- ``Makefile``: The file used by ``make`` to build the library 2217- ``hellolib.c``: The implementation of the library 2218- ``hellolib.h``: The C header of the library 2219 2220``sayhello`` sources are as follows: 2221 2222- ``LICENSE``: This is the license associated with this project 2223- ``Makefile``: The file used by ``make`` to build the project 2224- ``sayhello.c``: The source file of the project 2225 2226Before presenting the contents of each file, here are the steps 2227that we need to follow to accomplish what we want in the first place, 2228which is integrating ``sayhello`` in our root file system: 2229 2230#. Create a Git repository for each project with the corresponding files 2231 2232#. Create a recipe for each project 2233 2234#. Make sure that ``sayhello`` recipe :term:`DEPENDS` on ``libhello`` 2235 2236#. Make sure that ``sayhello`` recipe :term:`RDEPENDS` on ``libhello`` 2237 2238#. Add ``sayhello`` to :term:`IMAGE_INSTALL` to integrate it into 2239 the root file system 2240 2241The contents of ``libhello/Makefile`` are:: 2242 2243 LIB=libhello.so 2244 2245 all: $(LIB) 2246 2247 $(LIB): hellolib.o 2248 $(CC) $< -Wl,-soname,$(LIB).1 -fPIC $(LDFLAGS) -shared -o $(LIB).1.0 2249 2250 %.o: %.c 2251 $(CC) -c $< 2252 2253 clean: 2254 rm -rf *.o *.so* 2255 2256.. note:: 2257 2258 When creating shared libraries, it is strongly recommended to follow the Linux 2259 conventions and guidelines (see `this article 2260 <https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__ 2261 for some background). 2262 2263.. note:: 2264 2265 When creating ``Makefile`` files, it is strongly recommended to use ``CC``, ``LDFLAGS`` 2266 and ``CFLAGS`` as BitBake will set them as environment variables according 2267 to your build configuration. 2268 2269The contents of ``libhello/hellolib.h`` are:: 2270 2271 #ifndef HELLOLIB_H 2272 #define HELLOLIB_H 2273 2274 void Hello(); 2275 2276 #endif 2277 2278The contents of ``libhello/hellolib.c`` are:: 2279 2280 #include <stdio.h> 2281 2282 void Hello(){ 2283 puts("Hello from a Yocto demo \n"); 2284 } 2285 2286The contents of ``sayhello/Makefile`` are:: 2287 2288 EXEC=sayhello 2289 LDFLAGS += -lhello 2290 2291 all: $(EXEC) 2292 2293 $(EXEC): sayhello.c 2294 $(CC) $< $(LDFLAGS) $(CFLAGS) -o $(EXEC) 2295 2296 clean: 2297 rm -rf $(EXEC) *.o 2298 2299The contents of ``sayhello/sayhello.c`` are:: 2300 2301 #include <hellolib.h> 2302 2303 int main(){ 2304 Hello(); 2305 return 0; 2306 } 2307 2308The contents of ``libhello_0.1.bb`` are:: 2309 2310 SUMMARY = "Hello demo library" 2311 DESCRIPTION = "Hello shared library used in Yocto demo" 2312 2313 # NOTE: Set the License according to the LICENSE file of your project 2314 # and then add LIC_FILES_CHKSUM accordingly 2315 LICENSE = "CLOSED" 2316 2317 # Assuming the branch is main 2318 # Change <username> accordingly 2319 SRC_URI = "git://github.com/<username>/libhello;branch=main;protocol=https" 2320 2321 S = "${WORKDIR}/git" 2322 2323 do_install(){ 2324 install -d ${D}${includedir} 2325 install -d ${D}${libdir} 2326 2327 install hellolib.h ${D}${includedir} 2328 oe_soinstall ${PN}.so.${PV} ${D}${libdir} 2329 } 2330 2331The contents of ``sayhello_0.1.bb`` are:: 2332 2333 SUMMARY = "SayHello demo" 2334 DESCRIPTION = "SayHello project used in Yocto demo" 2335 2336 # NOTE: Set the License according to the LICENSE file of your project 2337 # and then add LIC_FILES_CHKSUM accordingly 2338 LICENSE = "CLOSED" 2339 2340 # Assuming the branch is main 2341 # Change <username> accordingly 2342 SRC_URI = "git://github.com/<username>/sayhello;branch=main;protocol=https" 2343 2344 DEPENDS += "libhello" 2345 RDEPENDS:${PN} += "libhello" 2346 2347 S = "${WORKDIR}/git" 2348 2349 do_install(){ 2350 install -d ${D}/usr/bin 2351 install -m 0700 sayhello ${D}/usr/bin 2352 } 2353 2354After placing the recipes in a custom layer we can run ``bitbake sayhello`` 2355to build the recipe. 2356 2357The following diagram shows the sequences of tasks that BitBake 2358executes to accomplish that. 2359 2360.. image:: svg/bitbake_tasks_map.* 2361 :width: 100% 2362