1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3Building 4******** 5 6This section describes various build procedures, such as the steps 7needed for a simple build, building a target for multiple configurations, 8generating an image for more than one machine, and so forth. 9 10Building a Simple Image 11======================= 12 13In the development environment, you need to build an image whenever you 14change hardware support, add or change system libraries, or add or 15change services that have dependencies. There are several methods that allow 16you to build an image within the Yocto Project. This section presents 17the basic steps you need to build a simple image using BitBake from a 18build host running Linux. 19 20.. note:: 21 22 - For information on how to build an image using 23 :term:`Toaster`, see the 24 :doc:`/toaster-manual/index`. 25 26 - For information on how to use ``devtool`` to build images, see the 27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`" 28 section in the Yocto Project Application Development and the 29 Extensible Software Development Kit (eSDK) manual. 30 31 - For a quick example on how to build an image using the 32 OpenEmbedded build system, see the 33 :doc:`/brief-yoctoprojectqs/index` document. 34 35 - You can also use the `Yocto Project BitBake 36 <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__ 37 extension for Visual Studio Code to build images. 38 39The build process creates an entire Linux distribution from source and 40places it in your :term:`Build Directory` under ``tmp/deploy/images``. For 41detailed information on the build process using BitBake, see the 42":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview 43and Concepts Manual. 44 45The following figure and list overviews the build process: 46 47.. image:: figures/bitbake-build-flow.png 48 :width: 100% 49 50#. *Set up Your Host Development System to Support Development Using the 51 Yocto Project*: See the ":doc:`start`" section for options on how to get a 52 build host ready to use the Yocto Project. 53 54#. *Initialize the Build Environment:* Initialize the build environment 55 by sourcing the build environment script (i.e. 56 :ref:`structure-core-script`):: 57 58 $ source oe-init-build-env [build_dir] 59 60 When you use the initialization script, the OpenEmbedded build system 61 uses ``build`` as the default :term:`Build Directory` in your current work 62 directory. You can use a `build_dir` argument with the script to 63 specify a different :term:`Build Directory`. 64 65 .. note:: 66 67 A common practice is to use a different :term:`Build Directory` for 68 different targets; for example, ``~/build/x86`` for a ``qemux86`` 69 target, and ``~/build/arm`` for a ``qemuarm`` target. In any 70 event, it's typically cleaner to locate the :term:`Build Directory` 71 somewhere outside of your source directory. 72 73#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the 74 ``conf/local.conf`` configuration file, which is found in the 75 :term:`Build Directory`, is set up how you want it. This file defines many 76 aspects of the build environment including the target machine architecture 77 through the :term:`MACHINE` variable, the packaging format used during 78 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download 79 directory through the :term:`DL_DIR` variable. 80 81#. *Build the Image:* Build the image using the ``bitbake`` command:: 82 83 $ bitbake target 84 85 .. note:: 86 87 For information on BitBake, see the :doc:`bitbake:index`. 88 89 The target is the name of the recipe you want to build. Common 90 targets are the images in ``meta/recipes-core/images``, 91 ``meta/recipes-sato/images``, and so forth all found in the 92 :term:`Source Directory`. Alternatively, the target 93 can be the name of a recipe for a specific piece of software such as 94 BusyBox. For more details about the images the OpenEmbedded build 95 system supports, see the 96 ":ref:`ref-manual/images:Images`" chapter in the Yocto 97 Project Reference Manual. 98 99 As an example, the following command builds the 100 ``core-image-minimal`` image:: 101 102 $ bitbake core-image-minimal 103 104 Once an 105 image has been built, it often needs to be installed. The images and 106 kernels built by the OpenEmbedded build system are placed in the 107 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to 108 run pre-built images such as ``qemux86`` and ``qemuarm``, see the 109 :doc:`/sdk-manual/index` manual. For 110 information about how to install these images, see the documentation 111 for your particular board or machine. 112 113Building Images for Multiple Targets Using Multiple Configurations 114================================================================== 115 116See the :doc:`multiconfig` section of the Yocto Project Development Tasks 117Manual. 118 119Building an Initial RAM Filesystem (Initramfs) Image 120==================================================== 121 122An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root 123filesystem used for early system initialization, typically providing tools and 124loading modules needed to locate and mount the final root filesystem. 125 126Follow these steps to create an :term:`Initramfs` image: 127 128#. *Create the Initramfs Image Recipe:* You can reference the 129 ``core-image-minimal-initramfs.bb`` recipe found in the 130 ``meta/recipes-core`` directory of the :term:`Source Directory` 131 as an example from which to work. The ``core-image-minimal-initramfs`` recipe 132 is based on the :ref:`initramfs-framework <dev-manual/building:Customizing an 133 Initramfs using \`\`initramfs-framework\`\`>` recipe described below. 134 135#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel 136 Image:* If you want the :term:`Initramfs` image that is built to be bundled 137 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE` 138 variable to ``"1"`` in your ``local.conf`` configuration file and set the 139 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image. 140 141 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs` 142 image to be unpacked into the ``${B}/usr/`` directory. The unpacked 143 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the 144 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs` 145 image to be built into the kernel normally. 146 147#. *Optionally Add Items to the Initramfs Image Through the Initramfs 148 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its 149 recipe, you should use :term:`PACKAGE_INSTALL` rather than 150 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of 151 what is added to the image as compared to the defaults you might not 152 necessarily want that are set by the :ref:`ref-classes-image` 153 or :ref:`ref-classes-core-image` classes. 154 155#. *Build the Kernel Image and the Initramfs Image:* Build your kernel 156 image using BitBake. Because the :term:`Initramfs` image recipe is a 157 dependency of the kernel image, the :term:`Initramfs` image is built as well 158 and bundled with the kernel image if you used the 159 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier. 160 161Customizing an Initramfs using ``initramfs-framework`` 162------------------------------------------------------ 163 164The ``core-image-minimal-initramfs.bb`` recipe found in 165:oe_git:`meta/recipes-core/images 166</openembedded-core/tree/meta/recipes-core/images>` uses the 167:oe_git:`initramfs-framework_1.0.bb 168</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework_1.0.bb>` 169recipe as its base component. The goal of the ``initramfs-framework`` recipe is 170to provide the building blocks to build a customized :term:`Initramfs`. 171 172The ``initramfs-framework`` recipe relies on shell initialization scripts 173defined in :oe_git:`meta/recipes-core/initrdscripts/initramfs-framework 174</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework>`. Since some of 175these scripts do not apply for all use cases, the ``initramfs-framework`` recipe 176defines different packages: 177 178- ``initramfs-framework-base``: this package installs the basic components of 179 an :term:`Initramfs`, such as the ``init`` script or the ``/dev/console`` 180 character special file. As this package is a runtime dependency of all 181 modules listed below, it is automatically pulled in when one of the modules 182 is installed in the image. 183- ``initramfs-module-exec``: support for execution of applications. 184- ``initramfs-module-mdev``: support for `mdev 185 <https://wiki.gentoo.org/wiki/Mdev>`__. 186- ``initramfs-module-udev``: support for :wikipedia:`Udev <Udev>`. 187- ``initramfs-module-e2fs``: support for :wikipedia:`ext4/ext3/ext2 188 <Extended_file_system>` filesystems. 189- ``initramfs-module-nfsrootfs``: support for locating and mounting the root 190 partition via :wikipedia:`NFS <Network_File_System>`. 191- ``initramfs-module-rootfs``: support for locating and mounting the root 192 partition. 193- ``initramfs-module-debug``: dynamic debug support. 194- ``initramfs-module-lvm``: :wikipedia:`LVM <Logical_volume_management>` rootfs support. 195- ``initramfs-module-overlayroot``: support for mounting a read-write overlay 196 on top of a read-only root filesystem. 197 198In addition to the packages defined by the ``initramfs-framework`` recipe 199itself, the following packages are defined by the recipes present in 200:oe_git:`meta/recipes-core/initrdscripts </openembedded-core/tree/meta/recipes-core/initrdscripts>`: 201 202- ``initramfs-module-install``: module to create and install a partition layout 203 on a selected block device. 204- ``initramfs-module-install-efi``: module to create and install an EFI 205 partition layout on a selected block device. 206- ``initramfs-module-setup-live``: module to start a shell in the 207 :term:`Initramfs` if ``root=/dev/ram0`` in passed in the `Kernel command-line 208 <https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html>`__ 209 or the ``root=`` parameter was not passed. 210 211To customize the :term:`Initramfs`, you can add or remove packages listed 212earlier from the :term:`PACKAGE_INSTALL` variable with a :ref:`bbappend 213<dev-manual/layers:Appending Other Layers Metadata With Your Layer>` on the 214``core-image-minimal-initramfs`` recipe, or create a custom recipe for the 215:term:`Initramfs` taking ``core-image-minimal-initramfs`` as example. 216 217Custom scripts can be added to the :term:`Initramfs` by writing your own 218recipes. The recipes are conventionally named ``initramfs-module-<module name>`` 219where ``<module name>`` is the name of the module. The recipe should set its 220:term:`RDEPENDS` package-specific variables to include 221``initramfs-framework-base`` and the other packages on which the module depends 222at runtime. 223 224The recipe must install shell initialization scripts in :term:`${D} <D>`\ 225``/init.d`` and must follow the ``<number>-<script name>`` naming scheme where: 226 227- ``<number>`` is a *two-digit* number that affects the execution order of the 228 script compared to others. For example, the script ``80-setup-live`` would be 229 executed after ``01-udev`` because 80 is greater than 01. 230 231 This number being two-digits is important here as the scripts are executed 232 alphabetically. For example, the script ``10-script`` would be executed 233 before the script ``8-script``, because ``1`` is inferior to ``8``. 234 Therefore, the script should be named ``08-script``. 235 236- ``<script name>`` is the script name which you can choose freely. 237 238 If two script use the same ``<number>``, they are sorted alphabetically based 239 on ``<script name>``. 240 241Bundling an Initramfs Image From a Separate Multiconfig 242------------------------------------------------------- 243 244There may be a case where we want to build an :term:`Initramfs` image which does not 245inherit the same distro policy as our main image, for example, we may want 246our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs` 247image to keep a smaller footprint. However, by performing the steps mentioned 248above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us 249to override it. 250 251To achieve this, you need to perform some additional steps: 252 253#. *Create a multiconfig for your Initramfs image:* You can perform the steps 254 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig. 255 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and 256 contains the variables:: 257 258 TMPDIR="${TOPDIR}/tmp-initramfscfg" 259 TCLIBC="musl" 260 261#. *Set additional Initramfs variables on your main configuration:* 262 Additionally, on your main configuration (``local.conf``) you need to set the 263 variables:: 264 265 INITRAMFS_MULTICONFIG = "initramfscfg" 266 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}" 267 268 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE` 269 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE` 270 to be built coming from the ``initramfscfg`` multiconfig, and to let the 271 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located. 272 273 Building a system with such configuration will build the kernel using the 274 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the 275 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE` 276 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel 277 but a glibc based main image. 278 279 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE` 280 or to build a different :term:`DISTRO` for it such as ``poky-tiny``. 281 282 283Building a Tiny System 284====================== 285 286Very small distributions have some significant advantages such as 287requiring less on-die or in-package memory (cheaper), better performance 288through efficient cache usage, lower power requirements due to less 289memory, faster boot times, and reduced development overhead. Some 290real-world examples where a very small distribution gives you distinct 291advantages are digital cameras, medical devices, and small headless 292systems. 293 294This section presents information that shows you how you can trim your 295distribution to even smaller sizes than the ``poky-tiny`` distribution, 296which is around 5 Mbytes, that can be built out-of-the-box using the 297Yocto Project. 298 299Tiny System Overview 300-------------------- 301 302The following list presents the overall steps you need to consider and 303perform to create distributions with smaller root filesystems, achieve 304faster boot times, maintain your critical functionality, and avoid 305initial RAM disks: 306 307- :ref:`Determine your goals and guiding principles 308 <dev-manual/building:goals and guiding principles>` 309 310- :ref:`dev-manual/building:understand what contributes to your image size` 311 312- :ref:`Reduce the size of the root filesystem 313 <dev-manual/building:trim the root filesystem>` 314 315- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>` 316 317- :ref:`dev-manual/building:remove package management requirements` 318 319- :ref:`dev-manual/building:look for other ways to minimize size` 320 321- :ref:`dev-manual/building:iterate on the process` 322 323Goals and Guiding Principles 324---------------------------- 325 326Before you can reach your destination, you need to know where you are 327going. Here is an example list that you can use as a guide when creating 328very small distributions: 329 330- Determine how much space you need (e.g. a kernel that is 1 Mbyte or 331 less and a root filesystem that is 3 Mbytes or less). 332 333- Find the areas that are currently taking 90% of the space and 334 concentrate on reducing those areas. 335 336- Do not create any difficult "hacks" to achieve your goals. 337 338- Leverage the device-specific options. 339 340- Work in a separate layer so that you keep changes isolated. For 341 information on how to create layers, see the 342 ":ref:`dev-manual/layers:understanding and creating layers`" section. 343 344Understand What Contributes to Your Image Size 345---------------------------------------------- 346 347It is easiest to have something to start with when creating your own 348distribution. You can use the Yocto Project out-of-the-box to create the 349``poky-tiny`` distribution. Ultimately, you will want to make changes in 350your own distribution that are likely modeled after ``poky-tiny``. 351 352.. note:: 353 354 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your 355 ``local.conf`` file to "poky-tiny" as described in the 356 ":ref:`dev-manual/custom-distribution:creating your own distribution`" 357 section. 358 359Understanding some memory concepts will help you reduce the system size. 360Memory consists of static, dynamic, and temporary memory. Static memory 361is the TEXT (code), DATA (initialized data in the code), and BSS 362(uninitialized data) sections. Dynamic memory represents memory that is 363allocated at runtime: stacks, hash tables, and so forth. Temporary 364memory is recovered after the boot process. This memory consists of 365memory used for decompressing the kernel and for the ``__init__`` 366functions. 367 368To help you see where you currently are with kernel and root filesystem 369sizes, you can use two tools found in the :term:`Source Directory` 370in the 371``scripts/tiny/`` directory: 372 373- ``ksize.py``: Reports component sizes for the kernel build objects. 374 375- ``dirsize.py``: Reports component sizes for the root filesystem. 376 377This next tool and command help you organize configuration fragments and 378view file dependencies in a human-readable form: 379 380- ``merge_config.sh``: Helps you manage configuration files and 381 fragments within the kernel. With this tool, you can merge individual 382 configuration fragments together. The tool allows you to make 383 overrides and warns you of any missing configuration options. The 384 tool is ideal for allowing you to iterate on configurations, create 385 minimal configurations, and create configuration files for different 386 machines without having to duplicate your process. 387 388 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git 389 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``, 390 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig`` 391 directory. 392 393 For more information on configuration fragments, see the 394 ":ref:`kernel-dev/common:creating configuration fragments`" 395 section in the Yocto Project Linux Kernel Development Manual. 396 397- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command 398 with these options brings up a Dependency Explorer from which you can 399 view file dependencies. Understanding these dependencies allows you 400 to make informed decisions when cutting out various pieces of the 401 kernel and root filesystem. 402 403Trim the Root Filesystem 404------------------------ 405 406The root filesystem is made up of packages for booting, libraries, and 407applications. To change things, you can configure how the packaging 408happens, which changes the way you build them. You can also modify the 409filesystem itself or select a different filesystem. 410 411First, find out what is hogging your root filesystem by running the 412``dirsize.py`` script from your root directory:: 413 414 $ cd root-directory-of-image 415 $ dirsize.py 100000 > dirsize-100k.log 416 $ cat dirsize-100k.log 417 418You can apply a filter to the script to ignore files 419under a certain size. The previous example filters out any files below 420100 Kbytes. The sizes reported by the tool are uncompressed, and thus 421will be smaller by a relatively constant factor in a compressed root 422filesystem. When you examine your log file, you can focus on areas of 423the root filesystem that take up large amounts of memory. 424 425You need to be sure that what you eliminate does not cripple the 426functionality you need. One way to see how packages relate to each other 427is by using the Dependency Explorer UI with the BitBake command:: 428 429 $ cd image-directory 430 $ bitbake -u taskexp -g image 431 432Use the interface to 433select potential packages you wish to eliminate and see their dependency 434relationships. 435 436When deciding how to reduce the size, get rid of packages that result in 437minimal impact on the feature set. For example, you might not need a VGA 438display. Or, you might be able to get by with ``devtmpfs`` and ``mdev`` 439instead of ``udev``. 440 441Use your ``local.conf`` file to make changes. For example, to eliminate 442``udev`` and ``glib``, set the following in the local configuration 443file:: 444 445 VIRTUAL-RUNTIME_dev_manager = "" 446 447Finally, you should consider exactly the type of root filesystem you 448need to meet your needs while also reducing its size. For example, 449consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an 450:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1 451Mbyte journal. If you are okay with running read-only, you do not need 452this journal. 453 454.. note:: 455 456 After each round of elimination, you need to rebuild your system and 457 then use the tools to see the effects of your reductions. 458 459Trim the Kernel 460--------------- 461 462The kernel is built by including policies for hardware-independent 463aspects. What subsystems do you enable? For what architecture are you 464building? Which drivers do you build by default? 465 466.. note:: 467 468 You can modify the kernel source if you want to help with boot time. 469 470Run the ``ksize.py`` script from the top-level Linux build directory to 471get an idea of what is making up the kernel:: 472 473 $ cd top-level-linux-build-directory 474 $ ksize.py > ksize.log 475 $ cat ksize.log 476 477When you examine the log, you will see how much space is taken up with 478the built-in ``.o`` files for drivers, networking, core kernel files, 479filesystem, sound, and so forth. The sizes reported by the tool are 480uncompressed, and thus will be smaller by a relatively constant factor 481in a compressed kernel image. Look to reduce the areas that are large 482and taking up around the "90% rule." 483 484To examine, or drill down, into any particular area, use the ``-d`` 485option with the script:: 486 487 $ ksize.py -d > ksize.log 488 489Using this option 490breaks out the individual file information for each area of the kernel 491(e.g. drivers, networking, and so forth). 492 493Use your log file to see what you can eliminate from the kernel based on 494features you can let go. For example, if you are not going to need 495sound, you do not need any drivers that support sound. 496 497After figuring out what to eliminate, you need to reconfigure the kernel 498to reflect those changes during the next build. You could run 499``menuconfig`` and make all your changes at once. However, that makes it 500difficult to see the effects of your individual eliminations and also 501makes it difficult to replicate the changes for perhaps another target 502device. A better method is to start with no configurations using 503``allnoconfig``, create configuration fragments for individual changes, 504and then manage the fragments into a single configuration file using 505``merge_config.sh``. The tool makes it easy for you to iterate using the 506configuration change and build cycle. 507 508Each time you make configuration changes, you need to rebuild the kernel 509and check to see what impact your changes had on the overall size. 510 511Remove Package Management Requirements 512-------------------------------------- 513 514Packaging requirements add size to the image. One way to reduce the size 515of the image is to remove all the packaging requirements from the image. 516This reduction includes both removing the package manager and its unique 517dependencies as well as removing the package management data itself. 518 519To eliminate all the packaging requirements for an image, be sure that 520"package-management" is not part of your 521:term:`IMAGE_FEATURES` 522statement for the image. When you remove this feature, you are removing 523the package manager as well as its dependencies from the root 524filesystem. 525 526Look for Other Ways to Minimize Size 527------------------------------------ 528 529Depending on your particular circumstances, other areas that you can 530trim likely exist. The key to finding these areas is through tools and 531methods described here combined with experimentation and iteration. Here 532are a couple of areas to experiment with: 533 534- ``glibc``: In general, follow this process: 535 536 #. Remove ``glibc`` features from 537 :term:`DISTRO_FEATURES` 538 that you think you do not need. 539 540 #. Build your distribution. 541 542 #. If the build fails due to missing symbols in a package, determine 543 if you can reconfigure the package to not need those features. For 544 example, change the configuration to not support wide character 545 support as is done for ``ncurses``. Or, if support for those 546 characters is needed, determine what ``glibc`` features provide 547 the support and restore the configuration. 548 549 4. Rebuild and repeat the process. 550 551- ``busybox``: For BusyBox, use a process similar as described for 552 ``glibc``. A difference is you will need to boot the resulting system 553 to see if you are able to do everything you expect from the running 554 system. You need to be sure to integrate configuration fragments into 555 Busybox because BusyBox handles its own core features and then allows 556 you to add configuration fragments on top. 557 558Iterate on the Process 559---------------------- 560 561If you have not reached your goals on system size, you need to iterate 562on the process. The process is the same. Use the tools and see just what 563is taking up 90% of the root filesystem and the kernel. Decide what you 564can eliminate without limiting your device beyond what you need. 565 566Depending on your system, a good place to look might be Busybox, which 567provides a stripped down version of Unix tools in a single, executable 568file. You might be able to drop virtual terminal services or perhaps 569ipv6. 570 571Building Images for More than One Machine 572========================================= 573 574A common scenario developers face is creating images for several 575different machines that use the same software environment. In this 576situation, it is tempting to set the tunings and optimization flags for 577each build specifically for the targeted hardware (i.e. "maxing out" the 578tunings). Doing so can considerably add to build times and package feed 579maintenance collectively for the machines. For example, selecting tunes 580that are extremely specific to a CPU core used in a system might enable 581some micro optimizations in GCC for that particular system but would 582otherwise not gain you much of a performance difference across the other 583systems as compared to using a more general tuning across all the builds 584(e.g. setting :term:`DEFAULTTUNE` 585specifically for each machine's build). Rather than "max out" each 586build's tunings, you can take steps that cause the OpenEmbedded build 587system to reuse software across the various machines where it makes 588sense. 589 590If build speed and package feed maintenance are considerations, you 591should consider the points in this section that can help you optimize 592your tunings to best consider build times and package feed maintenance. 593 594- *Share the :term:`Build Directory`:* If at all possible, share the 595 :term:`TMPDIR` across builds. The Yocto Project supports switching between 596 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice 597 is well supported and regularly used by developers when building for 598 multiple machines. When you use the same :term:`TMPDIR` for multiple 599 machine builds, the OpenEmbedded build system can reuse the existing native 600 and often cross-recipes for multiple machines. Thus, build time decreases. 601 602 .. note:: 603 604 If :term:`DISTRO` settings change or fundamental configuration settings 605 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`. 606 Sharing :term:`TMPDIR` under these circumstances might work but since it is 607 not guaranteed, you should use a clean :term:`TMPDIR`. 608 609- *Enable the Appropriate Package Architecture:* By default, the 610 OpenEmbedded build system enables three levels of package 611 architectures: "all", "tune" or "package", and "machine". Any given 612 recipe usually selects one of these package architectures (types) for 613 its output. Depending for what a given recipe creates packages, 614 making sure you enable the appropriate package architecture can 615 directly impact the build time. 616 617 A recipe that just generates scripts can enable "all" architecture 618 because there are no binaries to build. To specifically enable "all" 619 architecture, be sure your recipe inherits the 620 :ref:`ref-classes-allarch` class. 621 This class is useful for "all" architectures because it configures 622 many variables so packages can be used across multiple architectures. 623 624 If your recipe needs to generate packages that are machine-specific 625 or when one of the build or runtime dependencies is already 626 machine-architecture dependent, which makes your recipe also 627 machine-architecture dependent, make sure your recipe enables the 628 "machine" package architecture through the 629 :term:`MACHINE_ARCH` 630 variable:: 631 632 PACKAGE_ARCH = "${MACHINE_ARCH}" 633 634 When you do not 635 specifically enable a package architecture through the 636 :term:`PACKAGE_ARCH`, The 637 OpenEmbedded build system defaults to the 638 :term:`TUNE_PKGARCH` setting:: 639 640 PACKAGE_ARCH = "${TUNE_PKGARCH}" 641 642- *Choose a Generic Tuning File if Possible:* Some tunes are more 643 generic and can run on multiple targets (e.g. an ``armv5`` set of 644 packages could run on ``armv6`` and ``armv7`` processors in most 645 cases). Similarly, ``i486`` binaries could work on ``i586`` and 646 higher processors. You should realize, however, that advances on 647 newer processor versions would not be used. 648 649 If you select the same tune for several different machines, the 650 OpenEmbedded build system reuses software previously built, thus 651 speeding up the overall build time. Realize that even though a new 652 sysroot for each machine is generated, the software is not recompiled 653 and only one package feed exists. 654 655- *Manage Granular Level Packaging:* Sometimes there are cases where 656 injecting another level of package architecture beyond the three 657 higher levels noted earlier can be useful. For example, consider how 658 NXP (formerly Freescale) allows for the easy reuse of binary packages 659 in their layer 660 :yocto_git:`meta-freescale </meta-freescale/>`. 661 In this example, the 662 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>` 663 class shares GPU packages for i.MX53 boards because all boards share 664 the AMD GPU. The i.MX6-based boards can do the same because all 665 boards share the Vivante GPU. This class inspects the BitBake 666 datastore to identify if the package provides or depends on one of 667 the sub-architecture values. If so, the class sets the 668 :term:`PACKAGE_ARCH` value 669 based on the ``MACHINE_SUBARCH`` value. If the package does not 670 provide or depend on one of the sub-architecture values but it 671 matches a value in the machine-specific filter, it sets 672 :term:`MACHINE_ARCH`. This 673 behavior reduces the number of packages built and saves build time by 674 reusing binaries. 675 676- *Use Tools to Debug Issues:* Sometimes you can run into situations 677 where software is being rebuilt when you think it should not be. For 678 example, the OpenEmbedded build system might not be using shared 679 state between machines when you think it should be. These types of 680 situations are usually due to references to machine-specific 681 variables such as :term:`MACHINE`, 682 :term:`SERIAL_CONSOLES`, 683 :term:`XSERVER`, 684 :term:`MACHINE_FEATURES`, 685 and so forth in code that is supposed to only be tune-specific or 686 when the recipe depends 687 (:term:`DEPENDS`, 688 :term:`RDEPENDS`, 689 :term:`RRECOMMENDS`, 690 :term:`RSUGGESTS`, and so forth) 691 on some other recipe that already has 692 :term:`PACKAGE_ARCH` defined 693 as "${MACHINE_ARCH}". 694 695 .. note:: 696 697 Patches to fix any issues identified are most welcome as these 698 issues occasionally do occur. 699 700 For such cases, you can use some tools to help you sort out the 701 situation: 702 703 - ``state-diff-machines.sh``*:* You can find this tool in the 704 ``scripts`` directory of the Source Repositories. See the comments 705 in the script for information on how to use the tool. 706 707 - *BitBake's "-S printdiff" Option:* Using this option causes 708 BitBake to try to establish the most recent signature match 709 (e.g. in the shared state cache) and then compare matched signatures 710 to determine the stamps and delta where these two stamp trees diverge. 711 712Building Software from an External Source 713========================================= 714 715By default, the OpenEmbedded build system uses the :term:`Build Directory` 716when building source code. The build process involves fetching the source 717files, unpacking them, and then patching them if necessary before the build 718takes place. 719 720There are situations where you might want to build software from source 721files that are external to and thus outside of the OpenEmbedded build 722system. For example, suppose you have a project that includes a new BSP 723with a heavily customized kernel. And, you want to minimize exposing the 724build system to the development team so that they can focus on their 725project and maintain everyone's workflow as much as possible. In this 726case, you want a kernel source directory on the development machine 727where the development occurs. You want the recipe's 728:term:`SRC_URI` variable to point to 729the external directory and use it as is, not copy it. 730 731To build from software that comes from an external source, all you need to do 732is inherit the :ref:`ref-classes-externalsrc` class and then set 733the :term:`EXTERNALSRC` variable to point to your external source code. Here 734are the statements to put in your ``local.conf`` file:: 735 736 INHERIT += "externalsrc" 737 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree" 738 739This next example shows how to accomplish the same thing by setting 740:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file:: 741 742 EXTERNALSRC = "path" 743 EXTERNALSRC_BUILD = "path" 744 745.. note:: 746 747 In order for these settings to take effect, you must globally or 748 locally inherit the :ref:`ref-classes-externalsrc` class. 749 750By default, :ref:`ref-classes-externalsrc` builds the source code in a 751directory separate from the external source directory as specified by 752:term:`EXTERNALSRC`. If you need 753to have the source built in the same directory in which it resides, or 754some other nominated directory, you can set 755:term:`EXTERNALSRC_BUILD` 756to point to that directory:: 757 758 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree" 759 760Replicating a Build Offline 761=========================== 762 763It can be useful to take a "snapshot" of upstream sources used in a 764build and then use that "snapshot" later to replicate the build offline. 765To do so, you need to first prepare and populate your downloads 766directory your "snapshot" of files. Once your downloads directory is 767ready, you can use it at any time and from any machine to replicate your 768build. 769 770Follow these steps to populate your Downloads directory: 771 772#. *Create a Clean Downloads Directory:* Start with an empty downloads 773 directory (:term:`DL_DIR`). You 774 start with an empty downloads directory by either removing the files 775 in the existing directory or by setting :term:`DL_DIR` to point to either 776 an empty location or one that does not yet exist. 777 778#. *Generate Tarballs of the Source Git Repositories:* Edit your 779 ``local.conf`` configuration file as follows:: 780 781 DL_DIR = "/home/your-download-dir/" 782 BB_GENERATE_MIRROR_TARBALLS = "1" 783 784 During 785 the fetch process in the next step, BitBake gathers the source files 786 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See 787 the 788 :term:`BB_GENERATE_MIRROR_TARBALLS` 789 variable for more information. 790 791#. *Populate Your Downloads Directory Without Building:* Use BitBake to 792 fetch your sources but inhibit the build:: 793 794 $ bitbake target --runonly=fetch 795 796 The downloads directory (i.e. ``${DL_DIR}``) now has 797 a "snapshot" of the source files in the form of tarballs, which can 798 be used for the build. 799 800#. *Optionally Remove Any Git or other SCM Subdirectories From the 801 Downloads Directory:* If you want, you can clean up your downloads 802 directory by removing any Git or other Source Control Management 803 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs 804 already contain these subdirectories. 805 806Once your downloads directory has everything it needs regarding source 807files, you can create your "own-mirror" and build your target. 808Understand that you can use the files to build the target offline from 809any machine and at any time. 810 811Follow these steps to build your target using the files in the downloads 812directory: 813 814#. *Using Local Files Only:* Inside your ``local.conf`` file, add the 815 :term:`SOURCE_MIRROR_URL` variable, inherit the 816 :ref:`ref-classes-own-mirrors` class, and use the 817 :term:`BB_NO_NETWORK` variable to your ``local.conf``:: 818 819 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/" 820 INHERIT += "own-mirrors" 821 BB_NO_NETWORK = "1" 822 823 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors` 824 class set up the system to use the downloads directory as your "own 825 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that 826 BitBake's fetching process in step 3 stays local, which means files 827 from your "own-mirror" are used. 828 829#. *Start With a Clean Build:* You can start with a clean build by 830 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new 831 :term:`Build Directory`. 832 833#. *Build Your Target:* Use BitBake to build your target:: 834 835 $ bitbake target 836 837 The build completes using the known local "snapshot" of source 838 files from your mirror. The resulting tarballs for your "snapshot" of 839 source files are in the downloads directory. 840 841 .. note:: 842 843 The offline build does not work if recipes attempt to find the 844 latest version of software by setting 845 :term:`SRCREV` to 846 ``${``\ :term:`AUTOREV`\ ``}``:: 847 848 SRCREV = "${AUTOREV}" 849 850 When a recipe sets :term:`SRCREV` to 851 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an 852 attempt to determine the latest version of software from the SCM. 853 Typically, recipes that use :term:`AUTOREV` are custom or modified 854 recipes. Recipes that reside in public repositories usually do not 855 use :term:`AUTOREV`. 856 857 If you do have recipes that use :term:`AUTOREV`, you can take steps to 858 still use the recipes in an offline build. Do the following: 859 860 #. Use a configuration generated by enabling :ref:`build 861 history <dev-manual/build-quality:maintaining build output quality>`. 862 863 #. Use the ``buildhistory-collect-srcrevs`` command to collect the 864 stored :term:`SRCREV` values from the build's history. For more 865 information on collecting these values, see the 866 ":ref:`dev-manual/build-quality:build history package information`" 867 section. 868 869 #. Once you have the correct source revisions, you can modify 870 those recipes to set :term:`SRCREV` to specific versions of the 871 software. 872 873
