1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3Maintaining Build Output Quality 4******************************** 5 6Many factors can influence the quality of a build. For example, if you 7upgrade a recipe to use a new version of an upstream software package or 8you experiment with some new configuration options, subtle changes can 9occur that you might not detect until later. Consider the case where 10your recipe is using a newer version of an upstream package. In this 11case, a new version of a piece of software might introduce an optional 12dependency on another library, which is auto-detected. If that library 13has already been built when the software is building, the software will 14link to the built library and that library will be pulled into your 15image along with the new software even if you did not want the library. 16 17The :ref:`ref-classes-buildhistory` class helps you maintain the quality of 18your build output. You can use the class to highlight unexpected and possibly 19unwanted changes in the build output. When you enable build history, it records 20information about the contents of each package and image and then commits that 21information to a local Git repository where you can examine the information. 22 23The remainder of this section describes the following: 24 25- :ref:`How you can enable and disable build history <dev-manual/build-quality:enabling and disabling build history>` 26 27- :ref:`How to understand what the build history contains <dev-manual/build-quality:understanding what the build history contains>` 28 29- :ref:`How to limit the information used for build history <dev-manual/build-quality:using build history to gather image information only>` 30 31- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/build-quality:examining build history information>` 32 33Enabling and Disabling Build History 34==================================== 35 36Build history is disabled by default. To enable it, add the following 37:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to 38"1" at the end of your ``conf/local.conf`` file found in the 39:term:`Build Directory`:: 40 41 INHERIT += "buildhistory" 42 BUILDHISTORY_COMMIT = "1" 43 44Enabling build history as 45previously described causes the OpenEmbedded build system to collect 46build output information and commit it as a single commit to a local 47:ref:`overview-manual/development-environment:git` repository. 48 49.. note:: 50 51 Enabling build history increases your build times slightly, 52 particularly for images, and increases the amount of disk space used 53 during the build. 54 55You can disable build history by removing the previous statements from 56your ``conf/local.conf`` file. 57 58Understanding What the Build History Contains 59============================================= 60 61Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory`` 62in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR` 63variable. Here is an example abbreviated listing: 64 65.. image:: figures/buildhistory.png 66 :align: center 67 :width: 50% 68 69At the top level, there is a ``metadata-revs`` file that lists the 70revisions of the repositories for the enabled layers when the build was 71produced. The rest of the data splits into separate ``packages``, 72``images`` and ``sdk`` directories, the contents of which are described 73as follows. 74 75Build History Package Information 76--------------------------------- 77 78The history for each package contains a text file that has name-value 79pairs with information about the package. For example, 80``buildhistory/packages/i586-poky-linux/busybox/busybox/latest`` 81contains the following: 82 83.. code-block:: none 84 85 PV = 1.22.1 86 PR = r32 87 RPROVIDES = 88 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg 89 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d 90 PKGSIZE = 540168 91 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ 92 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ 93 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ 94 /usr/share/pixmaps /usr/share/applications /usr/share/idl \ 95 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers 96 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ 97 /etc/busybox.links.nosuid /etc/busybox.links.suid 98 99Most of these 100name-value pairs correspond to variables used to produce the package. 101The exceptions are ``FILELIST``, which is the actual list of files in 102the package, and ``PKGSIZE``, which is the total size of files in the 103package in bytes. 104 105There is also a file that corresponds to the recipe from which the package 106came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``): 107 108.. code-block:: none 109 110 PV = 1.22.1 111 PR = r32 112 DEPENDS = initscripts kern-tools-native update-rc.d-native \ 113 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ 114 virtual/libc virtual/update-alternatives 115 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ 116 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ 117 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox 118 119Finally, for those recipes fetched from a version control system (e.g., 120Git), there is a file that lists source revisions that are specified in 121the recipe and the actual revisions used during the build. Listed 122and actual revisions might differ when 123:term:`SRCREV` is set to 124${:term:`AUTOREV`}. Here is an 125example assuming 126``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):: 127 128 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" 129 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" 130 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" 131 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f" 132 133You can use the 134``buildhistory-collect-srcrevs`` command with the ``-a`` option to 135collect the stored :term:`SRCREV` values from build history and report them 136in a format suitable for use in global configuration (e.g., 137``local.conf`` or a distro include file) to override floating 138:term:`AUTOREV` values to a fixed set of revisions. Here is some example 139output from this command:: 140 141 $ buildhistory-collect-srcrevs -a 142 # all-poky-linux 143 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8" 144 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac" 145 # core2-64-poky-linux 146 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66" 147 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8" 148 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0" 149 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0" 150 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1" 151 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c" 152 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7" 153 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee" 154 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1" 155 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100" 156 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3" 157 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf" 158 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced" 159 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8" 160 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f" 161 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa" 162 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70" 163 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3" 164 # qemux86_64-poky-linux 165 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c" 166 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0" 167 # x86_64-linux 168 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66" 169 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c" 170 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87" 171 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57" 172 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb" 173 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b" 174 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3" 175 176.. note:: 177 178 Here are some notes on using the ``buildhistory-collect-srcrevs`` command: 179 180 - By default, only values where the :term:`SRCREV` was not hardcoded 181 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a`` 182 option to see all :term:`SRCREV` values. 183 184 - The output statements might not have any effect if overrides are 185 applied elsewhere in the build system configuration. Use the 186 ``-f`` option to add the ``forcevariable`` override to each output 187 line if you need to work around this restriction. 188 189 - The script does apply special handling when building for multiple 190 machines. However, the script does place a comment before each set 191 of values that specifies which triplet to which they belong as 192 previously shown (e.g., ``i586-poky-linux``). 193 194Build History Image Information 195------------------------------- 196 197The files produced for each image are as follows: 198 199- ``image-files:`` A directory containing selected files from the root 200 filesystem. The files are defined by 201 :term:`BUILDHISTORY_IMAGE_FILES`. 202 203- ``build-id.txt:`` Human-readable information about the build 204 configuration and metadata source revisions. This file contains the 205 full build header as printed by BitBake. 206 207- ``*.dot:`` Dependency graphs for the image that are compatible with 208 ``graphviz``. 209 210- ``files-in-image.txt:`` A list of files in the image with 211 permissions, owner, group, size, and symlink information. 212 213- ``image-info.txt:`` A text file containing name-value pairs with 214 information about the image. See the following listing example for 215 more information. 216 217- ``installed-package-names.txt:`` A list of installed packages by name 218 only. 219 220- ``installed-package-sizes.txt:`` A list of installed packages ordered 221 by size. 222 223- ``installed-packages.txt:`` A list of installed packages with full 224 package filenames. 225 226.. note:: 227 228 Installed package information is able to be gathered and produced 229 even if package management is disabled for the final image. 230 231Here is an example of ``image-info.txt``: 232 233.. code-block:: none 234 235 DISTRO = poky 236 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be 237 USER_CLASSES = buildstats 238 IMAGE_CLASSES = qemuboot qemuboot license_image 239 IMAGE_FEATURES = debug-tweaks 240 IMAGE_LINGUAS = 241 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp 242 BAD_RECOMMENDATIONS = 243 NO_RECOMMENDATIONS = 244 PACKAGE_EXCLUDE = 245 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible; 246 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; 247 IMAGESIZE = 9265 248 249Other than ``IMAGESIZE``, 250which is the total size of the files in the image in Kbytes, the 251name-value pairs are variables that may have influenced the content of 252the image. This information is often useful when you are trying to 253determine why a change in the package or file listings has occurred. 254 255Using Build History to Gather Image Information Only 256---------------------------------------------------- 257 258As you can see, build history produces image information, including 259dependency graphs, so you can see why something was pulled into the 260image. If you are just interested in this information and not interested 261in collecting specific package or SDK information, you can enable 262writing only image information without any history by adding the 263following to your ``conf/local.conf`` file found in the 264:term:`Build Directory`:: 265 266 INHERIT += "buildhistory" 267 BUILDHISTORY_COMMIT = "0" 268 BUILDHISTORY_FEATURES = "image" 269 270Here, you set the 271:term:`BUILDHISTORY_FEATURES` 272variable to use the image feature only. 273 274Build History SDK Information 275----------------------------- 276 277Build history collects similar information on the contents of SDKs (e.g. 278``bitbake -c populate_sdk imagename``) as compared to information it 279collects for images. Furthermore, this information differs depending on 280whether an extensible or standard SDK is being produced. 281 282The following list shows the files produced for SDKs: 283 284- ``files-in-sdk.txt:`` A list of files in the SDK with permissions, 285 owner, group, size, and symlink information. This list includes both 286 the host and target parts of the SDK. 287 288- ``sdk-info.txt:`` A text file containing name-value pairs with 289 information about the SDK. See the following listing example for more 290 information. 291 292- ``sstate-task-sizes.txt:`` A text file containing name-value pairs 293 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot` 294 tasks have a total size). The ``sstate-task-sizes.txt`` file exists 295 only when an extensible SDK is created. 296 297- ``sstate-package-sizes.txt:`` A text file containing name-value pairs 298 with information for the shared-state packages and sizes in the SDK. 299 The ``sstate-package-sizes.txt`` file exists only when an extensible 300 SDK is created. 301 302- ``sdk-files:`` A folder that contains copies of the files mentioned 303 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output. 304 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is 305 specific to the extensible SDK although you can set it differently if 306 you would like to pull in specific files from the standard SDK. 307 308 The default files are ``conf/local.conf``, ``conf/bblayers.conf``, 309 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and 310 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get 311 copied into the ``sdk-files`` directory. 312 313- The following information appears under each of the ``host`` and 314 ``target`` directories for the portions of the SDK that run on the 315 host and on the target, respectively: 316 317 .. note:: 318 319 The following files for the most part are empty when producing an 320 extensible SDK because this type of SDK is not constructed from 321 packages as is the standard SDK. 322 323 - ``depends.dot:`` Dependency graph for the SDK that is compatible 324 with ``graphviz``. 325 326 - ``installed-package-names.txt:`` A list of installed packages by 327 name only. 328 329 - ``installed-package-sizes.txt:`` A list of installed packages 330 ordered by size. 331 332 - ``installed-packages.txt:`` A list of installed packages with full 333 package filenames. 334 335Here is an example of ``sdk-info.txt``: 336 337.. code-block:: none 338 339 DISTRO = poky 340 DISTRO_VERSION = 1.3+snapshot-20130327 341 SDK_NAME = poky-glibc-i686-arm 342 SDK_VERSION = 1.3+snapshot 343 SDKMACHINE = 344 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs 345 BAD_RECOMMENDATIONS = 346 SDKSIZE = 352712 347 348Other than ``SDKSIZE``, which is 349the total size of the files in the SDK in Kbytes, the name-value pairs 350are variables that might have influenced the content of the SDK. This 351information is often useful when you are trying to determine why a 352change in the package or file listings has occurred. 353 354Examining Build History Information 355----------------------------------- 356 357You can examine build history output from the command line or from a web 358interface. 359 360To see any changes that have occurred (assuming you have 361:term:`BUILDHISTORY_COMMIT` = "1"), 362you can simply use any Git command that allows you to view the history 363of a repository. Here is one method:: 364 365 $ git log -p 366 367You need to realize, 368however, that this method does show changes that are not significant 369(e.g. a package's size changing by a few bytes). 370 371There is a command-line tool called ``buildhistory-diff``, though, 372that queries the Git repository and prints just the differences that 373might be significant in human-readable form. Here is an example:: 374 375 $ poky/poky/scripts/buildhistory-diff . HEAD^ 376 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): 377 /etc/anotherpkg.conf was added 378 /sbin/anotherpkg was added 379 * (installed-package-names.txt): 380 * anotherpkg was added 381 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): 382 anotherpkg was added 383 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" 384 * PR changed from "r0" to "r1" 385 * PV changed from "0.1.10" to "0.1.12" 386 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) 387 * PR changed from "r0" to "r1" 388 * PV changed from "0.1.10" to "0.1.12" 389 390.. note:: 391 392 The ``buildhistory-diff`` tool requires the ``GitPython`` 393 package. Be sure to install it using Pip3 as follows:: 394 395 $ pip3 install GitPython --user 396 397 398 Alternatively, you can install ``python3-git`` using the appropriate 399 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``). 400 401To see changes to the build history using a web interface, follow the 402instruction in the ``README`` file 403:yocto_git:`here </buildhistory-web/>`. 404 405Here is a sample screenshot of the interface: 406 407.. image:: figures/buildhistory-web.png 408 :width: 100% 409 410
