1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3Working with Packages 4********************* 5 6This section describes a few tasks that involve packages: 7 8- :ref:`dev-manual/packages:excluding packages from an image` 9 10- :ref:`dev-manual/packages:incrementing a package version` 11 12- :ref:`dev-manual/packages:handling optional module packaging` 13 14- :ref:`dev-manual/packages:using runtime package management` 15 16- :ref:`dev-manual/packages:generating and using signed packages` 17 18- :ref:`Setting up and running package test 19 (ptest) <dev-manual/packages:testing packages with ptest>` 20 21- :ref:`dev-manual/packages:creating node package manager (npm) packages` 22 23- :ref:`dev-manual/packages:adding custom metadata to packages` 24 25Excluding Packages from an Image 26================================ 27 28You might find it necessary to prevent specific packages from being 29installed into an image. If so, you can use several variables to direct 30the build system to essentially ignore installing recommended packages 31or to not install a package at all. 32 33The following list introduces variables you can use to prevent packages 34from being installed into your image. Each of these variables only works 35with IPK and RPM package types, not for Debian packages. 36Also, you can use these variables from your ``local.conf`` file 37or attach them to a specific image recipe by using a recipe name 38override. For more detail on the variables, see the descriptions in the 39Yocto Project Reference Manual's glossary chapter. 40 41- :term:`BAD_RECOMMENDATIONS`: 42 Use this variable to specify "recommended-only" packages that you do 43 not want installed. 44 45- :term:`NO_RECOMMENDATIONS`: 46 Use this variable to prevent all "recommended-only" packages from 47 being installed. 48 49- :term:`PACKAGE_EXCLUDE`: 50 Use this variable to prevent specific packages from being installed 51 regardless of whether they are "recommended-only" or not. You need to 52 realize that the build process could fail with an error when you 53 prevent the installation of a package whose presence is required by 54 an installed package. 55 56Incrementing a Package Version 57============================== 58 59This section provides some background on how binary package versioning 60is accomplished and presents some of the services, variables, and 61terminology involved. 62 63In order to understand binary package versioning, you need to consider 64the following: 65 66- Binary Package: The binary package that is eventually built and 67 installed into an image. 68 69- Binary Package Version: The binary package version is composed of two 70 components --- a version and a revision. 71 72 .. note:: 73 74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved 75 but this discussion for the most part ignores :term:`PE`. 76 77 The version and revision are taken from the 78 :term:`PV` and 79 :term:`PR` variables, respectively. 80 81- :term:`PV`: The recipe version. :term:`PV` represents the version of the 82 software being packaged. Do not confuse :term:`PV` with the binary 83 package version. 84 85- :term:`PR`: The recipe revision. 86 87- :term:`SRCPV`: The OpenEmbedded 88 build system uses this string to help define the value of :term:`PV` when 89 the source code revision needs to be included in it. 90 91- :yocto_wiki:`PR Service </PR_Service>`: A 92 network-based service that helps automate keeping package feeds 93 compatible with existing package manager applications such as RPM, 94 APT, and OPKG. 95 96Whenever the binary package content changes, the binary package version 97must change. Changing the binary package version is accomplished by 98changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these 99values occurs one of two ways: 100 101- Automatically using a Package Revision Service (PR Service). 102 103- Manually incrementing the :term:`PR` and/or :term:`PV` variables. 104 105Given a primary challenge of any build system and its users is how to 106maintain a package feed that is compatible with existing package manager 107applications such as RPM, APT, and OPKG, using an automated system is 108much preferred over a manual system. In either system, the main 109requirement is that binary package version numbering increases in a 110linear fashion and that there is a number of version components that 111support that linear progression. For information on how to ensure 112package revisioning remains linear, see the 113":ref:`dev-manual/packages:automatically incrementing a package version number`" 114section. 115 116The following three sections provide related information on the PR 117Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on 118how to ensure binary package revisioning remains linear. 119 120Working With a PR Service 121------------------------- 122 123As mentioned, attempting to maintain revision numbers in the 124:term:`Metadata` is error prone, inaccurate, 125and causes problems for people submitting recipes. Conversely, the PR 126Service automatically generates increasing numbers, particularly the 127revision field, which removes the human element. 128 129.. note:: 130 131 For additional information on using a PR Service, you can see the 132 :yocto_wiki:`PR Service </PR_Service>` wiki page. 133 134The Yocto Project uses variables in order of decreasing priority to 135facilitate revision numbering (i.e. 136:term:`PE`, 137:term:`PV`, and 138:term:`PR` for epoch, version, and 139revision, respectively). The values are highly dependent on the policies 140and procedures of a given distribution and package feed. 141 142Because the OpenEmbedded build system uses 143":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are 144unique to a given build, the build system knows when to rebuild 145packages. All the inputs into a given task are represented by a 146signature, which can trigger a rebuild when different. Thus, the build 147system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to 148trigger a rebuild. The signatures, however, can be used to generate 149these values. 150 151The PR Service works with both ``OEBasic`` and ``OEBasicHash`` 152generators. The value of :term:`PR` bumps when the checksum changes and the 153different generator mechanisms change signatures under different 154circumstances. 155 156As implemented, the build system includes values from the PR Service 157into the :term:`PR` field as an addition using the form "``.x``" so ``r0`` 158becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing 159:term:`PR` values to be used for whatever reasons, which include manual 160:term:`PR` bumps, should it be necessary. 161 162By default, the PR Service is not enabled or running. Thus, the packages 163generated are just "self consistent". The build system adds and removes 164packages and there are no guarantees about upgrade paths but images will 165be consistent and correct with the latest changes. 166 167The simplest form for a PR Service is for a single host development system 168that builds the package feed (building system). For this scenario, you can 169enable a local PR Service by setting :term:`PRSERV_HOST` in your 170``local.conf`` file in the :term:`Build Directory`:: 171 172 PRSERV_HOST = "localhost:0" 173 174Once the service is started, packages will automatically 175get increasing :term:`PR` values and BitBake takes care of starting and 176stopping the server. 177 178If you have a more complex setup where multiple host development systems 179work against a common, shared package feed, you have a single PR Service 180running and it is connected to each building system. For this scenario, 181you need to start the PR Service using the ``bitbake-prserv`` command:: 182 183 bitbake-prserv --host ip --port port --start 184 185In addition to 186hand-starting the service, you need to update the ``local.conf`` file of 187each building system as described earlier so each system points to the 188server and port. 189 190It is also recommended you use build history, which adds some sanity 191checks to binary package versions, in conjunction with the server that 192is running the PR Service. To enable build history, add the following to 193each building system's ``local.conf`` file:: 194 195 # It is recommended to activate "buildhistory" for testing the PR service 196 INHERIT += "buildhistory" 197 BUILDHISTORY_COMMIT = "1" 198 199For information on build 200history, see the 201":ref:`dev-manual/build-quality:maintaining build output quality`" section. 202 203.. note:: 204 205 The OpenEmbedded build system does not maintain :term:`PR` information as 206 part of the shared state (sstate) packages. If you maintain an sstate 207 feed, it's expected that either all your building systems that 208 contribute to the sstate feed use a shared PR service, or you do not 209 run a PR service on any of your building systems. 210 211 That's because if you had multiple machines sharing a PR service but 212 not their sstate feed, you could end up with "diverging" hashes for 213 the same output artefacts. When presented to the share PR service, 214 each would be considered as new and would increase the revision 215 number, causing many unnecessary package upgrades. 216 217 For more information on shared state, see the 218 ":ref:`overview-manual/concepts:shared state cache`" 219 section in the Yocto Project Overview and Concepts Manual. 220 221Manually Bumping PR 222------------------- 223 224The alternative to setting up a PR Service is to manually "bump" the 225:term:`PR` variable. 226 227If a committed change results in changing the package output, then the 228value of the :term:`PR` variable needs to be increased (or "bumped") as part of 229that commit. For new recipes you should add the :term:`PR` variable and set 230its initial value equal to "r0", which is the default. Even though the 231default value is "r0", the practice of adding it to a new recipe makes 232it harder to forget to bump the variable when you make changes to the 233recipe in future. 234 235Usually, version increases occur only to binary packages. However, if 236for some reason :term:`PV` changes but does not increase, you can increase 237the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to 238"0". 239 240Binary package version numbering strives to follow the `Debian Version 241Field Policy 242Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__. 243These guidelines define how versions are compared and what "increasing" 244a version means. 245 246Automatically Incrementing a Package Version Number 247--------------------------------------------------- 248 249When fetching a repository, BitBake uses the 250:term:`SRCREV` variable to determine 251the specific source code revision from which to build. You set the 252:term:`SRCREV` variable to 253:term:`AUTOREV` to cause the 254OpenEmbedded build system to automatically use the latest revision of 255the software:: 256 257 SRCREV = "${AUTOREV}" 258 259Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to 260automatically update the version whenever the revision of the source 261code changes. Here is an example:: 262 263 PV = "1.0+git${SRCPV}" 264 265The OpenEmbedded build system substitutes :term:`SRCPV` with the following: 266 267.. code-block:: none 268 269 AUTOINC+source_code_revision 270 271The build system replaces the ``AUTOINC`` 272with a number. The number used depends on the state of the PR Service: 273 274- If PR Service is enabled, the build system increments the number, 275 which is similar to the behavior of 276 :term:`PR`. This behavior results in 277 linearly increasing package versions, which is desirable. Here is an 278 example: 279 280 .. code-block:: none 281 282 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk 283 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk 284 285- If PR Service is not enabled, the build system replaces the 286 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in 287 changing the package version since the source revision is included. 288 However, package versions are not increased linearly. Here is an 289 example: 290 291 .. code-block:: none 292 293 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk 294 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk 295 296In summary, the OpenEmbedded build system does not track the history of 297binary package versions for this purpose. ``AUTOINC``, in this case, is 298comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the 299package version is simply replaced by "0". If PR server is enabled, the 300build system keeps track of the package versions and bumps the number 301when the package revision changes. 302 303Handling Optional Module Packaging 304================================== 305 306Many pieces of software split functionality into optional modules (or 307plugins) and the plugins that are built might depend on configuration 308options. To avoid having to duplicate the logic that determines what 309modules are available in your recipe or to avoid having to package each 310module by hand, the OpenEmbedded build system provides functionality to 311handle module packaging dynamically. 312 313To handle optional module packaging, you need to do two things: 314 315- Ensure the module packaging is actually done. 316 317- Ensure that any dependencies on optional modules from other recipes 318 are satisfied by your recipe. 319 320Making Sure the Packaging is Done 321--------------------------------- 322 323To ensure the module packaging actually gets done, you use the 324``do_split_packages`` function within the ``populate_packages`` Python 325function in your recipe. The ``do_split_packages`` function searches for 326a pattern of files or directories under a specified path and creates a 327package for each one it finds by appending to the 328:term:`PACKAGES` variable and 329setting the appropriate values for ``FILES:packagename``, 330``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth. 331Here is an example from the ``lighttpd`` recipe:: 332 333 python populate_packages:prepend () { 334 lighttpd_libdir = d.expand('${libdir}') 335 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$', 336 'lighttpd-module-%s', 'Lighttpd module for %s', 337 extra_depends='') 338 } 339 340The previous example specifies a number of things in the call to 341``do_split_packages``. 342 343- A directory within the files installed by your recipe through 344 :ref:`ref-tasks-install` in which to search. 345 346- A regular expression used to match module files in that directory. In 347 the example, note the parentheses () that mark the part of the 348 expression from which the module name should be derived. 349 350- A pattern to use for the package names. 351 352- A description for each package. 353 354- An empty string for ``extra_depends``, which disables the default 355 dependency on the main ``lighttpd`` package. Thus, if a file in 356 ``${libdir}`` called ``mod_alias.so`` is found, a package called 357 ``lighttpd-module-alias`` is created for it and the 358 :term:`DESCRIPTION` is set to 359 "Lighttpd module for alias". 360 361Often, packaging modules is as simple as the previous example. However, 362there are more advanced options that you can use within 363``do_split_packages`` to modify its behavior. And, if you need to, you 364can add more logic by specifying a hook function that is called for each 365package. It is also perfectly acceptable to call ``do_split_packages`` 366multiple times if you have more than one set of modules to package. 367 368For more examples that show how to use ``do_split_packages``, see the 369``connman.inc`` file in the ``meta/recipes-connectivity/connman/`` 370directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can 371also find examples in ``meta/classes-recipe/kernel.bbclass``. 372 373Here is a reference that shows ``do_split_packages`` mandatory and 374optional arguments:: 375 376 Mandatory arguments 377 378 root 379 The path in which to search 380 file_regex 381 Regular expression to match searched files. 382 Use parentheses () to mark the part of this 383 expression that should be used to derive the 384 module name (to be substituted where %s is 385 used in other function arguments as noted below) 386 output_pattern 387 Pattern to use for the package names. Must 388 include %s. 389 description 390 Description to set for each package. Must 391 include %s. 392 393 Optional arguments 394 395 postinst 396 Postinstall script to use for all packages 397 (as a string) 398 recursive 399 True to perform a recursive search --- default 400 False 401 hook 402 A hook function to be called for every match. 403 The function will be called with the following 404 arguments (in the order listed): 405 406 f 407 Full path to the file/directory match 408 pkg 409 The package name 410 file_regex 411 As above 412 output_pattern 413 As above 414 modulename 415 The module name derived using file_regex 416 extra_depends 417 Extra runtime dependencies (RDEPENDS) to be 418 set for all packages. The default value of None 419 causes a dependency on the main package 420 (${PN}) --- if you do not want this, pass empty 421 string '' for this parameter. 422 aux_files_pattern 423 Extra item(s) to be added to FILES for each 424 package. Can be a single string item or a list 425 of strings for multiple items. Must include %s. 426 postrm 427 postrm script to use for all packages (as a 428 string) 429 allow_dirs 430 True to allow directories to be matched - 431 default False 432 prepend 433 If True, prepend created packages to PACKAGES 434 instead of the default False which appends them 435 match_path 436 match file_regex on the whole relative path to 437 the root rather than just the filename 438 aux_files_pattern_verbatim 439 Extra item(s) to be added to FILES for each 440 package, using the actual derived module name 441 rather than converting it to something legal 442 for a package name. Can be a single string item 443 or a list of strings for multiple items. Must 444 include %s. 445 allow_links 446 True to allow symlinks to be matched --- default 447 False 448 summary 449 Summary to set for each package. Must include %s; 450 defaults to description if not set. 451 452 453 454Satisfying Dependencies 455----------------------- 456 457The second part for handling optional module packaging is to ensure that 458any dependencies on optional modules from other recipes are satisfied by 459your recipe. You can be sure these dependencies are satisfied by using 460the :term:`PACKAGES_DYNAMIC` 461variable. Here is an example that continues with the ``lighttpd`` recipe 462shown earlier:: 463 464 PACKAGES_DYNAMIC = "lighttpd-module-.*" 465 466The name 467specified in the regular expression can of course be anything. In this 468example, it is ``lighttpd-module-`` and is specified as the prefix to 469ensure that any :term:`RDEPENDS` and 470:term:`RRECOMMENDS` on a package 471name starting with the prefix are satisfied during build time. If you 472are using ``do_split_packages`` as described in the previous section, 473the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name 474pattern specified in the call to ``do_split_packages``. 475 476Using Runtime Package Management 477================================ 478 479During a build, BitBake always transforms a recipe into one or more 480packages. For example, BitBake takes the ``bash`` recipe and produces a 481number of packages (e.g. ``bash``, ``bash-bashbug``, 482``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``, 483``bash-completion-extra``, ``bash-dbg``, and so forth). Not all 484generated packages are included in an image. 485 486In several situations, you might need to update, add, remove, or query 487the packages on a target device at runtime (i.e. without having to 488generate a new image). Examples of such situations include: 489 490- You want to provide in-the-field updates to deployed devices (e.g. 491 security updates). 492 493- You want to have a fast turn-around development cycle for one or more 494 applications that run on your device. 495 496- You want to temporarily install the "debug" packages of various 497 applications on your device so that debugging can be greatly improved 498 by allowing access to symbols and source debugging. 499 500- You want to deploy a more minimal package selection of your device 501 but allow in-the-field updates to add a larger selection for 502 customization. 503 504In all these situations, you have something similar to a more 505traditional Linux distribution in that in-field devices are able to 506receive pre-compiled packages from a server for installation or update. 507Being able to install these packages on a running, in-field device is 508what is termed "runtime package management". 509 510In order to use runtime package management, you need a host or server 511machine that serves up the pre-compiled packages plus the required 512metadata. You also need package manipulation tools on the target. The 513build machine is a likely candidate to act as the server. However, that 514machine does not necessarily have to be the package server. The build 515machine could push its artifacts to another machine that acts as the 516server (e.g. Internet-facing). In fact, doing so is advantageous for a 517production environment as getting the packages away from the development 518system's :term:`Build Directory` prevents accidental overwrites. 519 520A simple build that targets just one device produces more than one 521package database. In other words, the packages produced by a build are 522separated out into a couple of different package groupings based on 523criteria such as the target's CPU architecture, the target board, or the 524C library used on the target. For example, a build targeting the 525``qemux86`` device produces the following three package databases: 526``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86`` 527device to be aware of all the packages that were available to it, you 528would need to point it to each of these databases individually. In a 529similar way, a traditional Linux distribution usually is configured to 530be aware of a number of software repositories from which it retrieves 531packages. 532 533Using runtime package management is completely optional and not required 534for a successful build or deployment in any way. But if you want to make 535use of runtime package management, you need to do a couple things above 536and beyond the basics. The remainder of this section describes what you 537need to do. 538 539Build Considerations 540-------------------- 541 542This section describes build considerations of which you need to be 543aware in order to provide support for runtime package management. 544 545When BitBake generates packages, it needs to know what format or formats 546to use. In your configuration, you use the 547:term:`PACKAGE_CLASSES` 548variable to specify the format: 549 550#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g. 551 ``poky/build/conf/local.conf``). 552 553#. Select the desired package format as follows:: 554 555 PACKAGE_CLASSES ?= "package_packageformat" 556 557 where packageformat can be "ipk", "rpm", 558 "deb", or "tar" which are the supported package formats. 559 560 .. note:: 561 562 Because the Yocto Project supports four different package formats, 563 you can set the variable with more than one argument. However, the 564 OpenEmbedded build system only uses the first argument when 565 creating an image or Software Development Kit (SDK). 566 567If you would like your image to start off with a basic package database 568containing the packages in your current build as well as to have the 569relevant tools available on the target for runtime package management, 570you can include "package-management" in the 571:term:`IMAGE_FEATURES` 572variable. Including "package-management" in this configuration variable 573ensures that when the image is assembled for your target, the image 574includes the currently-known package databases as well as the 575target-specific tools required for runtime package management to be 576performed on the target. However, this is not strictly necessary. You 577could start your image off without any databases but only include the 578required on-target package tool(s). As an example, you could include 579"opkg" in your 580:term:`IMAGE_INSTALL` variable 581if you are using the IPK package format. You can then initialize your 582target's package database(s) later once your image is up and running. 583 584Whenever you perform any sort of build step that can potentially 585generate a package or modify existing package, it is always a good idea 586to re-generate the package index after the build by using the following 587command:: 588 589 $ bitbake package-index 590 591It might be tempting to build the 592package and the package index at the same time with a command such as 593the following:: 594 595 $ bitbake some-package package-index 596 597Do not do this as 598BitBake does not schedule the package index for after the completion of 599the package you are building. Consequently, you cannot be sure of the 600package index including information for the package you just built. 601Thus, be sure to run the package update step separately after building 602any packages. 603 604You can use the 605:term:`PACKAGE_FEED_ARCHS`, 606:term:`PACKAGE_FEED_BASE_PATHS`, 607and 608:term:`PACKAGE_FEED_URIS` 609variables to pre-configure target images to use a package feed. If you 610do not define these variables, then manual steps as described in the 611subsequent sections are necessary to configure the target. You should 612set these variables before building the image in order to produce a 613correctly configured image. 614 615.. note:: 616 617 Your image will need enough free storage space to run package upgrades, 618 especially if many of them need to be downloaded at the same time. 619 You should make sure images are created with enough free space 620 by setting the :term:`IMAGE_ROOTFS_EXTRA_SPACE` variable. 621 622When your build is complete, your packages reside in the 623``${TMPDIR}/deploy/packageformat`` directory. For example, if 624``${``\ :term:`TMPDIR`\ ``}`` is 625``tmp`` and your selected package type is RPM, then your RPM packages 626are available in ``tmp/deploy/rpm``. 627 628Host or Server Machine Setup 629---------------------------- 630 631Although other protocols are possible, a server using HTTP typically 632serves packages. If you want to use HTTP, then set up and configure a 633web server such as Apache 2, lighttpd, or Python web server on the 634machine serving the packages. 635 636To keep things simple, this section describes how to set up a 637Python web server to share package feeds from the developer's 638machine. Although this server might not be the best for a production 639environment, the setup is simple and straight forward. Should you want 640to use a different server more suited for production (e.g. Apache 2, 641Lighttpd, or Nginx), take the appropriate steps to do so. 642 643From within the :term:`Build Directory` where you have built an image based on 644your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start 645the server. The following example assumes a :term:`Build Directory` of ``poky/build`` 646and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`":: 647 648 $ cd poky/build/tmp/deploy/rpm 649 $ python3 -m http.server 650 651Target Setup 652------------ 653 654Setting up the target differs depending on the package management 655system. This section provides information for RPM, IPK, and DEB. 656 657Using RPM 658~~~~~~~~~ 659 660The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs 661runtime package management of RPM packages. In order to use DNF for 662runtime package management, you must perform an initial setup on the 663target machine for cases where the ``PACKAGE_FEED_*`` variables were not 664set as part of the image that is running on the target. This means if 665you built your image and did not use these variables as part of the 666build and your image is now running on the target, you need to perform 667the steps in this section if you want to use runtime package management. 668 669.. note:: 670 671 For information on the ``PACKAGE_FEED_*`` variables, see 672 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and 673 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables 674 glossary. 675 676On the target, you must inform DNF that package databases are available. 677You do this by creating a file named 678``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``. 679 680As an example, assume the target is able to use the following package 681databases: ``all``, ``i586``, and ``qemux86`` from a server named 682``my.server``. The specifics for setting up the web server are up to 683you. The critical requirement is that the URIs in the target repository 684configuration point to the correct remote location for the feeds. 685 686.. note:: 687 688 For development purposes, you can point the web server to the build 689 system's ``deploy`` directory. However, for production use, it is better to 690 copy the package directories to a location outside of the build area and use 691 that location. Doing so avoids situations where the build system 692 overwrites or changes the ``deploy`` directory. 693 694When telling DNF where to look for the package databases, you must 695declare individual locations per architecture or a single location used 696for all architectures. You cannot do both: 697 698- *Create an Explicit List of Architectures:* Define individual base 699 URLs to identify where each package database is located: 700 701 .. code-block:: none 702 703 [oe-packages] 704 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all 705 706 This example 707 informs DNF about individual package databases for all three 708 architectures. 709 710- *Create a Single (Full) Package Index:* Define a single base URL that 711 identifies where a full package database is located:: 712 713 [oe-packages] 714 baseurl=http://my.server/rpm 715 716 This example informs DNF about a single 717 package database that contains all the package index information for 718 all supported architectures. 719 720Once you have informed DNF where to find the package databases, you need 721to fetch them: 722 723.. code-block:: none 724 725 # dnf makecache 726 727DNF is now able to find, install, and 728upgrade packages from the specified repository or repositories. 729 730.. note:: 731 732 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for 733 additional information. 734 735Using IPK 736~~~~~~~~~ 737 738The ``opkg`` application performs runtime package management of IPK 739packages. You must perform an initial setup for ``opkg`` on the target 740machine if the 741:term:`PACKAGE_FEED_ARCHS`, 742:term:`PACKAGE_FEED_BASE_PATHS`, 743and 744:term:`PACKAGE_FEED_URIS` 745variables have not been set or the target image was built before the 746variables were set. 747 748The ``opkg`` application uses configuration files to find available 749package databases. Thus, you need to create a configuration file inside 750the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository 751you want to use. 752 753As an example, suppose you are serving packages from a ``ipk/`` 754directory containing the ``i586``, ``all``, and ``qemux86`` databases 755through an HTTP server named ``my.server``. On the target, create a 756configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/`` 757directory containing the following: 758 759.. code-block:: none 760 761 src/gz all http://my.server/ipk/all 762 src/gz i586 http://my.server/ipk/i586 763 src/gz qemux86 http://my.server/ipk/qemux86 764 765Next, instruct ``opkg`` to fetch the 766repository information: 767 768.. code-block:: none 769 770 # opkg update 771 772The ``opkg`` application is now able to find, install, and upgrade packages 773from the specified repository. 774 775Using DEB 776~~~~~~~~~ 777 778The ``apt`` application performs runtime package management of DEB 779packages. This application uses a source list file to find available 780package databases. You must perform an initial setup for ``apt`` on the 781target machine if the 782:term:`PACKAGE_FEED_ARCHS`, 783:term:`PACKAGE_FEED_BASE_PATHS`, 784and 785:term:`PACKAGE_FEED_URIS` 786variables have not been set or the target image was built before the 787variables were set. 788 789To inform ``apt`` of the repository you want to use, you might create a 790list file (e.g. ``my_repo.list``) inside the 791``/etc/apt/sources.list.d/`` directory. As an example, suppose you are 792serving packages from a ``deb/`` directory containing the ``i586``, 793``all``, and ``qemux86`` databases through an HTTP server named 794``my.server``. The list file should contain: 795 796.. code-block:: none 797 798 deb http://my.server/deb/all ./ 799 deb http://my.server/deb/i586 ./ 800 deb http://my.server/deb/qemux86 ./ 801 802Next, instruct the ``apt`` application 803to fetch the repository information: 804 805.. code-block:: none 806 807 $ sudo apt update 808 809After this step, 810``apt`` is able to find, install, and upgrade packages from the 811specified repository. 812 813Generating and Using Signed Packages 814==================================== 815 816In order to add security to RPM packages used during a build, you can 817take steps to securely sign them. Once a signature is verified, the 818OpenEmbedded build system can use the package in the build. If security 819fails for a signed package, the build system stops the build. 820 821This section describes how to sign RPM packages during a build and how 822to use signed package feeds (repositories) when doing a build. 823 824Signing RPM Packages 825-------------------- 826 827To enable signing RPM packages, you must set up the following 828configurations in either your ``local.config`` or ``distro.config`` 829file:: 830 831 # Inherit sign_rpm.bbclass to enable signing functionality 832 INHERIT += " sign_rpm" 833 # Define the GPG key that will be used for signing. 834 RPM_GPG_NAME = "key_name" 835 # Provide passphrase for the key 836 RPM_GPG_PASSPHRASE = "passphrase" 837 838.. note:: 839 840 Be sure to supply appropriate values for both `key_name` and 841 `passphrase`. 842 843Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in 844the previous example, two optional variables related to signing are available: 845 846- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed 847 when the package is signed. 848 849- *GPG_PATH:* Specifies the ``gpg`` home directory used when the 850 package is signed. 851 852Processing Package Feeds 853------------------------ 854 855In addition to being able to sign RPM packages, you can also enable 856signed package feeds for IPK and RPM packages. 857 858The steps you need to take to enable signed package feed use are similar 859to the steps used to sign RPM packages. You must define the following in 860your ``local.config`` or ``distro.config`` file:: 861 862 INHERIT += "sign_package_feed" 863 PACKAGE_FEED_GPG_NAME = "key_name" 864 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase" 865 866For signed package feeds, the passphrase must be specified in a separate file, 867which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` 868variable. Regarding security, keeping a plain text passphrase out of the 869configuration is more secure. 870 871Aside from the ``PACKAGE_FEED_GPG_NAME`` and 872``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables 873related to signed package feeds are available: 874 875- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed 876 when the package is signed. 877 878- *GPG_PATH:* Specifies the ``gpg`` home directory used when the 879 package is signed. 880 881- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg`` 882 signature. This variable applies only to RPM and IPK package feeds. 883 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are 884 "ASC", which is the default and specifies ascii armored, and "BIN", 885 which specifies binary. 886 887Testing Packages With ptest 888=========================== 889 890A Package Test (ptest) runs tests against packages built by the 891OpenEmbedded build system on the target machine. A ptest contains at 892least two items: the actual test, and a shell script (``run-ptest``) 893that starts the test. The shell script that starts the test must not 894contain the actual test --- the script only starts the test. On the other 895hand, the test can be anything from a simple shell script that runs a 896binary and checks the output to an elaborate system of test binaries and 897data files. 898 899The test generates output in the format used by Automake:: 900 901 result: testname 902 903where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and 904the testname can be any identifying string. 905 906For a list of Yocto Project recipes that are already enabled with ptest, 907see the :yocto_wiki:`Ptest </Ptest>` wiki page. 908 909.. note:: 910 911 A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest` 912 class. 913 914Adding ptest to Your Build 915-------------------------- 916 917To add package testing to your build, add the :term:`DISTRO_FEATURES` and 918:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which 919is found in the :term:`Build Directory`:: 920 921 DISTRO_FEATURES:append = " ptest" 922 EXTRA_IMAGE_FEATURES += "ptest-pkgs" 923 924Once your build is complete, the ptest files are installed into the 925``/usr/lib/package/ptest`` directory within the image, where ``package`` 926is the name of the package. 927 928Running ptest 929------------- 930 931The ``ptest-runner`` package installs a shell script that loops through 932all installed ptest test suites and runs them in sequence. Consequently, 933you might want to add this package to your image. 934 935Getting Your Package Ready 936-------------------------- 937 938In order to enable a recipe to run installed ptests on target hardware, 939you need to prepare the recipes that build the packages you want to 940test. Here is what you have to do for each recipe: 941 942- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:* 943 Include the following line in each recipe:: 944 945 inherit ptest 946 947- *Create run-ptest:* This script starts your test. Locate the 948 script where you will refer to it using 949 :term:`SRC_URI`. Here is an 950 example that starts a test for ``dbus``:: 951 952 #!/bin/sh 953 cd test 954 make -k runtest-TESTS 955 956- *Ensure dependencies are met:* If the test adds build or runtime 957 dependencies that normally do not exist for the package (such as 958 requiring "make" to run the test suite), use the 959 :term:`DEPENDS` and 960 :term:`RDEPENDS` variables in 961 your recipe in order for the package to meet the dependencies. Here 962 is an example where the package has a runtime dependency on "make":: 963 964 RDEPENDS:${PN}-ptest += "make" 965 966- *Add a function to build the test suite:* Not many packages support 967 cross-compilation of their test suites. Consequently, you usually 968 need to add a cross-compilation function to the package. 969 970 Many packages based on Automake compile and run the test suite by 971 using a single command such as ``make check``. However, the host 972 ``make check`` builds and runs on the same computer, while 973 cross-compiling requires that the package is built on the host but 974 executed for the target architecture (though often, as in the case 975 for ptest, the execution occurs on the host). The built version of 976 Automake that ships with the Yocto Project includes a patch that 977 separates building and execution. Consequently, packages that use the 978 unaltered, patched version of ``make check`` automatically 979 cross-compiles. 980 981 Regardless, you still must add a ``do_compile_ptest`` function to 982 build the test suite. Add a function similar to the following to your 983 recipe:: 984 985 do_compile_ptest() { 986 oe_runmake buildtest-TESTS 987 } 988 989- *Ensure special configurations are set:* If the package requires 990 special configurations prior to compiling the test code, you must 991 insert a ``do_configure_ptest`` function into the recipe. 992 993- *Install the test suite:* The :ref:`ref-classes-ptest` class 994 automatically copies the file ``run-ptest`` to the target and then runs make 995 ``install-ptest`` to run the tests. If this is not enough, you need 996 to create a ``do_install_ptest`` function and make sure it gets 997 called after the "make install-ptest" completes. 998 999Creating Node Package Manager (NPM) Packages 1000============================================ 1001 1002:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript 1003programming language. The Yocto Project supports the NPM 1004:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. 1005You can use this fetcher in combination with 1006:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce 1007NPM packages. 1008 1009There are two workflows that allow you to create NPM packages using 1010``devtool``: the NPM registry modules method and the NPM project code 1011method. 1012 1013.. note:: 1014 1015 While it is possible to create NPM recipes manually, using 1016 ``devtool`` is far simpler. 1017 1018Additionally, some requirements and caveats exist. 1019 1020Requirements and Caveats 1021------------------------ 1022 1023You need to be aware of the following before using ``devtool`` to create 1024NPM packages: 1025 1026- Of the two methods that you can use ``devtool`` to create NPM 1027 packages, the registry approach is slightly simpler. However, you 1028 might consider the project approach because you do not have to 1029 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__, 1030 which is NPM's public registry. 1031 1032- Be familiar with 1033 :doc:`devtool </ref-manual/devtool-reference>`. 1034 1035- The NPM host tools need the native ``nodejs-npm`` package, which is 1036 part of the OpenEmbedded environment. You need to get the package by 1037 cloning the :oe_git:`meta-openembedded </meta-openembedded>` 1038 repository. Be sure to add the path to your local copy 1039 to your ``bblayers.conf`` file. 1040 1041- ``devtool`` cannot detect native libraries in module dependencies. 1042 Consequently, you must manually add packages to your recipe. 1043 1044- While deploying NPM packages, ``devtool`` cannot determine which 1045 dependent packages are missing on the target (e.g. the node runtime 1046 ``nodejs``). Consequently, you need to find out what files are 1047 missing and be sure they are on the target. 1048 1049- Although you might not need NPM to run your node package, it is 1050 useful to have NPM on your target. The NPM package name is 1051 ``nodejs-npm``. 1052 1053Using the Registry Modules Method 1054--------------------------------- 1055 1056This section presents an example that uses the ``cute-files`` module, 1057which is a file browser web application. 1058 1059.. note:: 1060 1061 You must know the ``cute-files`` module version. 1062 1063The first thing you need to do is use ``devtool`` and the NPM fetcher to 1064create the recipe:: 1065 1066 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2" 1067 1068The 1069``devtool add`` command runs ``recipetool create`` and uses the same 1070fetch URI to download each dependency and capture license details where 1071possible. The result is a generated recipe. 1072 1073After running for quite a long time, in particular building the 1074``nodejs-native`` package, the command should end as follows:: 1075 1076 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional 1077 1078The recipe file is fairly simple and contains every license that 1079``recipetool`` finds and includes the licenses in the recipe's 1080:term:`LIC_FILES_CHKSUM` 1081variables. You need to examine the variables and look for those with 1082"unknown" in the :term:`LICENSE` 1083field. You need to track down the license information for "unknown" 1084modules and manually add the information to the recipe. 1085 1086``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap 1087files capture the version of all dependent modules. Many packages do not 1088provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it 1089runs. 1090 1091.. note:: 1092 1093 A package is created for each sub-module. This policy is the only 1094 practical way to have the licenses for all of the dependencies 1095 represented in the license manifest of the image. 1096 1097The ``devtool edit-recipe`` command lets you take a look at the recipe:: 1098 1099 $ devtool edit-recipe cute-files 1100 # Recipe created by recipetool 1101 # This is the basis of a recipe and may need further editing in order to be fully functional. 1102 # (Feel free to remove these comments when editing.) 1103 1104 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network." 1105 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is 1106 # your responsibility to verify that the values are complete and correct. 1107 # 1108 # NOTE: multiple licenses have been detected; they have been separated with & 1109 # in the LICENSE value for now since it is a reasonable assumption that all 1110 # of the licenses apply. If instead there is a choice between the multiple 1111 # licenses then you should change the value to separate the licenses with | 1112 # instead of &. If there is any doubt, check the accompanying documentation 1113 # to determine which situation is applicable. 1114 1115 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network." 1116 LICENSE = "BSD-3-Clause & ISC & MIT" 1117 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \ 1118 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \ 1119 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \ 1120 ... 1121 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a" 1122 1123 SRC_URI = " \ 1124 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 1125 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 1126 " 1127 1128 S = "${WORKDIR}/npm" 1129 1130 inherit npm 1131 1132 LICENSE:${PN} = "MIT" 1133 LICENSE:${PN}-accepts = "MIT" 1134 LICENSE:${PN}-array-flatten = "MIT" 1135 ... 1136 LICENSE:${PN}-vary = "MIT" 1137 1138Three key points in the previous example are: 1139 1140- :term:`SRC_URI` uses the NPM 1141 scheme so that the NPM fetcher is used. 1142 1143- ``recipetool`` collects all the license information. If a 1144 sub-module's license is unavailable, the sub-module's name appears in 1145 the comments. 1146 1147- The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to 1148 package up all the modules. 1149 1150You can run the following command to build the ``cute-files`` package:: 1151 1152 $ devtool build cute-files 1153 1154Remember that ``nodejs`` must be installed on 1155the target before your package. 1156 1157Assuming 192.168.7.2 for the target's IP address, use the following 1158command to deploy your package:: 1159 1160 $ devtool deploy-target -s cute-files root@192.168.7.2 1161 1162Once the package is installed on the target, you can 1163test the application to show the contents of any directory:: 1164 1165 $ cd /usr/lib/node_modules/cute-files 1166 $ cute-files 1167 1168On a browser, 1169go to ``http://192.168.7.2:3000`` and you see the following: 1170 1171.. image:: figures/cute-files-npm-example.png 1172 :width: 100% 1173 1174You can find the recipe in ``workspace/recipes/cute-files``. You can use 1175the recipe in any layer you choose. 1176 1177Using the NPM Projects Code Method 1178---------------------------------- 1179 1180Although it is useful to package modules already in the NPM registry, 1181adding ``node.js`` projects under development is a more common developer 1182use case. 1183 1184This section covers the NPM projects code method, which is very similar 1185to the "registry" approach described in the previous section. In the NPM 1186projects method, you provide ``devtool`` with an URL that points to the 1187source files. 1188 1189Replicating the same example, (i.e. ``cute-files``) use the following 1190command:: 1191 1192 $ devtool add https://github.com/martinaglv/cute-files.git 1193 1194The recipe this command generates is very similar to the recipe created in 1195the previous section. However, the :term:`SRC_URI` looks like the following:: 1196 1197 SRC_URI = " \ 1198 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \ 1199 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 1200 " 1201 1202In this example, 1203the main module is taken from the Git repository and dependencies are 1204taken from the NPM registry. Other than those differences, the recipe is 1205basically the same between the two methods. You can build and deploy the 1206package exactly as described in the previous section that uses the 1207registry modules method. 1208 1209Adding custom metadata to packages 1210================================== 1211 1212The variable 1213:term:`PACKAGE_ADD_METADATA` 1214can be used to add additional metadata to packages. This is reflected in 1215the package control/spec file. To take the ipk format for example, the 1216CONTROL file stored inside would contain the additional metadata as 1217additional lines. 1218 1219The variable can be used in multiple ways, including using suffixes to 1220set it for a specific package type and/or package. Note that the order 1221of precedence is the same as this list: 1222 1223- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>`` 1224 1225- ``PACKAGE_ADD_METADATA_<PKGTYPE>`` 1226 1227- ``PACKAGE_ADD_METADATA:<PN>`` 1228 1229- :term:`PACKAGE_ADD_METADATA` 1230 1231`<PKGTYPE>` is a parameter and expected to be a distinct name of specific 1232package type: 1233 1234- IPK for .ipk packages 1235 1236- DEB for .deb packages 1237 1238- RPM for .rpm packages 1239 1240`<PN>` is a parameter and expected to be a package name. 1241 1242The variable can contain multiple [one-line] metadata fields separated 1243by the literal sequence '\\n'. The separator can be redefined using the 1244variable flag ``separator``. 1245 1246Here is an example that adds two custom fields for ipk 1247packages:: 1248 1249 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets" 1250 1251