1Migration notes for 3.4 (honister) 2---------------------------------- 3 4This section provides migration information for moving to the Yocto 5Project 3.4 Release (codename "honister") from the prior release. 6 7Override syntax changes 8~~~~~~~~~~~~~~~~~~~~~~~ 9 10In this release, the ``:`` character replaces the use of ``_`` to 11refer to an override, most commonly when making a conditional assignment 12of a variable. This means that an entry like:: 13 14 SRC_URI_qemux86 = "file://somefile" 15 16now becomes:: 17 18 SRC_URI:qemux86 = "file://somefile" 19 20since ``qemux86`` is an override. This applies to any use of override 21syntax, so the following:: 22 23 SRC_URI_append = " file://somefile" 24 SRC_URI_append_qemux86 = " file://somefile2" 25 SRC_URI_remove_qemux86-64 = "file://somefile3" 26 SRC_URI_prepend_qemuarm = "file://somefile4 " 27 FILES_${PN}-ptest = "${bindir}/xyz" 28 IMAGE_CMD_tar = "tar" 29 BASE_LIB_tune-cortexa76 = "lib" 30 SRCREV_pn-bash = "abc" 31 BB_TASK_NICE_LEVEL_task-testimage = '0' 32 33would now become:: 34 35 SRC_URI:append = " file://somefile" 36 SRC_URI:append:qemux86 = " file://somefile2" 37 SRC_URI:remove:qemux86-64 = "file://somefile3" 38 SRC_URI:prepend:qemuarm = "file://somefile4 " 39 FILES:${PN}-ptest = "${bindir}/xyz" 40 IMAGE_CMD:tar = "tar" 41 BASE_LIB:tune-cortexa76 = "lib" 42 SRCREV:pn-bash = "abc" 43 BB_TASK_NICE_LEVEL:task-testimage = '0' 44 45This also applies to 46:ref:`variable queries to the datastore <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:functions for accessing datastore variables>`, 47for example using ``getVar`` and similar so ``d.getVar("RDEPENDS_${PN}")`` 48becomes ``d.getVar("RDEPENDS:${PN}")``. 49 50Whilst some of these are fairly obvious such as :term:`MACHINE` and :term:`DISTRO` 51overrides, some are less obvious, for example the packaging variables such as 52:term:`RDEPENDS`, :term:`FILES` and so on taking package names (e.g. ``${PN}``, 53``${PN}-ptest``) as overrides. These overrides are not always in 54:term:`OVERRIDES` but applied conditionally in specific contexts 55such as packaging. ``task-<taskname>`` is another context specific override, the 56context being specific tasks in that case. Tune overrides are another special 57case where some code does use them as overrides but some does not. We plan to try 58and make the tune code use overrides more consistently in the future. 59 60There are some variables which do not use override syntax which include the 61suffix to variables in ``layer.conf`` files such as :term:`BBFILE_PATTERN`, 62:term:`SRCREV`\ ``_xxx`` where ``xxx`` is a name from :term:`SRC_URI` and 63:term:`PREFERRED_VERSION`\ ``_xxx``. In particular, ``layer.conf`` suffixes 64may be the same as a :term:`DISTRO` override causing some confusion. We do 65plan to try and improve consistency as these issues are identified. 66 67To help with migration of layers, a script has been provided in OE-Core. 68Once configured with the overrides used by a layer, this can be run as:: 69 70 <oe-core>/scripts/contrib/convert-overrides.py <layerdir> 71 72.. note:: 73 74 Please read the notes in the script as it isn't entirely automatic and it isn't 75 expected to handle every case. In particular, it needs to be told which overrides 76 the layer uses (usually machine and distro names/overrides) and the result should 77 be carefully checked since it can be a little enthusiastic and will convert 78 references to ``_append``, ``_remove`` and ``_prepend`` in function and variable 79 names. 80 81For reference, this conversion is important as it allows BitBake to more reliably 82determine what is an override and what is not, as underscores are also used in 83variable names without intending to be overrides. This should allow us to proceed 84with other syntax improvements and simplifications for usability. It also means 85BitBake no longer has to guess and maintain large lookup lists just in case 86e.g. ``functionname`` in ``my_functionname`` is an override, and thus should improve 87efficiency. 88 89New host dependencies 90~~~~~~~~~~~~~~~~~~~~~ 91 92The ``lz4c``, ``pzstd`` and ``zstd`` commands are now required to be 93installed on the build host to support LZ4 and Zstandard compression 94functionality. These are typically provided by ``lz4`` and ``zstd`` 95packages in most Linux distributions. Alternatively they are available 96as part of ``buildtools-tarball`` if your distribution does not provide 97them. For more information see 98:ref:`ref-manual/system-requirements:required packages for the build host`. 99 100Removed recipes 101~~~~~~~~~~~~~~~ 102 103The following recipes have been removed in this release: 104 105- ``assimp``: problematic from a licensing perspective and no longer 106 needed by anything else 107- ``clutter-1.0``: legacy component moved to meta-gnome 108- ``clutter-gst-3.0``: legacy component moved to meta-gnome 109- ``clutter-gtk-1.0``: legacy component moved to meta-gnome 110- ``cogl-1.0``: legacy component moved to meta-gnome 111- ``core-image-clutter``: removed along with clutter 112- ``linux-yocto``: removed version 5.4 recipes (5.14 and 5.10 still 113 provided) 114- ``mklibs-native``: not actively tested and upstream mklibs still 115 requires Python 2 116- ``mx-1.0``: obsolete (last release 2012) and isn't used by anything in 117 any known layer 118- ``packagegroup-core-clutter``: removed along with clutter 119 120Removed classes 121~~~~~~~~~~~~~~~ 122 123- ``clutter``: moved to meta-gnome along with clutter itself 124- ``image-mklibs``: not actively tested and upstream mklibs still 125 requires Python 2 126- ``meta``: no longer useful. Recipes that need to skip installing 127 packages should inherit ``nopackages`` instead. 128 129Prelinking disabled by default 130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 131 132Recent tests have shown that prelinking works only when PIE is not 133enabled (see `here <https://rlbl.me/prelink-1>`__ and `here <https://rlbl.me/prelink-2>`__), 134and as PIE is both a desirable security feature, and the only 135configuration provided and tested by the Yocto Project, there is 136simply no sense in continuing to enable prelink. 137 138There's also a concern that no one is maintaining the code, and there 139are open bugs (including :yocto_bugs:`this serious one </show_bug.cgi?id=14429>`). 140Given that prelink does intricate address arithmetic and rewriting 141of binaries the best option is to disable the feature. It is recommended 142that you consider disabling this feature in your own configuration if 143it is currently enabled. 144 145Virtual runtime provides 146~~~~~~~~~~~~~~~~~~~~~~~~ 147 148Recipes shouldn't use the ``virtual/`` string in :term:`RPROVIDES` and 149:term:`RDEPENDS` --- it is confusing because ``virtual/`` has no special 150meaning in :term:`RPROVIDES` and :term:`RDEPENDS` (unlike in the 151corresponding build-time :term:`PROVIDES` and :term:`DEPENDS`). 152 153Tune files moved to architecture-specific directories 154~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 155 156The tune files found in ``conf/machine/include`` have now been moved 157into their respective architecture name directories under that same 158location; e.g. x86 tune files have moved into an ``x86`` subdirectory, 159MIPS tune files have moved into a ``mips`` subdirectory, etc. 160The ARM tunes have an extra level (``armv8a``, ``armv8m``, etc.) and 161some have been renamed to make them uniform with the rest of the tunes. 162See :yocto_git:`this commit </poky/commit/?id=1d381f21f5f13aa0c4e1a45683ed656ebeedd37d>` 163for reference. 164 165If you have any references to tune files (e.g. in custom machine 166configuration files) they will need to be updated. 167 168Extensible SDK host extension 169~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 170 171For a normal SDK, some layers append to :term:`TOOLCHAIN_HOST_TASK` 172unconditionally which is fine, until the eSDK tries to override the 173variable to its own values. Instead of installing packages specified 174in this variable it uses native recipes instead --- a very different 175approach. This has led to confusing errors when binaries are added 176to the SDK but not relocated. 177 178To avoid these issues, a new :term:`TOOLCHAIN_HOST_TASK_ESDK` variable has 179been created. If you wish to extend what is installed in the host 180portion of the eSDK then you will now need to set this variable. 181 182Package/recipe splitting 183~~~~~~~~~~~~~~~~~~~~~~~~ 184 185- ``perl-cross`` has been split out from the main ``perl`` recipe to 186 its own ``perlcross`` recipe for maintenance reasons. If you have 187 bbappends for the perl recipe then these may need extending. 188 189- The ``wayland`` recipe now packages its binaries in a 190 ``wayland-tools`` package rather than putting them into 191 ``wayland-dev``. 192 193- Xwayland has been split out of the xserver-xorg tree and thus is now 194 in its own ``xwayland`` recipe. If you need Xwayland in your image 195 then you may now need to add it explicitly. 196 197- The ``rpm`` package no longer has ``rpm-build`` in its :term:`RRECOMMENDS`; 198 if by chance you still need rpm package building functionality in 199 your image and you have not already done so then you should add 200 ``rpm-build`` to your image explicitly. 201 202- The Python ``statistics`` standard module is now packaged in its own 203 ``python3-statistics`` package instead of ``python3-misc`` as 204 previously. 205 206Image / SDK generation changes 207~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 208 209- Recursive dependencies on the :ref:`ref-tasks-build` task are now disabled when 210 building SDKs. These are generally not needed; in the unlikely event 211 that you do encounter problems then it will probably be as a result of 212 missing explicit dependencies that need to be added. 213 214- Errors during "complementary" package installation (e.g. for ``*-dbg`` 215 and ``*-dev`` packages) during image construction are no longer 216 ignored. Historically some of these packages had installation problems, 217 that is no longer the case. In the unlikely event that you see errors 218 as a result, you will need to fix the installation/packaging issues. 219 220- When building an image, only packages that will be used in building 221 the image (i.e. the first entry in :term:`PACKAGE_CLASSES`) will be 222 produced if multiple package types are enabled (which is not a typical 223 configuration). If in your CI system you need to have the original 224 behaviour, use ``bitbake --runall build <target>``. 225 226- The ``-lic`` package is no longer automatically added to 227 :term:`RRECOMMENDS` for every other package when 228 :term:`LICENSE_CREATE_PACKAGE` is set to "1". If you wish all license 229 packages to be installed corresponding to packages in your image, then 230 you should instead add the new ``lic-pkgs`` feature to 231 :term:`IMAGE_FEATURES`. 232 233Miscellaneous 234~~~~~~~~~~~~~ 235 236- Certificates are now properly checked when BitBake fetches sources 237 over HTTPS. If you receive errors as a result for your custom recipes, 238 you will need to use a mirror or address the issue with the operators 239 of the server in question. 240 241- ``avahi`` has had its GTK+ support disabled by default. If you wish to 242 re-enable it, set ``AVAHI_GTK = "gtk3"`` in a bbappend for the 243 ``avahi`` recipe or in your custom distro configuration file. 244 245- Setting the ``BUILD_REPRODUCIBLE_BINARIES`` variable to "0" no longer 246 uses a strangely old fallback date of April 2011, it instead disables 247 building reproducible binaries as you would logically expect. 248 249- Setting noexec/nostamp/fakeroot varflags to any value besides "1" will 250 now trigger a warning. These should be either set to "1" to enable, or 251 not set at all to disable. 252 253- The previously deprecated ``COMPRESS_CMD`` and 254 ``CVE_CHECK_CVE_WHITELIST`` variables have been removed. Use 255 :term:`CONVERSION_CMD` and ``CVE_CHECK_WHITELIST`` (replaced by 256 :term:`CVE_CHECK_IGNORE` in version 3.5) respectively 257 instead. 258 259- The obsolete ``oe_machinstall`` function previously provided in the 260 :ref:`utils <ref-classes-utils>` class has been removed. For 261 machine-specific installation it is recommended that you use the 262 built-in override support in the fetcher or overrides in general 263 instead. 264 265- The ``-P`` (``--clear-password``) option can no longer be used with 266 ``useradd`` and ``usermod`` entries in :term:`EXTRA_USERS_PARAMS`. 267 It was being implemented using a custom patch to the ``shadow`` recipe 268 which clashed with a ``-P`` option that was added upstream in 269 ``shadow`` version 4.9, and in any case is fundamentally insecure. 270 Hardcoded passwords are still supported but they need to be hashed, see 271 examples in :term:`EXTRA_USERS_PARAMS`. 272 273 274