1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3************************ 4Advanced Kernel Concepts 5************************ 6 7Yocto Project Kernel Development and Maintenance 8================================================ 9 10Kernels available through the Yocto Project (Yocto Linux kernels), like 11other kernels, are based off the Linux kernel releases from 12https://www.kernel.org. At the beginning of a major Linux kernel 13development cycle, the Yocto Project team chooses a Linux kernel based 14on factors such as release timing, the anticipated release timing of 15final upstream ``kernel.org`` versions, and Yocto Project feature 16requirements. Typically, the Linux kernel chosen is in the final stages 17of development by the Linux community. In other words, the Linux kernel 18is in the release candidate or "rc" phase and has yet to reach final 19release. But, by being in the final stages of external development, the 20team knows that the ``kernel.org`` final release will clearly be within 21the early stages of the Yocto Project development window. 22 23This balance allows the Yocto Project team to deliver the most 24up-to-date Yocto Linux kernel possible, while still ensuring that the 25team has a stable official release for the baseline Linux kernel 26version. 27 28As implied earlier, the ultimate source for Yocto Linux kernels are 29released kernels from ``kernel.org``. In addition to a foundational 30kernel from ``kernel.org``, the available Yocto Linux kernels contain a 31mix of important new mainline developments, non-mainline developments 32(when no alternative exists), Board Support Package (BSP) developments, 33and custom features. These additions result in a commercially released 34Yocto Project Linux kernel that caters to specific embedded designer 35needs for targeted hardware. 36 37You can find a web interface to the Yocto Linux kernels in the 38:ref:`overview-manual/development-environment:yocto project source repositories` 39at :yocto_git:`/`. If you look at the interface, you will see to 40the left a grouping of Git repositories titled "Yocto Linux Kernel". 41Within this group, you will find several Linux Yocto kernels developed 42and included with Yocto Project releases: 43 44- *linux-yocto-4.1:* The stable Yocto Project kernel to use with 45 the Yocto Project Release 2.0. This kernel is based on the Linux 4.1 46 released kernel. 47 48- *linux-yocto-4.4:* The stable Yocto Project kernel to use with 49 the Yocto Project Release 2.1. This kernel is based on the Linux 4.4 50 released kernel. 51 52- *linux-yocto-4.6:* A temporary kernel that is not tied to any 53 Yocto Project release. 54 55- *linux-yocto-4.8:* The stable yocto Project kernel to use with 56 the Yocto Project Release 2.2. 57 58- *linux-yocto-4.9:* The stable Yocto Project kernel to use with 59 the Yocto Project Release 2.3. This kernel is based on the Linux 4.9 60 released kernel. 61 62- *linux-yocto-4.10:* The default stable Yocto Project kernel to 63 use with the Yocto Project Release 2.3. This kernel is based on the 64 Linux 4.10 released kernel. 65 66- *linux-yocto-4.12:* The default stable Yocto Project kernel to 67 use with the Yocto Project Release 2.4. This kernel is based on the 68 Linux 4.12 released kernel. 69 70- *yocto-kernel-cache:* The ``linux-yocto-cache`` contains patches 71 and configurations for the linux-yocto kernel tree. This repository 72 is useful when working on the linux-yocto kernel. For more 73 information on this "Advanced Kernel Metadata", see the 74 ":doc:`/kernel-dev/advanced`" Chapter. 75 76- *linux-yocto-dev:* A development kernel based on the latest 77 upstream release candidate available. 78 79.. note:: 80 81 Long Term Support Initiative (LTSI) for Yocto Linux kernels is as 82 follows: 83 84 - For Yocto Project releases 1.7, 1.8, and 2.0, the LTSI kernel is 85 ``linux-yocto-3.14``. 86 87 - For Yocto Project releases 2.1, 2.2, and 2.3, the LTSI kernel is 88 ``linux-yocto-4.1``. 89 90 - For Yocto Project release 2.4, the LTSI kernel is 91 ``linux-yocto-4.9`` 92 93 - ``linux-yocto-4.4`` is an LTS kernel. 94 95Once a Yocto Linux kernel is officially released, the Yocto Project team 96goes into their next development cycle, or upward revision (uprev) 97cycle, while still continuing maintenance on the released kernel. It is 98important to note that the most sustainable and stable way to include 99feature development upstream is through a kernel uprev process. 100Back-porting hundreds of individual fixes and minor features from 101various kernel versions is not sustainable and can easily compromise 102quality. 103 104During the uprev cycle, the Yocto Project team uses an ongoing analysis 105of Linux kernel development, BSP support, and release timing to select 106the best possible ``kernel.org`` Linux kernel version on which to base 107subsequent Yocto Linux kernel development. The team continually monitors 108Linux community kernel development to look for significant features of 109interest. The team does consider back-porting large features if they 110have a significant advantage. User or community demand can also trigger 111a back-port or creation of new functionality in the Yocto Project 112baseline kernel during the uprev cycle. 113 114Generally speaking, every new Linux kernel both adds features and 115introduces new bugs. These consequences are the basic properties of 116upstream Linux kernel development and are managed by the Yocto Project 117team's Yocto Linux kernel development strategy. It is the Yocto Project 118team's policy to not back-port minor features to the released Yocto 119Linux kernel. They only consider back-porting significant technological 120jumps --- and, that is done after a complete gap analysis. The reason 121for this policy is that back-porting any small to medium sized change 122from an evolving Linux kernel can easily create mismatches, 123incompatibilities and very subtle errors. 124 125The policies described in this section result in both a stable and a 126cutting edge Yocto Linux kernel that mixes forward ports of existing 127Linux kernel features and significant and critical new functionality. 128Forward porting Linux kernel functionality into the Yocto Linux kernels 129available through the Yocto Project can be thought of as a "micro 130uprev". The many "micro uprevs" produce a Yocto Linux kernel version 131with a mix of important new mainline, non-mainline, BSP developments and 132feature integrations. This Yocto Linux kernel gives insight into new 133features and allows focused amounts of testing to be done on the kernel, 134which prevents surprises when selecting the next major uprev. The 135quality of these cutting edge Yocto Linux kernels is evolving and the 136kernels are used in leading edge feature and BSP development. 137 138Yocto Linux Kernel Architecture and Branching Strategies 139======================================================== 140 141As mentioned earlier, a key goal of the Yocto Project is to present the 142developer with a kernel that has a clear and continuous history that is 143visible to the user. The architecture and mechanisms, in particular the 144branching strategies, used achieve that goal in a manner similar to 145upstream Linux kernel development in ``kernel.org``. 146 147You can think of a Yocto Linux kernel as consisting of a baseline Linux 148kernel with added features logically structured on top of the baseline. 149The features are tagged and organized by way of a branching strategy 150implemented by the Yocto Project team using the Source Code Manager 151(SCM) Git. 152 153.. note:: 154 155 - Git is the obvious SCM for meeting the Yocto Linux kernel 156 organizational and structural goals described in this section. Not 157 only is Git the SCM for Linux kernel development in ``kernel.org`` 158 but, Git continues to grow in popularity and supports many 159 different work flows, front-ends and management techniques. 160 161 - You can find documentation on Git at https://git-scm.com/doc. You can 162 also get an introduction to Git as it applies to the Yocto Project in the 163 ":ref:`overview-manual/development-environment:git`" section in the Yocto Project 164 Overview and Concepts Manual. The latter reference provides an 165 overview of Git and presents a minimal set of Git commands that 166 allows you to be functional using Git. You can use as much, or as 167 little, of what Git has to offer to accomplish what you need for 168 your project. You do not have to be a "Git Expert" in order to use 169 it with the Yocto Project. 170 171Using Git's tagging and branching features, the Yocto Project team 172creates kernel branches at points where functionality is no longer 173shared and thus, needs to be isolated. For example, board-specific 174incompatibilities would require different functionality and would 175require a branch to separate the features. Likewise, for specific kernel 176features, the same branching strategy is used. 177 178This "tree-like" architecture results in a structure that has features 179organized to be specific for particular functionality, single kernel 180types, or a subset of kernel types. Thus, the user has the ability to 181see the added features and the commits that make up those features. In 182addition to being able to see added features, the user can also view the 183history of what made up the baseline Linux kernel. 184 185Another consequence of this strategy results in not having to store the 186same feature twice internally in the tree. Rather, the kernel team 187stores the unique differences required to apply the feature onto the 188kernel type in question. 189 190.. note:: 191 192 The Yocto Project team strives to place features in the tree such 193 that features can be shared by all boards and kernel types where 194 possible. However, during development cycles or when large features 195 are merged, the team cannot always follow this practice. In those 196 cases, the team uses isolated branches to merge features. 197 198BSP-specific code additions are handled in a similar manner to 199kernel-specific additions. Some BSPs only make sense given certain 200kernel types. So, for these types, the team creates branches off the end 201of that kernel type for all of the BSPs that are supported on that 202kernel type. From the perspective of the tools that create the BSP 203branch, the BSP is really no different than a feature. Consequently, the 204same branching strategy applies to BSPs as it does to kernel features. 205So again, rather than store the BSP twice, the team only stores the 206unique differences for the BSP across the supported multiple kernels. 207 208While this strategy can result in a tree with a significant number of 209branches, it is important to realize that from the developer's point of 210view, there is a linear path that travels from the baseline 211``kernel.org``, through a select group of features and ends with their 212BSP-specific commits. In other words, the divisions of the kernel are 213transparent and are not relevant to the developer on a day-to-day basis. 214From the developer's perspective, this path is the development branch. 215The developer does not need to be aware of the existence of 216any other branches at all. Of course, it can make sense to have these 217branches in the tree, should a person decide to explore them. For 218example, a comparison between two BSPs at either the commit level or at 219the line-by-line code ``diff`` level is now a trivial operation. 220 221The following illustration shows the conceptual Yocto Linux kernel. 222 223.. image:: figures/kernel-architecture-overview.png 224 :width: 100% 225 226In the illustration, the "Kernel.org Branch Point" marks the specific 227spot (or Linux kernel release) from which the Yocto Linux kernel is 228created. From this point forward in the tree, features and differences 229are organized and tagged. 230 231The "Yocto Project Baseline Kernel" contains functionality that is 232common to every kernel type and BSP that is organized further along in 233the tree. Placing these common features in the tree this way means 234features do not have to be duplicated along individual branches of the 235tree structure. 236 237From the "Yocto Project Baseline Kernel", branch points represent 238specific functionality for individual Board Support Packages (BSPs) as 239well as real-time kernels. The illustration represents this through 240three BSP-specific branches and a real-time kernel branch. Each branch 241represents some unique functionality for the BSP or for a real-time 242Yocto Linux kernel. 243 244In this example structure, the "Real-time (rt) Kernel" branch has common 245features for all real-time Yocto Linux kernels and contains more 246branches for individual BSP-specific real-time kernels. The illustration 247shows three branches as an example. Each branch points the way to 248specific, unique features for a respective real-time kernel as they 249apply to a given BSP. 250 251The resulting tree structure presents a clear path of markers (or 252branches) to the developer that, for all practical purposes, is the 253Yocto Linux kernel needed for any given set of requirements. 254 255.. note:: 256 257 Keep in mind the figure does not take into account all the supported 258 Yocto Linux kernels, but rather shows a single generic kernel just 259 for conceptual purposes. Also keep in mind that this structure 260 represents the 261 :ref:`overview-manual/development-environment:yocto project source repositories` 262 that are either pulled from during the build or established on the 263 host development system prior to the build by either cloning a 264 particular kernel's Git repository or by downloading and unpacking a 265 tarball. 266 267Working with the kernel as a structured tree follows recognized 268community best practices. In particular, the kernel as shipped with the 269product, should be considered an "upstream source" and viewed as a 270series of historical and documented modifications (commits). These 271modifications represent the development and stabilization done by the 272Yocto Project kernel development team. 273 274Because commits only change at significant release points in the product 275life cycle, developers can work on a branch created from the last 276relevant commit in the shipped Yocto Project Linux kernel. As mentioned 277previously, the structure is transparent to the developer because the 278kernel tree is left in this state after cloning and building the kernel. 279 280Kernel Build File Hierarchy 281=========================== 282 283Upstream storage of all the available kernel source code is one thing, 284while representing and using the code on your host development system is 285another. Conceptually, you can think of the kernel source repositories 286as all the source files necessary for all the supported Yocto Linux 287kernels. As a developer, you are just interested in the source files for 288the kernel on which you are working. And, furthermore, you need them 289available on your host system. 290 291Kernel source code is available on your host system several different 292ways: 293 294- *Files Accessed While using devtool:* ``devtool``, which is 295 available with the Yocto Project, is the preferred method by which to 296 modify the kernel. See the ":ref:`kernel-dev/intro:kernel modification workflow`" section. 297 298- *Cloned Repository:* If you are working in the kernel all the time, 299 you probably would want to set up your own local Git repository of 300 the Yocto Linux kernel tree. For information on how to clone a Yocto 301 Linux kernel Git repository, see the 302 ":ref:`kernel-dev/common:preparing the build host to work on the kernel`" 303 section. 304 305- *Temporary Source Files from a Build:* If you just need to make some 306 patches to the kernel using a traditional BitBake workflow (i.e. not 307 using the ``devtool``), you can access temporary kernel source files 308 that were extracted and used during a kernel build. 309 310The temporary kernel source files resulting from a build using BitBake 311have a particular hierarchy. When you build the kernel on your 312development system, all files needed for the build are taken from the 313source repositories pointed to by the 314:term:`SRC_URI` variable and gathered 315in a temporary work area where they are subsequently used to create the 316unique kernel. Thus, in a sense, the process constructs a local source 317tree specific to your kernel from which to generate the new kernel 318image. 319 320The following figure shows the temporary file structure created on your 321host system when you build the kernel using BitBake. This 322:term:`Build Directory` contains all the source files used during the build. 323 324.. image:: figures/kernel-overview-2-generic.png 325 :align: center 326 :width: 70% 327 328Again, for additional information on the Yocto Project kernel's 329architecture and its branching strategy, see the 330":ref:`kernel-dev/concepts-appx:yocto linux kernel architecture and branching strategies`" 331section. You can also reference the 332":ref:`kernel-dev/common:using \`\`devtool\`\` to patch the kernel`" 333and 334":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`" 335sections for detailed example that modifies the kernel. 336 337Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase 338======================================================================================= 339 340This section describes part of the kernel configuration audit phase that 341most developers can ignore. For general information on kernel 342configuration including ``menuconfig``, ``defconfig`` files, and 343configuration fragments, see the 344":ref:`kernel-dev/common:configuring the kernel`" section. 345 346During this part of the audit phase, the contents of the final 347``.config`` file are compared against the fragments specified by the 348system. These fragments can be system fragments, distro fragments, or 349user-specified configuration elements. Regardless of their origin, the 350OpenEmbedded build system warns the user if a specific option is not 351included in the final kernel configuration. 352 353By default, in order to not overwhelm the user with configuration 354warnings, the system only reports missing "hardware" options as they 355could result in a boot failure or indicate that important hardware is 356not available. 357 358To determine whether or not a given option is "hardware" or 359"non-hardware", the kernel Metadata in ``yocto-kernel-cache`` contains 360files that classify individual or groups of options as either hardware 361or non-hardware. To better show this, consider a situation where the 362``yocto-kernel-cache`` contains the following files:: 363 364 yocto-kernel-cache/features/drm-psb/hardware.cfg 365 yocto-kernel-cache/features/kgdb/hardware.cfg 366 yocto-kernel-cache/ktypes/base/hardware.cfg 367 yocto-kernel-cache/bsp/mti-malta32/hardware.cfg 368 yocto-kernel-cache/bsp/qemu-ppc32/hardware.cfg 369 yocto-kernel-cache/bsp/qemuarma9/hardware.cfg 370 yocto-kernel-cache/bsp/mti-malta64/hardware.cfg 371 yocto-kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg 372 yocto-kernel-cache/bsp/common-pc/hardware.cfg 373 yocto-kernel-cache/bsp/common-pc-64/hardware.cfg 374 yocto-kernel-cache/features/rfkill/non-hardware.cfg 375 yocto-kernel-cache/ktypes/base/non-hardware.cfg 376 yocto-kernel-cache/features/aufs/non-hardware.kcf 377 yocto-kernel-cache/features/ocf/non-hardware.kcf 378 yocto-kernel-cache/ktypes/base/non-hardware.kcf 379 yocto-kernel-cache/ktypes/base/hardware.kcf 380 yocto-kernel-cache/bsp/qemu-ppc32/hardware.kcf 381 382Here are explanations for the various files: 383 384- ``hardware.kcf``: Specifies a list of kernel Kconfig files that 385 contain hardware options only. 386 387- ``non-hardware.kcf``: Specifies a list of kernel Kconfig files that 388 contain non-hardware options only. 389 390- ``hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options that 391 are hardware, regardless of whether or not they are within a Kconfig 392 file specified by a hardware or non-hardware Kconfig file (i.e. 393 ``hardware.kcf`` or ``non-hardware.kcf``). 394 395- ``non-hardware.cfg``: Specifies a list of kernel ``CONFIG_`` options 396 that are not hardware, regardless of whether or not they are within a 397 Kconfig file specified by a hardware or non-hardware Kconfig file 398 (i.e. ``hardware.kcf`` or ``non-hardware.kcf``). 399 400Here is a specific example using the 401``kernel-cache/bsp/mti-malta32/hardware.cfg``:: 402 403 CONFIG_SERIAL_8250 404 CONFIG_SERIAL_8250_CONSOLE 405 CONFIG_SERIAL_8250_NR_UARTS 406 CONFIG_SERIAL_8250_PCI 407 CONFIG_SERIAL_CORE 408 CONFIG_SERIAL_CORE_CONSOLE 409 CONFIG_VGA_ARB 410 411The kernel configuration audit automatically detects 412these files (hence the names must be exactly the ones discussed here), 413and uses them as inputs when generating warnings about the final 414``.config`` file. 415 416A user-specified kernel Metadata repository, or recipe space feature, 417can use these same files to classify options that are found within its 418``.cfg`` files as hardware or non-hardware, to prevent the OpenEmbedded 419build system from producing an error or warning when an option is not in 420the final ``.config`` file. 421
