1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3Debugging Tools and Techniques 4****************************** 5 6The exact method for debugging build failures depends on the nature of 7the problem and on the system's area from which the bug originates. 8Standard debugging practices such as comparison against the last known 9working version with examination of the changes and the re-application 10of steps to identify the one causing the problem are valid for the Yocto 11Project just as they are for any other system. Even though it is 12impossible to detail every possible potential failure, this section 13provides some general tips to aid in debugging given a variety of 14situations. 15 16.. note:: 17 18 A useful feature for debugging is the error reporting tool. 19 Configuring the Yocto Project to use this tool causes the 20 OpenEmbedded build system to produce error reporting commands as part 21 of the console output. You can enter the commands after the build 22 completes to log error information into a common database, that can 23 help you figure out what might be going wrong. For information on how 24 to enable and use this feature, see the 25 ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`" 26 section. 27 28The following list shows the debugging topics in the remainder of this 29section: 30 31- ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes 32 how to find and view logs from tasks that failed during the build 33 process. 34 35- ":ref:`dev-manual/debugging:viewing variable values`" describes how to 36 use the BitBake ``-e`` option to examine variable values after a 37 recipe has been parsed. 38 39- ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``" 40 describes how to use the ``oe-pkgdata-util`` utility to query 41 :term:`PKGDATA_DIR` and 42 display package-related information for built packages. 43 44- ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`" 45 describes how to use the BitBake ``-g`` option to display recipe 46 dependency information used during the build. 47 48- ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes 49 how to use the ``bitbake-dumpsig`` command in conjunction with key 50 subdirectories in the :term:`Build Directory` to determine variable 51 dependencies. 52 53- ":ref:`dev-manual/debugging:running specific tasks`" describes 54 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``) 55 to run specific tasks in the build chain. It can be useful to run 56 tasks "out-of-order" when trying isolate build issues. 57 58- ":ref:`dev-manual/debugging:general BitBake problems`" describes how 59 to use BitBake's ``-D`` debug output option to reveal more about what 60 BitBake is doing during the build. 61 62- ":ref:`dev-manual/debugging:building with no dependencies`" 63 describes how to use the BitBake ``-b`` option to build a recipe 64 while ignoring dependencies. 65 66- ":ref:`dev-manual/debugging:recipe logging mechanisms`" 67 describes how to use the many recipe logging functions to produce 68 debugging output and report errors and warnings. 69 70- ":ref:`dev-manual/debugging:debugging parallel make races`" 71 describes how to debug situations where the build consists of several 72 parts that are run simultaneously and when the output or result of 73 one part is not ready for use with a different part of the build that 74 depends on that output. 75 76- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`" 77 describes how to use GDB to allow you to examine running programs, which can 78 help you fix problems. 79 80- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`" 81 describes how to use GDB directly on target hardware for debugging. 82 83- ":ref:`dev-manual/debugging:other debugging tips`" describes 84 miscellaneous debugging tips that can be useful. 85 86Viewing Logs from Failed Tasks 87============================== 88 89You can find the log for a task in the file 90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`. 91For example, the log for the 92:ref:`ref-tasks-compile` task of the 93QEMU minimal image for the x86 machine (``qemux86``) might be in 94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``. 95To see the commands :term:`BitBake` ran 96to generate a log, look at the corresponding ``run.do_``\ `taskname` file 97in the same directory. 98 99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic 100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and 101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had 102when it ran. The symlinks always point to the files corresponding to the 103most recent run. 104 105Viewing Variable Values 106======================= 107 108Sometimes you need to know the value of a variable as a result of 109BitBake's parsing step. This could be because some unexpected behavior 110occurred in your project. Perhaps an attempt to :ref:`modify a variable 111<bitbake-user-manual/bitbake-user-manual-metadata:modifying existing 112variables>` did not work out as expected. 113 114BitBake's ``-e`` option is used to display variable values after 115parsing. The following command displays the variable values after the 116configuration files (i.e. ``local.conf``, ``bblayers.conf``, 117``bitbake.conf`` and so forth) have been parsed:: 118 119 $ bitbake -e 120 121The following command displays variable values after a specific recipe has 122been parsed. The variables include those from the configuration as well:: 123 124 $ bitbake -e recipename 125 126.. note:: 127 128 Each recipe has its own private set of variables (datastore). 129 Internally, after parsing the configuration, a copy of the resulting 130 datastore is made prior to parsing each recipe. This copying implies 131 that variables set in one recipe will not be visible to other 132 recipes. 133 134 Likewise, each task within a recipe gets a private datastore based on 135 the recipe datastore, which means that variables set within one task 136 will not be visible to other tasks. 137 138In the output of ``bitbake -e``, each variable is preceded by a 139description of how the variable got its value, including temporary 140values that were later overridden. This description also includes 141variable flags (varflags) set on the variable. The output can be very 142helpful during debugging. 143 144Variables that are exported to the environment are preceded by 145``export`` in the output of ``bitbake -e``. See the following example:: 146 147 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" 148 149In addition to variable values, the output of the ``bitbake -e`` and 150``bitbake -e`` recipe commands includes the following information: 151 152- The output starts with a tree listing all configuration files and 153 classes included globally, recursively listing the files they include 154 or inherit in turn. Much of the behavior of the OpenEmbedded build 155 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is 156 implemented in the :ref:`ref-classes-base` class and the 157 classes it inherits, rather than being built into BitBake itself. 158 159- After the variable values, all functions appear in the output. For 160 shell functions, variables referenced within the function body are 161 expanded. If a function has been modified using overrides or using 162 override-style operators like ``:append`` and ``:prepend``, then the 163 final assembled function body appears in the output. 164 165Viewing Package Information with ``oe-pkgdata-util`` 166==================================================== 167 168You can use the ``oe-pkgdata-util`` command-line utility to query 169:term:`PKGDATA_DIR` and display 170various package-related information. When you use the utility, you must 171use it to view information on packages that have already been built. 172 173Following are a few of the available ``oe-pkgdata-util`` subcommands. 174 175.. note:: 176 177 You can use the standard \* and ? globbing wildcards as part of 178 package names and paths. 179 180- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages 181 that have been built, optionally limiting the match to packages that 182 match pattern. 183 184- ``oe-pkgdata-util list-pkg-files package ...``: Lists the 185 files and directories contained in the given packages. 186 187 .. note:: 188 189 A different way to view the contents of a package is to look at 190 the 191 ``${``\ :term:`WORKDIR`\ ``}/packages-split`` 192 directory of the recipe that generates the package. This directory 193 is created by the 194 :ref:`ref-tasks-package` task 195 and has one subdirectory for each package the recipe generates, 196 which contains the files stored in that package. 197 198 If you want to inspect the ``${WORKDIR}/packages-split`` 199 directory, make sure that :ref:`ref-classes-rm-work` is not 200 enabled when you build the recipe. 201 202- ``oe-pkgdata-util find-path path ...``: Lists the names of 203 the packages that contain the given paths. For example, the following 204 tells us that ``/usr/share/man/man1/make.1`` is contained in the 205 ``make-doc`` package:: 206 207 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 208 make-doc: /usr/share/man/man1/make.1 209 210- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name 211 of the recipes that produce the given packages. 212 213For more information on the ``oe-pkgdata-util`` command, use the help 214facility:: 215 216 $ oe-pkgdata-util --help 217 $ oe-pkgdata-util subcommand --help 218 219Viewing Dependencies Between Recipes and Tasks 220============================================== 221 222Sometimes it can be hard to see why BitBake wants to build other recipes 223before the one you have specified. Dependency information can help you 224understand why a recipe is built. 225 226To generate dependency information for a recipe, run the following 227command:: 228 229 $ bitbake -g recipename 230 231This command writes the following files in the current directory: 232 233- ``pn-buildlist``: A list of recipes/targets involved in building 234 `recipename`. "Involved" here means that at least one task from the 235 recipe needs to run when building `recipename` from scratch. Targets 236 that are in 237 :term:`ASSUME_PROVIDED` 238 are not listed. 239 240- ``task-depends.dot``: A graph showing dependencies between tasks. 241 242The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>` 243format and can be converted to images (e.g. using the ``dot`` tool from 244`Graphviz <https://www.graphviz.org/>`__). 245 246.. note:: 247 248 - DOT files use a plain text format. The graphs generated using the 249 ``bitbake -g`` command are often so large as to be difficult to 250 read without special pruning (e.g. with BitBake's ``-I`` option) 251 and processing. Despite the form and size of the graphs, the 252 corresponding ``.dot`` files can still be possible to read and 253 provide useful information. 254 255 As an example, the ``task-depends.dot`` file contains lines such 256 as the following:: 257 258 "libxslt.do_configure" -> "libxml2.do_populate_sysroot" 259 260 The above example line reveals that the 261 :ref:`ref-tasks-configure` 262 task in ``libxslt`` depends on the 263 :ref:`ref-tasks-populate_sysroot` 264 task in ``libxml2``, which is a normal 265 :term:`DEPENDS` dependency 266 between the two recipes. 267 268 - For an example of how ``.dot`` files can be processed, see the 269 ``scripts/contrib/graph-tool`` Python script, which finds and 270 displays paths between graph nodes. 271 272You can use a different method to view dependency information by using 273the following command:: 274 275 $ bitbake -g -u taskexp recipename 276 277This command 278displays a GUI window from which you can view build-time and runtime 279dependencies for the recipes involved in building recipename. 280 281Viewing Task Variable Dependencies 282================================== 283 284As mentioned in the 285":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" 286section of the BitBake User Manual, BitBake tries to automatically determine 287what variables a task depends on so that it can rerun the task if any values of 288the variables change. This determination is usually reliable. However, if you 289do things like construct variable names at runtime, then you might have to 290manually declare dependencies on those variables using ``vardeps`` as described 291in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" 292section of the BitBake User Manual. 293 294If you are unsure whether a variable dependency is being picked up 295automatically for a given task, you can list the variable dependencies 296BitBake has determined by doing the following: 297 298#. Build the recipe containing the task:: 299 300 $ bitbake recipename 301 302#. Inside the :term:`STAMPS_DIR` 303 directory, find the signature data (``sigdata``) file that 304 corresponds to the task. The ``sigdata`` files contain a pickled 305 Python database of all the metadata that went into creating the input 306 checksum for the task. As an example, for the 307 :ref:`ref-tasks-fetch` task of the 308 ``db`` recipe, the ``sigdata`` file might be found in the following 309 location:: 310 311 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 312 313 For tasks that are accelerated through the shared state 314 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an 315 additional ``siginfo`` file is written into 316 :term:`SSTATE_DIR` along with 317 the cached task output. The ``siginfo`` files contain exactly the 318 same information as ``sigdata`` files. 319 320#. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here 321 is an example:: 322 323 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 324 325 In the output of the above command, you will find a line like the 326 following, which lists all the (inferred) variable dependencies for 327 the task. This list also includes indirect dependencies from 328 variables depending on other variables, recursively:: 329 330 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch'] 331 332 .. note:: 333 334 Functions (e.g. ``base_do_fetch``) also count as variable dependencies. 335 These functions in turn depend on the variables they reference. 336 337 The output of ``bitbake-dumpsig`` also includes the value each 338 variable had, a list of dependencies for each variable, and 339 :term:`BB_BASEHASH_IGNORE_VARS` 340 information. 341 342There is also a ``bitbake-diffsigs`` command for comparing two 343``siginfo`` or ``sigdata`` files. This command can be helpful when 344trying to figure out what changed between two versions of a task. If you 345call ``bitbake-diffsigs`` with just one file, the command behaves like 346``bitbake-dumpsig``. 347 348You can also use BitBake to dump out the signature construction 349information without executing tasks by using either of the following 350BitBake command-line options:: 351 352 ‐‐dump-signatures=SIGNATURE_HANDLER 353 -S SIGNATURE_HANDLER 354 355 356.. note:: 357 358 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which 359 dump only the signature or compare the dumped signature with the cached one, 360 respectively. 361 362Using BitBake with either of these options causes BitBake to dump out 363``sigdata`` files in the ``stamps`` directory for every task it would 364have executed instead of building the specified target package. 365 366Viewing Metadata Used to Create the Input Signature of a Shared State Task 367========================================================================== 368 369Seeing what metadata went into creating the input signature of a shared 370state (sstate) task can be a useful debugging aid. This information is 371available in signature information (``siginfo``) files in 372:term:`SSTATE_DIR`. For 373information on how to view and interpret information in ``siginfo`` 374files, see the 375":ref:`dev-manual/debugging:viewing task variable dependencies`" section. 376 377For conceptual information on shared state, see the 378":ref:`overview-manual/concepts:shared state`" 379section in the Yocto Project Overview and Concepts Manual. 380 381Invalidating Shared State to Force a Task to Run 382================================================ 383 384The OpenEmbedded build system uses 385:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and 386:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily 387rebuilding tasks. Collectively, this scheme is known as "shared state 388code". 389 390As with all schemes, this one has some drawbacks. It is possible that 391you could make implicit changes to your code that the checksum 392calculations do not take into account. These implicit changes affect a 393task's output but do not trigger the shared state code into rebuilding a 394recipe. Consider an example during which a tool changes its output. 395Assume that the output of ``rpmdeps`` changes. The result of the change 396should be that all the ``package`` and ``package_write_rpm`` shared 397state cache items become invalid. However, because the change to the 398output is external to the code and therefore implicit, the associated 399shared state cache items do not become invalidated. In this case, the 400build process uses the cached items rather than running the task again. 401Obviously, these types of implicit changes can cause problems. 402 403To avoid these problems during the build, you need to understand the 404effects of any changes you make. Realize that changes you make directly 405to a function are automatically factored into the checksum calculation. 406Thus, these explicit changes invalidate the associated area of shared 407state cache. However, you need to be aware of any implicit changes that 408are not obvious changes to the code and could affect the output of a 409given task. 410 411When you identify an implicit change, you can easily take steps to 412invalidate the cache and force the tasks to run. The steps you can take 413are as simple as changing a function's comments in the source code. For 414example, to invalidate package shared state files, change the comment 415statements of 416:ref:`ref-tasks-package` or the 417comments of one of the functions it calls. Even though the change is 418purely cosmetic, it causes the checksum to be recalculated and forces 419the build system to run the task again. 420 421.. note:: 422 423 For an example of a commit that makes a cosmetic change to invalidate 424 shared state, see this 425 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`. 426 427Running Specific Tasks 428====================== 429 430Any given recipe consists of a set of tasks. The standard BitBake 431behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`, 432:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`, 433:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is 434:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks, 435such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you 436wish to run a task that is not part of the default build chain, you can 437use the ``-c`` option in BitBake. Here is an example:: 438 439 $ bitbake matchbox-desktop -c devshell 440 441The ``-c`` option respects task dependencies, which means that all other 442tasks (including tasks from other recipes) that the specified task 443depends on will be run before the task. Even when you manually specify a 444task to run with ``-c``, BitBake will only run the task if it considers 445it "out of date". See the 446":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`" 447section in the Yocto Project Overview and Concepts Manual for how 448BitBake determines whether a task is "out of date". 449 450If you want to force an up-to-date task to be rerun (e.g. because you 451made manual modifications to the recipe's 452:term:`WORKDIR` that you want to try 453out), then you can use the ``-f`` option. 454 455.. note:: 456 457 The reason ``-f`` is never required when running the 458 :ref:`ref-tasks-devshell` task is because the 459 [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ] 460 variable flag is already set for the task. 461 462The following example shows one way you can use the ``-f`` option:: 463 464 $ bitbake matchbox-desktop 465 . 466 . 467 make some changes to the source code in the work directory 468 . 469 . 470 $ bitbake matchbox-desktop -c compile -f 471 $ bitbake matchbox-desktop 472 473This sequence first builds and then recompiles ``matchbox-desktop``. The 474last command reruns all tasks (basically the packaging tasks) after the 475compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and 476therefore understands that the other tasks also need to be run again. 477 478Another, shorter way to rerun a task and all 479:ref:`ref-manual/tasks:normal recipe build tasks` 480that depend on it is to use the ``-C`` option. 481 482.. note:: 483 484 This option is upper-cased and is separate from the ``-c`` 485 option, which is lower-cased. 486 487Using this option invalidates the given task and then runs the 488:ref:`ref-tasks-build` task, which is 489the default task if no task is given, and the tasks on which it depends. 490You could replace the final two commands in the previous example with 491the following single command:: 492 493 $ bitbake matchbox-desktop -C compile 494 495Internally, the ``-f`` and ``-C`` options work by tainting (modifying) 496the input checksum of the specified task. This tainting indirectly 497causes the task and its dependent tasks to be rerun through the normal 498task dependency mechanisms. 499 500.. note:: 501 502 BitBake explicitly keeps track of which tasks have been tainted in 503 this fashion, and will print warnings such as the following for 504 builds involving such tasks: 505 506 .. code-block:: none 507 508 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run 509 510 511 The purpose of the warning is to let you know that the work directory 512 and build output might not be in the clean state they would be in for 513 a "normal" build, depending on what actions you took. To get rid of 514 such warnings, you can remove the work directory and rebuild the 515 recipe, as follows:: 516 517 $ bitbake matchbox-desktop -c clean 518 $ bitbake matchbox-desktop 519 520 521You can view a list of tasks in a given package by running the 522:ref:`ref-tasks-listtasks` task as follows:: 523 524 $ bitbake matchbox-desktop -c listtasks 525 526The results appear as output to the console and are also in 527the file ``${WORKDIR}/temp/log.do_listtasks``. 528 529General BitBake Problems 530======================== 531 532You can see debug output from BitBake by using the ``-D`` option. The 533debug output gives more information about what BitBake is doing and the 534reason behind it. Each ``-D`` option you use increases the logging 535level. The most common usage is ``-DDD``. 536 537The output from ``bitbake -DDD -v targetname`` can reveal why BitBake 538chose a certain version of a package or why BitBake picked a certain 539provider. This command could also help you in a situation where you 540think BitBake did something unexpected. 541 542Building with No Dependencies 543============================= 544 545To build a specific recipe (``.bb`` file), you can use the following 546command form:: 547 548 $ bitbake -b somepath/somerecipe.bb 549 550This command form does 551not check for dependencies. Consequently, you should use it only when 552you know existing dependencies have been met. 553 554.. note:: 555 556 You can also specify fragments of the filename. In this case, BitBake 557 checks for a unique match. 558 559Recipe Logging Mechanisms 560========================= 561 562The Yocto Project provides several logging functions for producing 563debugging output and reporting errors and warnings. For Python 564functions, the following logging functions are available. All of these functions 565log to ``${T}/log.do_``\ `task`, and can also log to standard output 566(stdout) with the right settings: 567 568- ``bb.plain(msg)``: Writes msg as is to the log while also 569 logging to stdout. 570 571- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to 572 stdout if BitBake is called with "-v". 573 574- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to 575 stdout if the log level is greater than or equal to level. See the 576 ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" 577 option in the BitBake User Manual for more information. 578 579- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also 580 logging to stdout. 581 582- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also 583 logging to standard out (stdout). 584 585 .. note:: 586 587 Calling this function does not cause the task to fail. 588 589- ``bb.fatal(msg)``: This logging function is similar to 590 ``bb.error(msg)`` but also causes the calling task to fail. 591 592 .. note:: 593 594 ``bb.fatal()`` raises an exception, which means you do not need to put a 595 "return" statement after the function. 596 597The same logging functions are also available in shell functions, under 598the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``, 599and ``bbfatal``. The :ref:`ref-classes-logging` class 600implements these functions. See that class in the ``meta/classes`` 601folder of the :term:`Source Directory` for information. 602 603Logging With Python 604------------------- 605 606When creating recipes using Python and inserting code that handles build 607logs, keep in mind the goal is to have informative logs while keeping 608the console as "silent" as possible. Also, if you want status messages 609in the log, use the "debug" loglevel. 610 611Following is an example written in Python. The code handles logging for 612a function that determines the number of tasks needed to be run. See the 613":ref:`ref-tasks-listtasks`" 614section for additional information:: 615 616 python do_listtasks() { 617 bb.debug(2, "Starting to figure out the task list") 618 if noteworthy_condition: 619 bb.note("There are 47 tasks to run") 620 bb.debug(2, "Got to point xyz") 621 if warning_trigger: 622 bb.warn("Detected warning_trigger, this might be a problem later.") 623 if recoverable_error: 624 bb.error("Hit recoverable_error, you really need to fix this!") 625 if fatal_error: 626 bb.fatal("fatal_error detected, unable to print the task list") 627 bb.plain("The tasks present are abc") 628 bb.debug(2, "Finished figuring out the tasklist") 629 } 630 631Logging With Bash 632----------------- 633 634When creating recipes using Bash and inserting code that handles build 635logs, you have the same goals --- informative with minimal console output. 636The syntax you use for recipes written in Bash is similar to that of 637recipes written in Python described in the previous section. 638 639Following is an example written in Bash. The code logs the progress of 640the ``do_my_function`` function:: 641 642 do_my_function() { 643 bbdebug 2 "Running do_my_function" 644 if [ exceptional_condition ]; then 645 bbnote "Hit exceptional_condition" 646 fi 647 bbdebug 2 "Got to point xyz" 648 if [ warning_trigger ]; then 649 bbwarn "Detected warning_trigger, this might cause a problem later." 650 fi 651 if [ recoverable_error ]; then 652 bberror "Hit recoverable_error, correcting" 653 fi 654 if [ fatal_error ]; then 655 bbfatal "fatal_error detected" 656 fi 657 bbdebug 2 "Completed do_my_function" 658 } 659 660 661Debugging Parallel Make Races 662============================= 663 664A parallel ``make`` race occurs when the build consists of several parts 665that are run simultaneously and a situation occurs when the output or 666result of one part is not ready for use with a different part of the 667build that depends on that output. Parallel make races are annoying and 668can sometimes be difficult to reproduce and fix. However, there are some simple 669tips and tricks that can help you debug and fix them. This section 670presents a real-world example of an error encountered on the Yocto 671Project autobuilder and the process used to fix it. 672 673.. note:: 674 675 If you cannot properly fix a ``make`` race condition, you can work around it 676 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST` 677 variables. 678 679The Failure 680----------- 681 682For this example, assume that you are building an image that depends on 683the "neard" package. And, during the build, BitBake runs into problems 684and creates the following output. 685 686.. note:: 687 688 This example log file has longer lines artificially broken to make 689 the listing easier to read. 690 691If you examine the output or the log file, you see the failure during 692``make``: 693 694.. code-block:: none 695 696 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] 697 | DEBUG: Executing shell function do_compile 698 | NOTE: make -j 16 699 | make --no-print-directory all-am 700 | /bin/mkdir -p include/near 701 | /bin/mkdir -p include/near 702 | /bin/mkdir -p include/near 703 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 704 0.14-r0/neard-0.14/include/types.h include/near/types.h 705 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 706 0.14-r0/neard-0.14/include/log.h include/near/log.h 707 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 708 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h 709 | /bin/mkdir -p include/near 710 | /bin/mkdir -p include/near 711 | /bin/mkdir -p include/near 712 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 713 0.14-r0/neard-0.14/include/tag.h include/near/tag.h 714 | /bin/mkdir -p include/near 715 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 716 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h 717 | /bin/mkdir -p include/near 718 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 719 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h 720 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 721 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h 722 | /bin/mkdir -p include/near 723 | /bin/mkdir -p include/near 724 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 725 0.14-r0/neard-0.14/include/setting.h include/near/setting.h 726 | /bin/mkdir -p include/near 727 | /bin/mkdir -p include/near 728 | /bin/mkdir -p include/near 729 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 730 0.14-r0/neard-0.14/include/device.h include/near/device.h 731 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 732 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h 733 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 734 0.14-r0/neard-0.14/include/snep.h include/near/snep.h 735 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 736 0.14-r0/neard-0.14/include/version.h include/near/version.h 737 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ 738 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h 739 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h 740 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/ 741 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/ 742 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0 743 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/ 744 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/ 745 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/ 746 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/ 747 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3 748 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\" 749 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c 750 -o tools/snep-send.o tools/snep-send.c 751 | In file included from tools/snep-send.c:16:0: 752 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory 753 | #include <near/dbus.h> 754 | ^ 755 | compilation terminated. 756 | make[1]: *** [tools/snep-send.o] Error 1 757 | make[1]: *** Waiting for unfinished jobs.... 758 | make: *** [all] Error 2 759 | ERROR: oe_runmake failed 760 761Reproducing the Error 762--------------------- 763 764Because race conditions are intermittent, they do not manifest 765themselves every time you do the build. In fact, most times the build 766will complete without problems even though the potential race condition 767exists. Thus, once the error surfaces, you need a way to reproduce it. 768 769In this example, compiling the "neard" package is causing the problem. 770So the first thing to do is build "neard" locally. Before you start the 771build, set the 772:term:`PARALLEL_MAKE` variable 773in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a 774high value for :term:`PARALLEL_MAKE` increases the chances of the race 775condition showing up:: 776 777 $ bitbake neard 778 779Once the local build for "neard" completes, start a ``devshell`` build:: 780 781 $ bitbake neard -c devshell 782 783For information on how to use a ``devshell``, see the 784":ref:`dev-manual/development-shell:using a development shell`" section. 785 786In the ``devshell``, do the following:: 787 788 $ make clean 789 $ make tools/snep-send.o 790 791The ``devshell`` commands cause the failure to clearly 792be visible. In this case, there is a missing dependency for the ``neard`` 793Makefile target. Here is some abbreviated, sample output with the 794missing dependency clearly visible at the end:: 795 796 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... 797 . 798 . 799 . 800 tools/snep-send.c 801 In file included from tools/snep-send.c:16:0: 802 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory 803 #include <near/dbus.h> 804 ^ 805 compilation terminated. 806 make: *** [tools/snep-send.o] Error 1 807 $ 808 809 810Creating a Patch for the Fix 811---------------------------- 812 813Because there is a missing dependency for the Makefile target, you need 814to patch the ``Makefile.am`` file, which is generated from 815``Makefile.in``. You can use Quilt to create the patch:: 816 817 $ quilt new parallelmake.patch 818 Patch patches/parallelmake.patch is now on top 819 $ quilt add Makefile.am 820 File Makefile.am added to patch patches/parallelmake.patch 821 822For more information on using Quilt, see the 823":ref:`dev-manual/quilt:using quilt in your workflow`" section. 824 825At this point you need to make the edits to ``Makefile.am`` to add the 826missing dependency. For our example, you have to add the following line 827to the file:: 828 829 tools/snep-send.$(OBJEXT): include/near/dbus.h 830 831Once you have edited the file, use the ``refresh`` command to create the 832patch:: 833 834 $ quilt refresh 835 Refreshed patch patches/parallelmake.patch 836 837Once the patch file is created, you need to add it back to the originating 838recipe folder. Here is an example assuming a top-level 839:term:`Source Directory` named ``poky``:: 840 841 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard 842 843The final thing you need to do to implement the fix in the build is to 844update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the 845:term:`SRC_URI` statement includes 846the patch file. The recipe file is in the folder above the patch. Here 847is what the edited :term:`SRC_URI` statement would look like:: 848 849 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ 850 file://neard.in \ 851 file://neard.service.in \ 852 file://parallelmake.patch \ 853 " 854 855With the patch complete and moved to the correct folder and the 856:term:`SRC_URI` statement updated, you can exit the ``devshell``:: 857 858 $ exit 859 860Testing the Build 861----------------- 862 863With everything in place, you can get back to trying the build again 864locally:: 865 866 $ bitbake neard 867 868This build should succeed. 869 870Now you can open up a ``devshell`` again and repeat the clean and make 871operations as follows:: 872 873 $ bitbake neard -c devshell 874 $ make clean 875 $ make tools/snep-send.o 876 877The build should work without issue. 878 879As with all solved problems, if they originated upstream, you need to 880submit the fix for the recipe in OE-Core and upstream so that the 881problem is taken care of at its source. See the 882":doc:`../contributor-guide/submit-changes`" section for more information. 883 884Debugging With the GNU Project Debugger (GDB) Remotely 885====================================================== 886 887GDB allows you to examine running programs, which in turn helps you to 888understand and fix problems. It also allows you to perform post-mortem 889style analysis of program crashes. GDB is available as a package within 890the Yocto Project and is installed in SDK images by default. See the 891":ref:`ref-manual/images:Images`" chapter in the Yocto 892Project Reference Manual for a description of these images. You can find 893information on GDB at https://sourceware.org/gdb/. 894 895.. note:: 896 897 For best results, install debug (``-dbg``) packages for the applications you 898 are going to debug. Doing so makes extra debug symbols available that give 899 you more meaningful output. 900 901Sometimes, due to memory or disk space constraints, it is not possible 902to use GDB directly on the remote target to debug applications. These 903constraints arise because GDB needs to load the debugging information 904and the binaries of the process being debugged. Additionally, GDB needs 905to perform many computations to locate information such as function 906names, variable names and values, stack traces and so forth --- even 907before starting the debugging process. These extra computations place 908more load on the target system and can alter the characteristics of the 909program being debugged. 910 911To help get past the previously mentioned constraints, there are two 912methods you can use: running a debuginfod server and using gdbserver. 913 914Using the debuginfod server method 915---------------------------------- 916 917``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files. 918Running a ``debuginfod`` server makes debug symbols readily available, 919which means you don't need to download debugging information 920and the binaries of the process being debugged. You can just fetch 921debug symbols from the server. 922 923To run a ``debuginfod`` server, you need to do the following: 924 925- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES` 926 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution). 927 If not, set in your distro config file or in ``local.conf``:: 928 929 DISTRO_FEATURES:append = " debuginfod" 930 931 This distro feature enables the server and client library in ``elfutils``, 932 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``). 933 934- Run the following commands to launch the ``debuginfod`` server on the host:: 935 936 $ oe-debuginfod 937 938- To use ``debuginfod`` on the target, you need to know the ip:port where 939 ``debuginfod`` is listening on the host (port defaults to 8002), and export 940 that into the shell environment, for example in ``qemu``:: 941 942 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/" 943 944- Then debug info fetching should simply work when running the target ``gdb``, 945 ``readelf`` or ``objdump``, for example:: 946 947 root@qemux86-64:~# gdb /bin/cat 948 ... 949 Reading symbols from /bin/cat... 950 Downloading separate debug info for /bin/cat... 951 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo... 952 953- It's also possible to use ``debuginfod-find`` to just query the server:: 954 955 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls 956 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo 957 958 959Using the gdbserver method 960-------------------------- 961 962gdbserver, which runs on the remote target and does not load any 963debugging information from the debugged process. Instead, a GDB instance 964processes the debugging information that is run on a remote computer - 965the host GDB. The host GDB then sends control commands to gdbserver to 966make it stop or start the debugged program, as well as read or write 967memory regions of that debugged program. All the debugging information 968loaded and processed as well as all the heavy debugging is done by the 969host GDB. Offloading these processes gives the gdbserver running on the 970target a chance to remain small and fast. 971 972Because the host GDB is responsible for loading the debugging 973information and for doing the necessary processing to make actual 974debugging happen, you have to make sure the host can access the 975unstripped binaries complete with their debugging information and also 976be sure the target is compiled with no optimizations. The host GDB must 977also have local access to all the libraries used by the debugged 978program. Because gdbserver does not need any local debugging 979information, the binaries on the remote target can remain stripped. 980However, the binaries must also be compiled without optimization so they 981match the host's binaries. 982 983To remain consistent with GDB documentation and terminology, the binary 984being debugged on the remote target machine is referred to as the 985"inferior" binary. For documentation on GDB see the `GDB 986site <https://sourceware.org/gdb/documentation/>`__. 987 988The following steps show you how to debug using the GNU project 989debugger. 990 991#. *Configure your build system to construct the companion debug 992 filesystem:* 993 994 In your ``local.conf`` file, set the following:: 995 996 IMAGE_GEN_DEBUGFS = "1" 997 IMAGE_FSTYPES_DEBUGFS = "tar.bz2" 998 999 These options cause the 1000 OpenEmbedded build system to generate a special companion filesystem 1001 fragment, which contains the matching source and debug symbols to 1002 your deployable filesystem. The build system does this by looking at 1003 what is in the deployed filesystem, and pulling the corresponding 1004 ``-dbg`` packages. 1005 1006 The companion debug filesystem is not a complete filesystem, but only 1007 contains the debug fragments. This filesystem must be combined with 1008 the full filesystem for debugging. Subsequent steps in this procedure 1009 show how to combine the partial filesystem with the full filesystem. 1010 1011#. *Configure the system to include gdbserver in the target filesystem:* 1012 1013 Make the following addition in your ``local.conf`` file:: 1014 1015 EXTRA_IMAGE_FEATURES:append = " tools-debug" 1016 1017 The change makes 1018 sure the ``gdbserver`` package is included. 1019 1020#. *Build the environment:* 1021 1022 Use the following command to construct the image and the companion 1023 Debug Filesystem:: 1024 1025 $ bitbake image 1026 1027 Build the cross GDB component and 1028 make it available for debugging. Build the SDK that matches the 1029 image. Building the SDK is best for a production build that can be 1030 used later for debugging, especially during long term maintenance:: 1031 1032 $ bitbake -c populate_sdk image 1033 1034 Alternatively, you can build the minimal toolchain components that 1035 match the target. Doing so creates a smaller than typical SDK and 1036 only contains a minimal set of components with which to build simple 1037 test applications, as well as run the debugger:: 1038 1039 $ bitbake meta-toolchain 1040 1041 A final method is to build Gdb itself within the build system:: 1042 1043 $ bitbake gdb-cross-<architecture> 1044 1045 Doing so produces a temporary copy of 1046 ``cross-gdb`` you can use for debugging during development. While 1047 this is the quickest approach, the two previous methods in this step 1048 are better when considering long-term maintenance strategies. 1049 1050 .. note:: 1051 1052 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests 1053 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the 1054 actual name you want to use. 1055 1056#. *Set up the* ``debugfs``\ *:* 1057 1058 Run the following commands to set up the ``debugfs``:: 1059 1060 $ mkdir debugfs 1061 $ cd debugfs 1062 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2 1063 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2 1064 1065#. *Set up GDB:* 1066 1067 Install the SDK (if you built one) and then source the correct 1068 environment file. Sourcing the environment file puts the SDK in your 1069 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger. 1070 1071 If you are using the build system, Gdb is located in 1072 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb`` 1073 1074#. *Boot the target:* 1075 1076 For information on how to run QEMU, see the `QEMU 1077 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__. 1078 1079 .. note:: 1080 1081 Be sure to verify that your host can access the target via TCP. 1082 1083#. *Debug a program:* 1084 1085 Debugging a program involves running gdbserver on the target and then 1086 running Gdb on the host. The example in this step debugs ``gzip``: 1087 1088 .. code-block:: shell 1089 1090 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help 1091 1092 For 1093 additional gdbserver options, see the `GDB Server 1094 Documentation <https://www.gnu.org/software/gdb/documentation/>`__. 1095 1096 After running gdbserver on the target, you need to run Gdb on the 1097 host and configure it and connect to the target. Use these commands:: 1098 1099 $ cd directory-holding-the-debugfs-directory 1100 $ arch-gdb 1101 (gdb) set sysroot debugfs 1102 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug 1103 (gdb) target remote IP-of-target:1234 1104 1105 At this 1106 point, everything should automatically load (i.e. matching binaries, 1107 symbols and headers). 1108 1109 .. note:: 1110 1111 The Gdb ``set`` commands in the previous example can be placed into the 1112 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever 1113 commands are in that file. 1114 1115#. *Deploying without a full image rebuild:* 1116 1117 In many cases, during development you want a quick method to deploy a 1118 new binary to the target and debug it, without waiting for a full 1119 image build. 1120 1121 One approach to solving this situation is to just build the component 1122 you want to debug. Once you have built the component, copy the 1123 executable directly to both the target and the host ``debugfs``. 1124 1125 If the binary is processed through the debug splitting in 1126 OpenEmbedded, you should also copy the debug items (i.e. ``.debug`` 1127 contents and corresponding ``/usr/src/debug`` files) from the work 1128 directory. Here is an example:: 1129 1130 $ bitbake bash 1131 $ bitbake -c devshell bash 1132 $ cd .. 1133 $ scp packages-split/bash/bin/bash target:/bin/bash 1134 $ cp -a packages-split/bash-dbg/\* path/debugfs 1135 1136Debugging with the GNU Project Debugger (GDB) on the Target 1137=========================================================== 1138 1139The previous section addressed using GDB remotely for debugging 1140purposes, which is the most usual case due to the inherent hardware 1141limitations on many embedded devices. However, debugging in the target 1142hardware itself is also possible with more powerful devices. This 1143section describes what you need to do in order to support using GDB to 1144debug on the target hardware. 1145 1146To support this kind of debugging, you need do the following: 1147 1148- Ensure that GDB is on the target. You can do this by making 1149 the following addition to your ``local.conf`` file:: 1150 1151 EXTRA_IMAGE_FEATURES:append = " tools-debug" 1152 1153- Ensure that debug symbols are present. You can do so by adding the 1154 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`:: 1155 1156 IMAGE_INSTALL:append = " packagename-dbg" 1157 1158 Alternatively, you can add the following to ``local.conf`` to include 1159 all the debug symbols:: 1160 1161 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs" 1162 1163.. note:: 1164 1165 To improve the debug information accuracy, you can reduce the level 1166 of optimization used by the compiler. For example, when adding the 1167 following line to your ``local.conf`` file, you will reduce optimization 1168 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION` 1169 of "-O -fno-omit-frame-pointer":: 1170 1171 DEBUG_BUILD = "1" 1172 1173 Consider that this will reduce the application's performance and is 1174 recommended only for debugging purposes. 1175 1176Other Debugging Tips 1177==================== 1178 1179Here are some other tips that you might find useful: 1180 1181- When adding new packages, it is worth watching for undesirable items 1182 making their way into compiler command lines. For example, you do not 1183 want references to local system files like ``/usr/lib/`` or 1184 ``/usr/include/``. 1185 1186- If you want to remove the ``psplash`` boot splashscreen, add 1187 ``psplash=false`` to the kernel command line. Doing so prevents 1188 ``psplash`` from loading and thus allows you to see the console. It 1189 is also possible to switch out of the splashscreen by switching the 1190 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus). 1191 1192- Removing :term:`TMPDIR` (usually ``tmp/``, within the 1193 :term:`Build Directory`) can often fix temporary build issues. Removing 1194 :term:`TMPDIR` is usually a relatively cheap operation, because task output 1195 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is 1196 also in the :term:`Build Directory`). 1197 1198 .. note:: 1199 1200 Removing :term:`TMPDIR` might be a workaround rather than a fix. 1201 Consequently, trying to determine the underlying cause of an issue before 1202 removing the directory is a good idea. 1203 1204- Understanding how a feature is used in practice within existing 1205 recipes can be very helpful. It is recommended that you configure 1206 some method that allows you to quickly search through files. 1207 1208 Using GNU Grep, you can use the following shell function to 1209 recursively search through common recipe-related files, skipping 1210 binary files, ``.git`` directories, and the :term:`Build Directory` 1211 (assuming its name starts with "build"):: 1212 1213 g() { 1214 grep -Ir \ 1215 --exclude-dir=.git \ 1216 --exclude-dir='build*' \ 1217 --include='*.bb*' \ 1218 --include='*.inc*' \ 1219 --include='*.conf*' \ 1220 --include='*.py*' \ 1221 "$@" 1222 } 1223 1224 Following are some usage examples:: 1225 1226 $ g FOO # Search recursively for "FOO" 1227 $ g -i foo # Search recursively for "foo", ignoring case 1228 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" 1229 1230 If figuring 1231 out how some feature works requires a lot of searching, it might 1232 indicate that the documentation should be extended or improved. In 1233 such cases, consider filing a documentation bug using the Yocto 1234 Project implementation of 1235 :yocto_bugs:`Bugzilla <>`. For information on 1236 how to submit a bug against the Yocto Project, see the Yocto Project 1237 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>` 1238 and the ":doc:`../contributor-guide/report-defect`" section. 1239 1240 .. note:: 1241 1242 The manuals might not be the right place to document variables 1243 that are purely internal and have a limited scope (e.g. internal 1244 variables used to implement a single ``.bbclass`` file). 1245 1246