1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Debugging Tools and Techniques
4******************************
5
6The exact method for debugging build failures depends on the nature of
7the problem and on the system's area from which the bug originates.
8Standard debugging practices such as comparison against the last known
9working version with examination of the changes and the re-application
10of steps to identify the one causing the problem are valid for the Yocto
11Project just as they are for any other system. Even though it is
12impossible to detail every possible potential failure, this section
13provides some general tips to aid in debugging given a variety of
14situations.
15
16.. note::
17
18   A useful feature for debugging is the error reporting tool.
19   Configuring the Yocto Project to use this tool causes the
20   OpenEmbedded build system to produce error reporting commands as part
21   of the console output. You can enter the commands after the build
22   completes to log error information into a common database, that can
23   help you figure out what might be going wrong. For information on how
24   to enable and use this feature, see the
25   ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`"
26   section.
27
28The following list shows the debugging topics in the remainder of this
29section:
30
31-  ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes
32   how to find and view logs from tasks that failed during the build
33   process.
34
35-  ":ref:`dev-manual/debugging:viewing variable values`" describes how to
36   use the BitBake ``-e`` option to examine variable values after a
37   recipe has been parsed.
38
39-  ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``"
40   describes how to use the ``oe-pkgdata-util`` utility to query
41   :term:`PKGDATA_DIR` and
42   display package-related information for built packages.
43
44-  ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`"
45   describes how to use the BitBake ``-g`` option to display recipe
46   dependency information used during the build.
47
48-  ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes
49   how to use the ``bitbake-dumpsig`` command in conjunction with key
50   subdirectories in the :term:`Build Directory` to determine variable
51   dependencies.
52
53-  ":ref:`dev-manual/debugging:running specific tasks`" describes
54   how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
55   to run specific tasks in the build chain. It can be useful to run
56   tasks "out-of-order" when trying isolate build issues.
57
58-  ":ref:`dev-manual/debugging:general BitBake problems`" describes how
59   to use BitBake's ``-D`` debug output option to reveal more about what
60   BitBake is doing during the build.
61
62-  ":ref:`dev-manual/debugging:building with no dependencies`"
63   describes how to use the BitBake ``-b`` option to build a recipe
64   while ignoring dependencies.
65
66-  ":ref:`dev-manual/debugging:recipe logging mechanisms`"
67   describes how to use the many recipe logging functions to produce
68   debugging output and report errors and warnings.
69
70-  ":ref:`dev-manual/debugging:debugging parallel make races`"
71   describes how to debug situations where the build consists of several
72   parts that are run simultaneously and when the output or result of
73   one part is not ready for use with a different part of the build that
74   depends on that output.
75
76-  ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`"
77   describes how to use GDB to allow you to examine running programs, which can
78   help you fix problems.
79
80-  ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`"
81   describes how to use GDB directly on target hardware for debugging.
82
83-  ":ref:`dev-manual/debugging:other debugging tips`" describes
84   miscellaneous debugging tips that can be useful.
85
86Viewing Logs from Failed Tasks
87==============================
88
89You can find the log for a task in the file
90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
91For example, the log for the
92:ref:`ref-tasks-compile` task of the
93QEMU minimal image for the x86 machine (``qemux86``) might be in
94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
95To see the commands :term:`BitBake` ran
96to generate a log, look at the corresponding ``run.do_``\ `taskname` file
97in the same directory.
98
99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
102when it ran. The symlinks always point to the files corresponding to the
103most recent run.
104
105Viewing Variable Values
106=======================
107
108Sometimes you need to know the value of a variable as a result of
109BitBake's parsing step. This could be because some unexpected behavior
110occurred in your project. Perhaps an attempt to :ref:`modify a variable
111<bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
112variables>` did not work out as expected.
113
114BitBake's ``-e`` option is used to display variable values after
115parsing. The following command displays the variable values after the
116configuration files (i.e. ``local.conf``, ``bblayers.conf``,
117``bitbake.conf`` and so forth) have been parsed::
118
119   $ bitbake -e
120
121The following command displays variable values after a specific recipe has
122been parsed. The variables include those from the configuration as well::
123
124   $ bitbake -e recipename
125
126.. note::
127
128   Each recipe has its own private set of variables (datastore).
129   Internally, after parsing the configuration, a copy of the resulting
130   datastore is made prior to parsing each recipe. This copying implies
131   that variables set in one recipe will not be visible to other
132   recipes.
133
134   Likewise, each task within a recipe gets a private datastore based on
135   the recipe datastore, which means that variables set within one task
136   will not be visible to other tasks.
137
138In the output of ``bitbake -e``, each variable is preceded by a
139description of how the variable got its value, including temporary
140values that were later overridden. This description also includes
141variable flags (varflags) set on the variable. The output can be very
142helpful during debugging.
143
144Variables that are exported to the environment are preceded by
145``export`` in the output of ``bitbake -e``. See the following example::
146
147   export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
148
149In addition to variable values, the output of the ``bitbake -e`` and
150``bitbake -e`` recipe commands includes the following information:
151
152-  The output starts with a tree listing all configuration files and
153   classes included globally, recursively listing the files they include
154   or inherit in turn. Much of the behavior of the OpenEmbedded build
155   system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
156   implemented in the :ref:`ref-classes-base` class and the
157   classes it inherits, rather than being built into BitBake itself.
158
159-  After the variable values, all functions appear in the output. For
160   shell functions, variables referenced within the function body are
161   expanded. If a function has been modified using overrides or using
162   override-style operators like ``:append`` and ``:prepend``, then the
163   final assembled function body appears in the output.
164
165Viewing Package Information with ``oe-pkgdata-util``
166====================================================
167
168You can use the ``oe-pkgdata-util`` command-line utility to query
169:term:`PKGDATA_DIR` and display
170various package-related information. When you use the utility, you must
171use it to view information on packages that have already been built.
172
173Here are a few of the available ``oe-pkgdata-util`` subcommands.
174
175.. note::
176
177   You can use the standard \* and ? globbing wildcards as part of
178   package names and paths.
179
180-  ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
181   that have been built, optionally limiting the match to packages that
182   match pattern.
183
184-  ``oe-pkgdata-util list-pkg-files package ...``: Lists the
185   files and directories contained in the given packages.
186
187   .. note::
188
189      A different way to view the contents of a package is to look at
190      the
191      ``${``\ :term:`WORKDIR`\ ``}/packages-split``
192      directory of the recipe that generates the package. This directory
193      is created by the
194      :ref:`ref-tasks-package` task
195      and has one subdirectory for each package the recipe generates,
196      which contains the files stored in that package.
197
198      If you want to inspect the ``${WORKDIR}/packages-split``
199      directory, make sure that :ref:`ref-classes-rm-work` is not
200      enabled when you build the recipe.
201
202-  ``oe-pkgdata-util find-path path ...``: Lists the names of
203   the packages that contain the given paths. For example, the following
204   tells us that ``/usr/share/man/man1/make.1`` is contained in the
205   ``make-doc`` package::
206
207      $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
208      make-doc: /usr/share/man/man1/make.1
209
210-  ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
211   of the recipes that produce the given packages.
212
213For more information on the ``oe-pkgdata-util`` command, use the help
214facility::
215
216   $ oe-pkgdata-util --help
217   $ oe-pkgdata-util subcommand --help
218
219Viewing Dependencies Between Recipes and Tasks
220==============================================
221
222Sometimes it can be hard to see why BitBake wants to build other recipes
223before the one you have specified. Dependency information can help you
224understand why a recipe is built.
225
226To generate dependency information for a recipe, run the following
227command::
228
229   $ bitbake -g recipename
230
231This command writes the following files in the current directory:
232
233-  ``pn-buildlist``: A list of recipes/targets involved in building
234   `recipename`. "Involved" here means that at least one task from the
235   recipe needs to run when building `recipename` from scratch. Targets
236   that are in
237   :term:`ASSUME_PROVIDED`
238   are not listed.
239
240-  ``task-depends.dot``: A graph showing dependencies between tasks.
241
242The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
243format and can be converted to images (e.g. using the ``dot`` tool from
244`Graphviz <https://www.graphviz.org/>`__).
245
246.. note::
247
248   -  DOT files use a plain text format. The graphs generated using the
249      ``bitbake -g`` command are often so large as to be difficult to
250      read without special pruning (e.g. with BitBake's ``-I`` option)
251      and processing. Despite the form and size of the graphs, the
252      corresponding ``.dot`` files can still be possible to read and
253      provide useful information.
254
255      As an example, the ``task-depends.dot`` file contains lines such
256      as the following::
257
258         "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
259
260      The above example line reveals that the
261      :ref:`ref-tasks-configure`
262      task in ``libxslt`` depends on the
263      :ref:`ref-tasks-populate_sysroot`
264      task in ``libxml2``, which is a normal
265      :term:`DEPENDS` dependency
266      between the two recipes.
267
268   -  For an example of how ``.dot`` files can be processed, see the
269      ``scripts/contrib/graph-tool`` Python script, which finds and
270      displays paths between graph nodes.
271
272You can use a different method to view dependency information by using
273either::
274
275   $ bitbake -g -u taskexp recipename
276
277or::
278
279   $ bitbake -g -u taskexp_ncurses recipename
280
281The ``-u taskdep`` option GUI window from which you can view build-time and
282runtime dependencies for the recipes involved in building recipename. The
283``-u taskexp_ncurses`` option uses ncurses instead of GTK to render the UI.
284
285Viewing Task Variable Dependencies
286==================================
287
288As mentioned in the
289":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`"
290section of the BitBake User Manual, BitBake tries to automatically determine
291what variables a task depends on so that it can rerun the task if any values of
292the variables change. This determination is usually reliable. However, if you
293do things like construct variable names at runtime, then you might have to
294manually declare dependencies on those variables using ``vardeps`` as described
295in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
296section of the BitBake User Manual.
297
298If you are unsure whether a variable dependency is being picked up
299automatically for a given task, you can list the variable dependencies
300BitBake has determined by doing the following:
301
302#. Build the recipe containing the task::
303
304   $ bitbake recipename
305
306#. Inside the :term:`STAMPS_DIR`
307   directory, find the signature data (``sigdata``) file that
308   corresponds to the task. The ``sigdata`` files contain a pickled
309   Python database of all the metadata that went into creating the input
310   checksum for the task. As an example, for the
311   :ref:`ref-tasks-fetch` task of the
312   ``db`` recipe, the ``sigdata`` file might be found in the following
313   location::
314
315      ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
316
317   For tasks that are accelerated through the shared state
318   (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
319   additional ``siginfo`` file is written into
320   :term:`SSTATE_DIR` along with
321   the cached task output. The ``siginfo`` files contain exactly the
322   same information as ``sigdata`` files.
323
324#. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
325   is an example::
326
327      $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
328
329   In the output of the above command, you will find a line like the
330   following, which lists all the (inferred) variable dependencies for
331   the task. This list also includes indirect dependencies from
332   variables depending on other variables, recursively::
333
334      Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch']
335
336   .. note::
337
338      Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
339      These functions in turn depend on the variables they reference.
340
341   The output of ``bitbake-dumpsig`` also includes the value each
342   variable had, a list of dependencies for each variable, and
343   :term:`BB_BASEHASH_IGNORE_VARS`
344   information.
345
346Debugging signature construction and unexpected task executions
347===============================================================
348
349There is a ``bitbake-diffsigs`` command for comparing two
350``siginfo`` or ``sigdata`` files. This command can be helpful when
351trying to figure out what changed between two versions of a task. If you
352call ``bitbake-diffsigs`` with just one file, the command behaves like
353``bitbake-dumpsig``.
354
355You can also use BitBake to dump out the signature construction
356information without executing tasks by using either of the following
357BitBake command-line options::
358
359   ‐‐dump-signatures=SIGNATURE_HANDLER
360   -S SIGNATURE_HANDLER
361
362
363.. note::
364
365   Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
366   dump only the signature or compare the dumped signature with the most recent one,
367   respectively. "printdiff" will try to establish the most recent
368   signature match (e.g. in the sstate cache) and then
369   compare the matched signatures to determine the stamps and delta
370   where these two stamp trees diverge. This can be used to determine why
371   tasks need to be re-run in situations where that is not expected.
372
373Using BitBake with either of these options causes BitBake to dump out
374``sigdata`` files in the ``stamps`` directory for every task it would
375have executed instead of building the specified target package.
376
377Viewing Metadata Used to Create the Input Signature of a Shared State Task
378==========================================================================
379
380Seeing what metadata went into creating the input signature of a shared
381state (sstate) task can be a useful debugging aid. This information is
382available in signature information (``siginfo``) files in
383:term:`SSTATE_DIR`. For
384information on how to view and interpret information in ``siginfo``
385files, see the
386":ref:`dev-manual/debugging:viewing task variable dependencies`" section.
387
388For conceptual information on shared state, see the
389":ref:`overview-manual/concepts:shared state`"
390section in the Yocto Project Overview and Concepts Manual.
391
392Invalidating Shared State to Force a Task to Run
393================================================
394
395The OpenEmbedded build system uses
396:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
397:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
398rebuilding tasks. Collectively, this scheme is known as "shared state
399code".
400
401As with all schemes, this one has some drawbacks. It is possible that
402you could make implicit changes to your code that the checksum
403calculations do not take into account. These implicit changes affect a
404task's output but do not trigger the shared state code into rebuilding a
405recipe. Consider an example during which a tool changes its output.
406Assume that the output of ``rpmdeps`` changes. The result of the change
407should be that all the ``package`` and ``package_write_rpm`` shared
408state cache items become invalid. However, because the change to the
409output is external to the code and therefore implicit, the associated
410shared state cache items do not become invalidated. In this case, the
411build process uses the cached items rather than running the task again.
412Obviously, these types of implicit changes can cause problems.
413
414To avoid these problems during the build, you need to understand the
415effects of any changes you make. Realize that changes you make directly
416to a function are automatically factored into the checksum calculation.
417Thus, these explicit changes invalidate the associated area of shared
418state cache. However, you need to be aware of any implicit changes that
419are not obvious changes to the code and could affect the output of a
420given task.
421
422When you identify an implicit change, you can easily take steps to
423invalidate the cache and force the tasks to run. The steps you can take
424are as simple as changing a function's comments in the source code. For
425example, to invalidate package shared state files, change the comment
426statements of
427:ref:`ref-tasks-package` or the
428comments of one of the functions it calls. Even though the change is
429purely cosmetic, it causes the checksum to be recalculated and forces
430the build system to run the task again.
431
432.. note::
433
434   For an example of a commit that makes a cosmetic change to invalidate
435   shared state, see this
436   :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
437
438Running Specific Tasks
439======================
440
441Any given recipe consists of a set of tasks. The standard BitBake
442behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
443:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
444:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
445:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
446such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
447wish to run a task that is not part of the default build chain, you can
448use the ``-c`` option in BitBake. Here is an example::
449
450   $ bitbake matchbox-desktop -c devshell
451
452The ``-c`` option respects task dependencies, which means that all other
453tasks (including tasks from other recipes) that the specified task
454depends on will be run before the task. Even when you manually specify a
455task to run with ``-c``, BitBake will only run the task if it considers
456it "out of date". See the
457":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
458section in the Yocto Project Overview and Concepts Manual for how
459BitBake determines whether a task is "out of date".
460
461If you want to force an up-to-date task to be rerun (e.g. because you
462made manual modifications to the recipe's
463:term:`WORKDIR` that you want to try
464out), then you can use the ``-f`` option.
465
466.. note::
467
468   The reason ``-f`` is never required when running the
469   :ref:`ref-tasks-devshell` task is because the
470   [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
471   variable flag is already set for the task.
472
473The following example shows one way you can use the ``-f`` option::
474
475   $ bitbake matchbox-desktop
476             .
477             .
478   make some changes to the source code in the work directory
479             .
480             .
481   $ bitbake matchbox-desktop -c compile -f
482   $ bitbake matchbox-desktop
483
484This sequence first builds and then recompiles ``matchbox-desktop``. The
485last command reruns all tasks (basically the packaging tasks) after the
486compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
487therefore understands that the other tasks also need to be run again.
488
489Another, shorter way to rerun a task and all
490:ref:`ref-manual/tasks:normal recipe build tasks`
491that depend on it is to use the ``-C`` option.
492
493.. note::
494
495   This option is upper-cased and is separate from the ``-c``
496   option, which is lower-cased.
497
498Using this option invalidates the given task and then runs the
499:ref:`ref-tasks-build` task, which is
500the default task if no task is given, and the tasks on which it depends.
501You could replace the final two commands in the previous example with
502the following single command::
503
504   $ bitbake matchbox-desktop -C compile
505
506Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
507the input checksum of the specified task. This tainting indirectly
508causes the task and its dependent tasks to be rerun through the normal
509task dependency mechanisms.
510
511.. note::
512
513   BitBake explicitly keeps track of which tasks have been tainted in
514   this fashion, and will print warnings such as the following for
515   builds involving such tasks:
516
517   .. code-block:: none
518
519      WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
520
521
522   The purpose of the warning is to let you know that the work directory
523   and build output might not be in the clean state they would be in for
524   a "normal" build, depending on what actions you took. To get rid of
525   such warnings, you can remove the work directory and rebuild the
526   recipe, as follows::
527
528      $ bitbake matchbox-desktop -c clean
529      $ bitbake matchbox-desktop
530
531
532You can view a list of tasks in a given package by running the
533:ref:`ref-tasks-listtasks` task as follows::
534
535   $ bitbake matchbox-desktop -c listtasks
536
537The results appear as output to the console and are also in
538the file ``${WORKDIR}/temp/log.do_listtasks``.
539
540General BitBake Problems
541========================
542
543You can see debug output from BitBake by using the ``-D`` option. The
544debug output gives more information about what BitBake is doing and the
545reason behind it. Each ``-D`` option you use increases the logging
546level. The most common usage is ``-DDD``.
547
548The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
549chose a certain version of a package or why BitBake picked a certain
550provider. This command could also help you in a situation where you
551think BitBake did something unexpected.
552
553Building with No Dependencies
554=============================
555
556To build a specific recipe (``.bb`` file), you can use the following
557command form::
558
559   $ bitbake -b somepath/somerecipe.bb
560
561This command form does
562not check for dependencies. Consequently, you should use it only when
563you know existing dependencies have been met.
564
565.. note::
566
567   You can also specify fragments of the filename. In this case, BitBake
568   checks for a unique match.
569
570Recipe Logging Mechanisms
571=========================
572
573The Yocto Project provides several logging functions for producing
574debugging output and reporting errors and warnings. For Python
575functions, the following logging functions are available. All of these functions
576log to ``${T}/log.do_``\ `task`, and can also log to standard output
577(stdout) with the right settings:
578
579-  ``bb.plain(msg)``: Writes msg as is to the log while also
580   logging to stdout.
581
582-  ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
583   stdout if BitBake is called with "-v".
584
585-  ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to
586   stdout if the log level is greater than or equal to level. See the
587   ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`"
588   option in the BitBake User Manual for more information.
589
590-  ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
591   logging to stdout.
592
593-  ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
594   logging to standard out (stdout).
595
596   .. note::
597
598      Calling this function does not cause the task to fail.
599
600-  ``bb.fatal(msg)``: This logging function is similar to
601   ``bb.error(msg)`` but also causes the calling task to fail.
602
603   .. note::
604
605      ``bb.fatal()`` raises an exception, which means you do not need to put a
606      "return" statement after the function.
607
608The same logging functions are also available in shell functions, under
609the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
610and ``bbfatal``. The :ref:`ref-classes-logging` class
611implements these functions. See that class in the ``meta/classes``
612folder of the :term:`Source Directory` for information.
613
614Logging With Python
615-------------------
616
617When creating recipes using Python and inserting code that handles build
618logs, keep in mind the goal is to have informative logs while keeping
619the console as "silent" as possible. Also, if you want status messages
620in the log, use the "debug" loglevel.
621
622Here is an example written in Python. The code handles logging for
623a function that determines the number of tasks needed to be run. See the
624":ref:`ref-tasks-listtasks`"
625section for additional information::
626
627   python do_listtasks() {
628       bb.debug(2, "Starting to figure out the task list")
629       if noteworthy_condition:
630           bb.note("There are 47 tasks to run")
631       bb.debug(2, "Got to point xyz")
632       if warning_trigger:
633           bb.warn("Detected warning_trigger, this might be a problem later.")
634       if recoverable_error:
635           bb.error("Hit recoverable_error, you really need to fix this!")
636       if fatal_error:
637           bb.fatal("fatal_error detected, unable to print the task list")
638       bb.plain("The tasks present are abc")
639       bb.debug(2, "Finished figuring out the tasklist")
640   }
641
642Logging With Bash
643-----------------
644
645When creating recipes using Bash and inserting code that handles build
646logs, you have the same goals --- informative with minimal console output.
647The syntax you use for recipes written in Bash is similar to that of
648recipes written in Python described in the previous section.
649
650Here is an example written in Bash. The code logs the progress of
651the ``do_my_function`` function::
652
653   do_my_function() {
654       bbdebug 2 "Running do_my_function"
655       if [ exceptional_condition ]; then
656           bbnote "Hit exceptional_condition"
657       fi
658       bbdebug 2  "Got to point xyz"
659       if [ warning_trigger ]; then
660           bbwarn "Detected warning_trigger, this might cause a problem later."
661       fi
662       if [ recoverable_error ]; then
663           bberror "Hit recoverable_error, correcting"
664       fi
665       if [ fatal_error ]; then
666           bbfatal "fatal_error detected"
667       fi
668       bbdebug 2 "Completed do_my_function"
669   }
670
671
672Debugging Parallel Make Races
673=============================
674
675A parallel ``make`` race occurs when the build consists of several parts
676that are run simultaneously and a situation occurs when the output or
677result of one part is not ready for use with a different part of the
678build that depends on that output. Parallel make races are annoying and
679can sometimes be difficult to reproduce and fix. However, there are some simple
680tips and tricks that can help you debug and fix them. This section
681presents a real-world example of an error encountered on the Yocto
682Project autobuilder and the process used to fix it.
683
684.. note::
685
686   If you cannot properly fix a ``make`` race condition, you can work around it
687   by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
688   variables.
689
690The Failure
691-----------
692
693For this example, assume that you are building an image that depends on
694the "neard" package. And, during the build, BitBake runs into problems
695and creates the following output.
696
697.. note::
698
699   This example log file has longer lines artificially broken to make
700   the listing easier to read.
701
702If you examine the output or the log file, you see the failure during
703``make``:
704
705.. code-block:: none
706
707   | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
708   | DEBUG: Executing shell function do_compile
709   | NOTE: make -j 16
710   | make --no-print-directory all-am
711   | /bin/mkdir -p include/near
712   | /bin/mkdir -p include/near
713   | /bin/mkdir -p include/near
714   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
715     0.14-r0/neard-0.14/include/types.h include/near/types.h
716   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
717     0.14-r0/neard-0.14/include/log.h include/near/log.h
718   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
719     0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
720   | /bin/mkdir -p include/near
721   | /bin/mkdir -p include/near
722   | /bin/mkdir -p include/near
723   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
724     0.14-r0/neard-0.14/include/tag.h include/near/tag.h
725   | /bin/mkdir -p include/near
726   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
727     0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
728   | /bin/mkdir -p include/near
729   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
730     0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
731   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
732     0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
733   | /bin/mkdir -p include/near
734   | /bin/mkdir -p include/near
735   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
736     0.14-r0/neard-0.14/include/setting.h include/near/setting.h
737   | /bin/mkdir -p include/near
738   | /bin/mkdir -p include/near
739   | /bin/mkdir -p include/near
740   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
741     0.14-r0/neard-0.14/include/device.h include/near/device.h
742   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
743     0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
744   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
745     0.14-r0/neard-0.14/include/snep.h include/near/snep.h
746   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
747     0.14-r0/neard-0.14/include/version.h include/near/version.h
748   | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
749     0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
750   | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
751   | i586-poky-linux-gcc  -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
752     build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus  -I/home/pokybuild/
753     yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
754     -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
755     lib/glib-2.0/include  -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
756     tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
757     nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include  -I/home/pokybuild/yocto-autobuilder/
758     nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
759     -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
760     -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
761     -o tools/snep-send.o tools/snep-send.c
762   | In file included from tools/snep-send.c:16:0:
763   | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
764   |  #include <near/dbus.h>
765   |                        ^
766   | compilation terminated.
767   | make[1]: *** [tools/snep-send.o] Error 1
768   | make[1]: *** Waiting for unfinished jobs....
769   | make: *** [all] Error 2
770   | ERROR: oe_runmake failed
771
772Reproducing the Error
773---------------------
774
775Because race conditions are intermittent, they do not manifest
776themselves every time you do the build. In fact, most times the build
777will complete without problems even though the potential race condition
778exists. Thus, once the error surfaces, you need a way to reproduce it.
779
780In this example, compiling the "neard" package is causing the problem.
781So the first thing to do is build "neard" locally. Before you start the
782build, set the
783:term:`PARALLEL_MAKE` variable
784in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
785high value for :term:`PARALLEL_MAKE` increases the chances of the race
786condition showing up::
787
788   $ bitbake neard
789
790Once the local build for "neard" completes, start a ``devshell`` build::
791
792   $ bitbake neard -c devshell
793
794For information on how to use a ``devshell``, see the
795":ref:`dev-manual/development-shell:using a development shell`" section.
796
797In the ``devshell``, do the following::
798
799   $ make clean
800   $ make tools/snep-send.o
801
802The ``devshell`` commands cause the failure to clearly
803be visible. In this case, there is a missing dependency for the ``neard``
804Makefile target. Here is some abbreviated, sample output with the
805missing dependency clearly visible at the end::
806
807   i586-poky-linux-gcc  -m32 -march=i586 --sysroot=/home/scott-lenovo/......
808      .
809      .
810      .
811   tools/snep-send.c
812   In file included from tools/snep-send.c:16:0:
813   tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
814    #include <near/dbus.h>
815                     ^
816   compilation terminated.
817   make: *** [tools/snep-send.o] Error 1
818   $
819
820
821Creating a Patch for the Fix
822----------------------------
823
824Because there is a missing dependency for the Makefile target, you need
825to patch the ``Makefile.am`` file, which is generated from
826``Makefile.in``. You can use Quilt to create the patch::
827
828   $ quilt new parallelmake.patch
829   Patch patches/parallelmake.patch is now on top
830   $ quilt add Makefile.am
831   File Makefile.am added to patch patches/parallelmake.patch
832
833For more information on using Quilt, see the
834":ref:`dev-manual/quilt:using quilt in your workflow`" section.
835
836At this point you need to make the edits to ``Makefile.am`` to add the
837missing dependency. For our example, you have to add the following line
838to the file::
839
840   tools/snep-send.$(OBJEXT): include/near/dbus.h
841
842Once you have edited the file, use the ``refresh`` command to create the
843patch::
844
845   $ quilt refresh
846   Refreshed patch patches/parallelmake.patch
847
848Once the patch file is created, you need to add it back to the originating
849recipe folder. Here is an example assuming a top-level
850:term:`Source Directory` named ``poky``::
851
852   $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
853
854The final thing you need to do to implement the fix in the build is to
855update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
856:term:`SRC_URI` statement includes
857the patch file. The recipe file is in the folder above the patch. Here
858is what the edited :term:`SRC_URI` statement would look like::
859
860   SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
861              file://neard.in \
862              file://neard.service.in \
863              file://parallelmake.patch \
864             "
865
866With the patch complete and moved to the correct folder and the
867:term:`SRC_URI` statement updated, you can exit the ``devshell``::
868
869   $ exit
870
871Testing the Build
872-----------------
873
874With everything in place, you can get back to trying the build again
875locally::
876
877   $ bitbake neard
878
879This build should succeed.
880
881Now you can open up a ``devshell`` again and repeat the clean and make
882operations as follows::
883
884   $ bitbake neard -c devshell
885   $ make clean
886   $ make tools/snep-send.o
887
888The build should work without issue.
889
890As with all solved problems, if they originated upstream, you need to
891submit the fix for the recipe in OE-Core and upstream so that the
892problem is taken care of at its source. See the
893":doc:`../contributor-guide/submit-changes`" section for more information.
894
895Debugging With the GNU Project Debugger (GDB) Remotely
896======================================================
897
898GDB allows you to examine running programs, which in turn helps you to
899understand and fix problems. It also allows you to perform post-mortem
900style analysis of program crashes. GDB is available as a package within
901the Yocto Project and is installed in SDK images by default. See the
902":ref:`ref-manual/images:Images`" chapter in the Yocto
903Project Reference Manual for a description of these images. You can find
904information on GDB at https://sourceware.org/gdb/.
905
906.. note::
907
908   For best results, install debug (``-dbg``) packages for the applications you
909   are going to debug. Doing so makes extra debug symbols available that give
910   you more meaningful output.
911
912Sometimes, due to memory or disk space constraints, it is not possible
913to use GDB directly on the remote target to debug applications. These
914constraints arise because GDB needs to load the debugging information
915and the binaries of the process being debugged. Additionally, GDB needs
916to perform many computations to locate information such as function
917names, variable names and values, stack traces and so forth --- even
918before starting the debugging process. These extra computations place
919more load on the target system and can alter the characteristics of the
920program being debugged.
921
922To help get past the previously mentioned constraints, there are two
923methods you can use: running a debuginfod server and using gdbserver.
924
925Using the debuginfod server method
926----------------------------------
927
928``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
929Running a ``debuginfod`` server makes debug symbols readily available,
930which means you don't need to download debugging information
931and the binaries of the process being debugged. You can just fetch
932debug symbols from the server.
933
934To run a ``debuginfod`` server, you need to do the following:
935
936-  Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
937   (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
938   If not, set in your distro config file or in ``local.conf``::
939
940      DISTRO_FEATURES:append = " debuginfod"
941
942   This distro feature enables the server and client library in ``elfutils``,
943   and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
944
945-  Run the following commands to launch the ``debuginfod`` server on the host::
946
947      $ oe-debuginfod
948
949-  To use ``debuginfod`` on the target, you need to know the ip:port where
950   ``debuginfod`` is listening on the host (port defaults to 8002), and export
951   that into the shell environment, for example in ``qemu``::
952
953      root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
954
955-  Then debug info fetching should simply work when running the target ``gdb``,
956   ``readelf`` or ``objdump``, for example::
957
958      root@qemux86-64:~# gdb /bin/cat
959      ...
960      Reading symbols from /bin/cat...
961      Downloading separate debug info for /bin/cat...
962      Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
963
964-  It's also possible to use ``debuginfod-find`` to just query the server::
965
966      root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
967      /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
968
969
970Using the gdbserver method
971--------------------------
972
973gdbserver, which runs on the remote target and does not load any
974debugging information from the debugged process. Instead, a GDB instance
975processes the debugging information that is run on a remote computer -
976the host GDB. The host GDB then sends control commands to gdbserver to
977make it stop or start the debugged program, as well as read or write
978memory regions of that debugged program. All the debugging information
979loaded and processed as well as all the heavy debugging is done by the
980host GDB. Offloading these processes gives the gdbserver running on the
981target a chance to remain small and fast.
982
983Because the host GDB is responsible for loading the debugging
984information and for doing the necessary processing to make actual
985debugging happen, you have to make sure the host can access the
986unstripped binaries complete with their debugging information and also
987be sure the target is compiled with no optimizations. The host GDB must
988also have local access to all the libraries used by the debugged
989program. Because gdbserver does not need any local debugging
990information, the binaries on the remote target can remain stripped.
991However, the binaries must also be compiled without optimization so they
992match the host's binaries.
993
994To remain consistent with GDB documentation and terminology, the binary
995being debugged on the remote target machine is referred to as the
996"inferior" binary. For documentation on GDB see the `GDB
997site <https://sourceware.org/gdb/documentation/>`__.
998
999The following steps show you how to debug using the GNU project
1000debugger.
1001
1002#. *Configure your build system to construct the companion debug
1003   filesystem:*
1004
1005   In your ``local.conf`` file, set the following::
1006
1007      IMAGE_GEN_DEBUGFS = "1"
1008      IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
1009
1010   These options cause the
1011   OpenEmbedded build system to generate a special companion filesystem
1012   fragment, which contains the matching source and debug symbols to
1013   your deployable filesystem. The build system does this by looking at
1014   what is in the deployed filesystem, and pulling the corresponding
1015   ``-dbg`` packages.
1016
1017   The companion debug filesystem is not a complete filesystem, but only
1018   contains the debug fragments. This filesystem must be combined with
1019   the full filesystem for debugging. Subsequent steps in this procedure
1020   show how to combine the partial filesystem with the full filesystem.
1021
1022#. *Configure the system to include gdbserver in the target filesystem:*
1023
1024   Make the following addition in your ``local.conf`` file::
1025
1026      EXTRA_IMAGE_FEATURES:append = " tools-debug"
1027
1028   The change makes
1029   sure the ``gdbserver`` package is included.
1030
1031#. *Build the environment:*
1032
1033   Use the following command to construct the image and the companion
1034   Debug Filesystem::
1035
1036      $ bitbake image
1037
1038   Build the cross GDB component and
1039   make it available for debugging. Build the SDK that matches the
1040   image. Building the SDK is best for a production build that can be
1041   used later for debugging, especially during long term maintenance::
1042
1043      $ bitbake -c populate_sdk image
1044
1045   Alternatively, you can build the minimal toolchain components that
1046   match the target. Doing so creates a smaller than typical SDK and
1047   only contains a minimal set of components with which to build simple
1048   test applications, as well as run the debugger::
1049
1050      $ bitbake meta-toolchain
1051
1052   A final method is to build Gdb itself within the build system::
1053
1054      $ bitbake gdb-cross-<architecture>
1055
1056   Doing so produces a temporary copy of
1057   ``cross-gdb`` you can use for debugging during development. While
1058   this is the quickest approach, the two previous methods in this step
1059   are better when considering long-term maintenance strategies.
1060
1061   .. note::
1062
1063      If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
1064      the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
1065      actual name you want to use.
1066
1067#. *Set up the* ``debugfs``\ *:*
1068
1069   Run the following commands to set up the ``debugfs``::
1070
1071      $ mkdir debugfs
1072      $ cd debugfs
1073      $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
1074      $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
1075
1076#. *Set up GDB:*
1077
1078   Install the SDK (if you built one) and then source the correct
1079   environment file. Sourcing the environment file puts the SDK in your
1080   ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
1081
1082   If you are using the build system, Gdb is located in
1083   `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
1084
1085#. *Boot the target:*
1086
1087   For information on how to run QEMU, see the `QEMU
1088   Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
1089
1090   .. note::
1091
1092      Be sure to verify that your host can access the target via TCP.
1093
1094#. *Debug a program:*
1095
1096   Debugging a program involves running gdbserver on the target and then
1097   running Gdb on the host. The example in this step debugs ``gzip``:
1098
1099   .. code-block:: shell
1100
1101      root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
1102
1103   For
1104   additional gdbserver options, see the `GDB Server
1105   Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
1106
1107   After running gdbserver on the target, you need to run Gdb on the
1108   host and configure it and connect to the target. Use these commands::
1109
1110      $ cd directory-holding-the-debugfs-directory
1111      $ arch-gdb
1112      (gdb) set sysroot debugfs
1113      (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
1114      (gdb) target remote IP-of-target:1234
1115
1116   At this
1117   point, everything should automatically load (i.e. matching binaries,
1118   symbols and headers).
1119
1120   .. note::
1121
1122      The Gdb ``set`` commands in the previous example can be placed into the
1123      users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
1124      commands are in that file.
1125
1126#. *Deploying without a full image rebuild:*
1127
1128   In many cases, during development you want a quick method to deploy a
1129   new binary to the target and debug it, without waiting for a full
1130   image build.
1131
1132   One approach to solving this situation is to just build the component
1133   you want to debug. Once you have built the component, copy the
1134   executable directly to both the target and the host ``debugfs``.
1135
1136   If the binary is processed through the debug splitting in
1137   OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
1138   contents and corresponding ``/usr/src/debug`` files) from the work
1139   directory. Here is an example::
1140
1141      $ bitbake bash
1142      $ bitbake -c devshell bash
1143      $ cd ..
1144      $ scp packages-split/bash/bin/bash target:/bin/bash
1145      $ cp -a packages-split/bash-dbg/\* path/debugfs
1146
1147Debugging with the GNU Project Debugger (GDB) on the Target
1148===========================================================
1149
1150The previous section addressed using GDB remotely for debugging
1151purposes, which is the most usual case due to the inherent hardware
1152limitations on many embedded devices. However, debugging in the target
1153hardware itself is also possible with more powerful devices. This
1154section describes what you need to do in order to support using GDB to
1155debug on the target hardware.
1156
1157To support this kind of debugging, you need do the following:
1158
1159-  Ensure that GDB is on the target. You can do this by making
1160   the following addition to your ``local.conf`` file::
1161
1162      EXTRA_IMAGE_FEATURES:append = " tools-debug"
1163
1164-  Ensure that debug symbols are present. You can do so by adding the
1165   corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
1166
1167      IMAGE_INSTALL:append = " packagename-dbg"
1168
1169   Alternatively, you can add the following to ``local.conf`` to include
1170   all the debug symbols::
1171
1172      EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
1173
1174.. note::
1175
1176   To improve the debug information accuracy, you can reduce the level
1177   of optimization used by the compiler. For example, when adding the
1178   following line to your ``local.conf`` file, you will reduce optimization
1179   from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
1180   of "-O -fno-omit-frame-pointer"::
1181
1182           DEBUG_BUILD = "1"
1183
1184   Consider that this will reduce the application's performance and is
1185   recommended only for debugging purposes.
1186
1187Enabling Minidebuginfo
1188======================
1189
1190Enabling the :term:`DISTRO_FEATURES` minidebuginfo adds a compressed ELF section ``.gnu_debugdata``
1191to all binary files, containing only function names, and thus increasing the size of the
1192binaries only by 5 to 10%. For comparison, full debug symbols can be 10 times as big as
1193a stripped binary, and it is thus not always possible to deploy full debug symbols.
1194Minidebuginfo data allows, on the one side, to retrieve a call-stack using
1195GDB (command backtrace) without deploying full debug symbols to the target. It also
1196allows to retrieve a symbolicated call-stack when using ``systemd-coredump`` to manage
1197coredumps (commands ``coredumpctl list`` and ``coredumpctl info``).
1198
1199This feature was created by Fedora, see https://fedoraproject.org/wiki/Features/MiniDebugInfo for
1200more details.
1201
1202Other Debugging Tips
1203====================
1204
1205Here are some other tips that you might find useful:
1206
1207-  When adding new packages, it is worth watching for undesirable items
1208   making their way into compiler command lines. For example, you do not
1209   want references to local system files like ``/usr/lib/`` or
1210   ``/usr/include/``.
1211
1212-  If you want to remove the ``psplash`` boot splashscreen, add
1213   ``psplash=false`` to the kernel command line. Doing so prevents
1214   ``psplash`` from loading and thus allows you to see the console. It
1215   is also possible to switch out of the splashscreen by switching the
1216   virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
1217
1218-  Removing :term:`TMPDIR` (usually ``tmp/``, within the
1219   :term:`Build Directory`) can often fix temporary build issues. Removing
1220   :term:`TMPDIR` is usually a relatively cheap operation, because task output
1221   will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
1222   also in the :term:`Build Directory`).
1223
1224   .. note::
1225
1226      Removing :term:`TMPDIR` might be a workaround rather than a fix.
1227      Consequently, trying to determine the underlying cause of an issue before
1228      removing the directory is a good idea.
1229
1230-  Understanding how a feature is used in practice within existing
1231   recipes can be very helpful. It is recommended that you configure
1232   some method that allows you to quickly search through files.
1233
1234   Using GNU Grep, you can use the following shell function to
1235   recursively search through common recipe-related files, skipping
1236   binary files, ``.git`` directories, and the :term:`Build Directory`
1237   (assuming its name starts with "build")::
1238
1239      g() {
1240          grep -Ir \
1241               --exclude-dir=.git \
1242               --exclude-dir='build*' \
1243               --include='*.bb*' \
1244               --include='*.inc*' \
1245               --include='*.conf*' \
1246               --include='*.py*' \
1247               "$@"
1248      }
1249
1250   Here are some usage examples::
1251
1252      $ g FOO # Search recursively for "FOO"
1253      $ g -i foo # Search recursively for "foo", ignoring case
1254      $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1255
1256   If figuring
1257   out how some feature works requires a lot of searching, it might
1258   indicate that the documentation should be extended or improved. In
1259   such cases, consider filing a documentation bug using the Yocto
1260   Project implementation of
1261   :yocto_bugs:`Bugzilla <>`. For information on
1262   how to submit a bug against the Yocto Project, see the Yocto Project
1263   Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
1264   and the ":doc:`../contributor-guide/report-defect`" section.
1265
1266   .. note::
1267
1268      The manuals might not be the right place to document variables
1269      that are purely internal and have a limited scope (e.g. internal
1270      variables used to implement a single ``.bbclass`` file).
1271
1272