xref: /openbmc/qemu/docs/devel/build-system.rst (revision 988f2442)
1==================================
2The QEMU build system architecture
3==================================
4
5This document aims to help developers understand the architecture of the
6QEMU build system. As with projects using GNU autotools, the QEMU build
7system has two stages; first the developer runs the "configure" script
8to determine the local build environment characteristics, then they run
9"make" to build the project.  This is about where the similarities with
10GNU autotools end, so try to forget what you know about them.
11
12The two general ways to perform a build are as follows:
13
14 - build artifacts outside of QEMU source tree entirely::
15
16     cd ../
17     mkdir build
18     cd build
19     ../qemu/configure
20     make
21
22 - build artifacts in a subdir of QEMU source tree::
23
24     mkdir build
25     cd build
26     ../configure
27     make
28
29Most of the actual build process uses Meson under the hood, therefore
30build artifacts cannot be placed in the source tree itself.
31
32
33Stage 1: configure
34==================
35
36The configure script has five tasks:
37
38 - detect the host architecture
39
40 - list the targets for which to build emulators; the list of
41   targets also affects which firmware binaries and tests to build
42
43 - find the compilers (native and cross) used to build executables,
44   firmware and tests.  The results are written as either Makefile
45   fragments (``config-host.mak``) or a Meson machine file
46   (``config-meson.cross``)
47
48 - create a virtual environment in which all Python code runs during
49   the build, and possibly install packages into it from PyPI
50
51 - invoke Meson in the virtual environment, to perform the actual
52   configuration step for the emulator build
53
54The configure script automatically recognizes command line options for
55which a same-named Meson option exists; dashes in the command line are
56replaced with underscores.
57
58Almost all QEMU developers that need to modify the build system will
59only be concerned with Meson, and therefore can skip the rest of this
60section.
61
62
63Modifying ``configure``
64-----------------------
65
66``configure`` is a shell script; it uses ``#!/bin/sh`` and therefore
67should be compatible with any POSIX shell. It is important to avoid
68using bash-isms to avoid breaking development platforms where bash is
69the primary host.
70
71The configure script provides a variety of functions to help writing
72portable shell code and providing consistent behavior across architectures
73and operating systems:
74
75``error_exit $MESSAGE $MORE...``
76   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
77   configure script with non-zero status.
78
79``has $COMMAND``
80   Determine if $COMMAND exists in the current environment, either as a
81   shell builtin, or executable binary, returning 0 on success.  The
82   replacement in Meson is ``find_program()``.
83
84``probe_target_compiler $TARGET``
85  Detect a cross compiler and cross tools for the QEMU target $TARGET (e.g.,
86  ``$CPU-softmmu``, ``$CPU-linux-user``, ``$CPU-bsd-user``).  If a working
87  compiler is present, return success and set variables ``$target_cc``,
88  ``$target_ar``, etc. to non-empty values.
89
90``write_target_makefile``
91  Write a Makefile fragment to stdout, exposing the result of the most
92  ``probe_target_compiler`` call as the usual Make variables (``CC``,
93  ``AR``, ``LD``, etc.).
94
95
96Configure does not generally perform tests for compiler options beyond
97basic checks to detect the host platform and ensure the compiler is
98functioning.  These are performed using a few more helper functions:
99
100``compile_object $CFLAGS``
101   Attempt to compile a test program with the system C compiler using
102   $CFLAGS. The test program must have been previously written to a file
103   called $TMPC.
104
105``compile_prog $CFLAGS $LDFLAGS``
106   Attempt to compile a test program with the system C compiler using
107   $CFLAGS and link it with the system linker using $LDFLAGS. The test
108   program must have been previously written to a file called $TMPC.
109
110``check_define $NAME``
111   Determine if the macro $NAME is defined by the system C compiler.
112
113``do_compiler $CC $ARGS...``
114   Attempt to run the C compiler $CC, passing it $ARGS...  This function
115   does not use flags passed via options such as ``--extra-cflags``, and
116   therefore can be used to check for cross compilers.  However, most
117   such checks are done at ``make`` time instead (see for example the
118   ``cc-option`` macro in ``pc-bios/option-rom/Makefile``).
119
120``write_c_skeleton``
121   Write a minimal C program main() function to the temporary file
122   indicated by $TMPC.
123
124
125Python virtual environments and the build process
126-------------------------------------------------
127
128An important step in ``configure`` is to create a Python virtual
129environment (venv) during the configuration phase.  The Python interpreter
130comes from the ``--python`` command line option, the ``$PYTHON`` variable
131from the environment, or the system PATH, in this order.  The venv resides
132in the ``pyvenv`` directory in the build tree, and provides consistency
133in how the build process runs Python code.
134
135At this stage, ``configure`` also queries the chosen Python interpreter
136about QEMU's build dependencies.  Note that the build process does  *not*
137look for ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
138likewise, there are no options such as ``--meson`` or ``--sphinx-build``.
139This avoids a potential mismatch, where Meson and Sphinx binaries on the
140PATH might operate in a different Python environment than the one chosen
141by the user during the build process.  On the other hand, it introduces
142a potential source of confusion where the user installs a dependency but
143``configure`` is not able to find it.  When this happens, the dependency
144was installed in the ``site-packages`` directory of another interpreter,
145or with the wrong ``pip`` program.
146
147If a package is available for the chosen interpreter, ``configure``
148prepares a small script that invokes it from the venv itself[#distlib]_.
149If not, ``configure`` can also optionally install dependencies in the
150virtual environment with ``pip``, either from wheels in ``python/wheels``
151or by downloading the package with PyPI.  Downloading can be disabled with
152``--disable-download``; and anyway, it only happens when a ``configure``
153option (currently, only ``--enable-docs``) is explicitly enabled but
154the dependencies are not present[#pip]_.
155
156.. [#distlib] The scripts are created based on the package's metadata,
157              specifically the ``console_script`` entry points.  This is the
158              same mechanism that ``pip`` uses when installing a package.
159              Currently, in all cases it would be possible to use ``python -m``
160              instead of an entry point script, which makes this approach a
161              bit overkill.  On the other hand, creating the scripts is
162              future proof and it makes the contents of the ``pyvenv/bin``
163              directory more informative.  Portability is also not an issue,
164              because the Python Packaging Authority provides a package
165              ``distlib.scripts`` to perform this task.
166
167.. [#pip] ``pip`` might also be used when running ``make check-avocado``
168           if downloading is enabled, to ensure that Avocado is
169           available.
170
171The required versions of the packages are stored in a configuration file
172``pythondeps.toml``.  The format is custom to QEMU, but it is documented
173at the top of the file itself and it should be easy to understand.  The
174requirements should make it possible to use the version that is packaged
175that is provided by supported distros.
176
177When dependencies are downloaded, instead, ``configure`` uses a "known
178good" version that is also listed in ``pythondeps.toml``.  In this
179scenario, ``pythondeps.toml`` behaves like the "lock file" used by
180``cargo``, ``poetry`` or other dependency management systems.
181
182
183Bundled Python packages
184-----------------------
185
186Python packages that are **mandatory** dependencies to build QEMU,
187but are not available in all supported distros, are bundled with the
188QEMU sources.  Currently this includes Meson (outdated in CentOS 8
189and derivatives, Ubuntu 20.04 and 22.04, and openSUSE Leap) and tomli
190(absent in Ubuntu 20.04).
191
192If you need to update these, please do so by modifying and rerunning
193``python/scripts/vendor.py``.  This script embeds the sha256 hash of
194package sources and checks it.  The pypi.org web site provides an easy
195way to retrieve the sha256 hash of the sources.
196
197
198Stage 2: Meson
199==============
200
201The Meson build system describes the build and install process for:
202
2031) executables, which include:
204
205   - Tools - ``qemu-img``, ``qemu-nbd``, ``qemu-ga`` (guest agent), etc
206
207   - System emulators - ``qemu-system-$ARCH``
208
209   - Userspace emulators - ``qemu-$ARCH``
210
211   - Unit tests
212
2132) documentation
214
2153) ROMs, whether provided as binary blobs in the QEMU distributions
216   or cross compiled under the direction of the configure script
217
2184) other data files, such as icons or desktop files
219
220All executables are built by default, except for some ``contrib/``
221binaries that are known to fail to build on some platforms (for example
22232-bit or big-endian platforms).  Tests are also built by default,
223though that might change in the future.
224
225The source code is highly modularized, split across many files to
226facilitate building of all of these components with as little duplicated
227compilation as possible. Using the Meson "sourceset" functionality,
228``meson.build`` files group the source files in rules that are
229enabled according to the available system libraries and to various
230configuration symbols.  Sourcesets belong to one of four groups:
231
232Subsystem sourcesets:
233  Various subsystems that are common to both tools and emulators have
234  their own sourceset, for example ``block_ss`` for the block device subsystem,
235  ``chardev_ss`` for the character device subsystem, etc.  These sourcesets
236  are then turned into static libraries as follows::
237
238    libchardev = static_library('chardev', chardev_ss.sources(),
239                                name_suffix: 'fa',
240                                build_by_default: false)
241
242    chardev = declare_dependency(link_whole: libchardev)
243
244  As of Meson 0.55.1, the special ``.fa`` suffix should be used for everything
245  that is used with ``link_whole``, to ensure that the link flags are placed
246  correctly in the command line.
247
248Target-independent emulator sourcesets:
249  Various general purpose helper code is compiled only once and
250  the .o files are linked into all output binaries that need it.
251  This includes error handling infrastructure, standard data structures,
252  platform portability wrapper functions, etc.
253
254  Target-independent code lives in the ``common_ss``, ``system_ss`` and
255  ``user_ss`` sourcesets.  ``common_ss`` is linked into all emulators,
256  ``system_ss`` only in system emulators, ``user_ss`` only in user-mode
257  emulators.
258
259Target-dependent emulator sourcesets:
260  In the target-dependent set lives CPU emulation, some device emulation and
261  much glue code. This sometimes also has to be compiled multiple times,
262  once for each target being built.  Target-dependent files are included
263  in the ``specific_ss`` sourceset.
264
265  Each emulator also includes sources for files in the ``hw/`` and ``target/``
266  subdirectories.  The subdirectory used for each emulator comes
267  from the target's definition of ``TARGET_BASE_ARCH`` or (if missing)
268  ``TARGET_ARCH``, as found in ``default-configs/targets/*.mak``.
269
270  Each subdirectory in ``hw/`` adds one sourceset to the ``hw_arch`` dictionary,
271  for example::
272
273    arm_ss = ss.source_set()
274    arm_ss.add(files('boot.c'), fdt)
275    ...
276    hw_arch += {'arm': arm_ss}
277
278  The sourceset is only used for system emulators.
279
280  Each subdirectory in ``target/`` instead should add one sourceset to each
281  of the ``target_arch`` and ``target_system_arch``, which are used respectively
282  for all emulators and for system emulators only.  For example::
283
284    arm_ss = ss.source_set()
285    arm_system_ss = ss.source_set()
286    ...
287    target_arch += {'arm': arm_ss}
288    target_system_arch += {'arm': arm_system_ss}
289
290Module sourcesets:
291  There are two dictionaries for modules: ``modules`` is used for
292  target-independent modules and ``target_modules`` is used for
293  target-dependent modules.  When modules are disabled the ``module``
294  source sets are added to ``system_ss`` and the ``target_modules``
295  source sets are added to ``specific_ss``.
296
297  Both dictionaries are nested.  One dictionary is created per
298  subdirectory, and these per-subdirectory dictionaries are added to
299  the toplevel dictionaries.  For example::
300
301    hw_display_modules = {}
302    qxl_ss = ss.source_set()
303    ...
304    hw_display_modules += { 'qxl': qxl_ss }
305    modules += { 'hw-display': hw_display_modules }
306
307Utility sourcesets:
308  All binaries link with a static library ``libqemuutil.a``.  This library
309  is built from several sourcesets; most of them however host generated
310  code, and the only two of general interest are ``util_ss`` and ``stub_ss``.
311
312  The separation between these two is purely for documentation purposes.
313  ``util_ss`` contains generic utility files.  Even though this code is only
314  linked in some binaries, sometimes it requires hooks only in some of
315  these and depend on other functions that are not fully implemented by
316  all QEMU binaries.  ``stub_ss`` links dummy stubs that will only be linked
317  into the binary if the real implementation is not present.  In a way,
318  the stubs can be thought of as a portable implementation of the weak
319  symbols concept.
320
321
322The following files concur in the definition of which files are linked
323into each emulator:
324
325``default-configs/devices/*.mak``
326  The files under ``default-configs/devices/`` control the boards and devices
327  that are built into each QEMU system emulation targets. They merely contain
328  a list of config variable definitions such as::
329
330    include arm-softmmu.mak
331    CONFIG_XLNX_ZYNQMP_ARM=y
332    CONFIG_XLNX_VERSAL=y
333
334``*/Kconfig``
335  These files are processed together with ``default-configs/devices/*.mak`` and
336  describe the dependencies between various features, subsystems and
337  device models.  They are described in :ref:`kconfig`
338
339``default-configs/targets/*.mak``
340  These files mostly define symbols that appear in the ``*-config-target.h``
341  file for each emulator [#cfgtarget]_.  However, the ``TARGET_ARCH``
342  and ``TARGET_BASE_ARCH`` will also be used to select the ``hw/`` and
343  ``target/`` subdirectories that are compiled into each target.
344
345.. [#cfgtarget] This header is included by ``qemu/osdep.h`` when
346                compiling files from the target-specific sourcesets.
347
348These files rarely need changing unless you are adding a completely
349new target, or enabling new devices or hardware for a particular
350system/userspace emulation target
351
352
353Adding checks
354-------------
355
356Compiler checks can be as simple as the following::
357
358  config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h'))
359
360A more complex task such as adding a new dependency usually
361comprises the following tasks:
362
363 - Add a Meson build option to meson_options.txt.
364
365 - Add code to perform the actual feature check.
366
367 - Add code to include the feature status in ``config-host.h``
368
369 - Add code to print out the feature status in the configure summary
370   upon completion.
371
372Taking the probe for SDL2_Image as an example, we have the following
373in ``meson_options.txt``::
374
375  option('sdl_image', type : 'feature', value : 'auto',
376         description: 'SDL Image support for icons')
377
378Unless the option was given a non-``auto`` value (on the configure
379command line), the detection code must be performed only if the
380dependency will be used::
381
382  sdl_image = not_found
383  if not get_option('sdl_image').auto() or have_system
384    sdl_image = dependency('SDL2_image', required: get_option('sdl_image'),
385                           method: 'pkg-config')
386  endif
387
388This avoids warnings on static builds of user-mode emulators, for example.
389Most of the libraries used by system-mode emulators are not available for
390static linking.
391
392The other supporting code is generally simple::
393
394  # Create config-host.h (if applicable)
395  config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found())
396
397  # Summary
398  summary_info += {'SDL image support': sdl_image.found()}
399
400For the configure script to parse the new option, the
401``scripts/meson-buildoptions.sh`` file must be up-to-date; ``make
402update-buildoptions`` (or just ``make``) will take care of updating it.
403
404
405Support scripts
406---------------
407
408Meson has a special convention for invoking Python scripts: if their
409first line is ``#! /usr/bin/env python3`` and the file is *not* executable,
410find_program() arranges to invoke the script under the same Python
411interpreter that was used to invoke Meson.  This is the most common
412and preferred way to invoke support scripts from Meson build files,
413because it automatically uses the value of configure's --python= option.
414
415In case the script is not written in Python, use a ``#! /usr/bin/env ...``
416line and make the script executable.
417
418Scripts written in Python, where it is desirable to make the script
419executable (for example for test scripts that developers may want to
420invoke from the command line, such as tests/qapi-schema/test-qapi.py),
421should be invoked through the ``python`` variable in meson.build. For
422example::
423
424  test('QAPI schema regression tests', python,
425       args: files('test-qapi.py'),
426       env: test_env, suite: ['qapi-schema', 'qapi-frontend'])
427
428This is needed to obey the --python= option passed to the configure
429script, which may point to something other than the first python3
430binary on the path.
431
432By the time Meson runs, Python dependencies are available in the virtual
433environment and should be invoked through the scripts that ``configure``
434places under ``pyvenv``.  One way to do so is as follows, using Meson's
435``find_program`` function::
436
437  sphinx_build = find_program(
438       fs.parent(python.full_path()) / 'sphinx-build',
439       required: get_option('docs'))
440
441
442Stage 3: Make
443=============
444
445The next step in building QEMU is to invoke make.  GNU Make is required
446to build QEMU, and may be installed as ``gmake`` on some hosts.
447
448The output of Meson is a ``build.ninja`` file, which is used with the
449Ninja build tool.  However, QEMU's build comprises other components than
450just the emulators (namely firmware and the tests in ``tests/tcg``) which
451need different cross compilers.  The QEMU Makefile wraps both Ninja and
452the smaller build systems for firmware and tests; it also takes care of
453running ``configure`` again when the script changes.  Apart from invoking
454these sub-Makefiles, the resulting build is largely non-recursive.
455
456Tests, whether defined in ``meson.build`` or not, are also ran by the
457Makefile with the traditional ``make check`` phony target, while benchmarks
458are run with ``make bench``.  Meson test suites such as ``unit`` can be ran
459with ``make check-unit``, and ``make check-tcg`` builds and runs "non-Meson"
460tests for all targets.
461
462If desired, it is also possible to use ``ninja`` and ``meson test``,
463respectively to build emulators and run tests defined in meson.build.
464The main difference is that ``make`` needs the ``-jN`` flag in order to
465enable parallel builds or tests.
466
467Useful make targets
468-------------------
469
470``help``
471  Print a help message for the most common build targets.
472
473``print-VAR``
474  Print the value of the variable VAR. Useful for debugging the build
475  system.
476
477
478Important files for the build system
479====================================
480
481Statically defined files
482------------------------
483
484The following key files are statically defined in the source tree, with
485the rules needed to build QEMU. Their behaviour is influenced by a
486number of dynamically created files listed later.
487
488``Makefile``
489  The main entry point used when invoking make to build all the components
490  of QEMU. The default 'all' target will naturally result in the build of
491  every component.
492
493``*/meson.build``
494  The meson.build file in the root directory is the main entry point for the
495  Meson build system, and it coordinates the configuration and build of all
496  executables.  Build rules for various subdirectories are included in
497  other meson.build files spread throughout the QEMU source tree.
498
499``python/scripts/mkvenv.py``
500  A wrapper for the Python ``venv`` and ``distlib.scripts`` packages.
501  It handles creating the virtual environment, creating scripts in
502  ``pyvenv/bin``, and calling ``pip`` to install dependencies.
503
504``tests/Makefile.include``
505  Rules for external test harnesses. These include the TCG tests
506  and the Avocado-based integration tests.
507
508``tests/docker/Makefile.include``
509  Rules for Docker tests. Like ``tests/Makefile.include``, this file is
510  included directly by the top level Makefile, anything defined in this
511  file will influence the entire build system.
512
513``tests/vm/Makefile.include``
514  Rules for VM-based tests. Like ``tests/Makefile.include``, this file is
515  included directly by the top level Makefile, anything defined in this
516  file will influence the entire build system.
517
518Dynamically created files
519-------------------------
520
521The following files are generated at run-time in order to control the
522behaviour of the Makefiles. This avoids the need for QEMU makefiles to
523go through any pre-processing as seen with autotools, where configure
524generates ``Makefile`` from ``Makefile.in``.
525
526Built by configure:
527
528``config-host.mak``
529  When configure has determined the characteristics of the build host it
530  will write the paths to various tools to this file, for use in ``Makefile``
531  and to a smaller extent ``meson.build``.
532
533  ``config-host.mak`` is also used as a dependency checking mechanism. If make
534  sees that the modification timestamp on configure is newer than that on
535  ``config-host.mak``, then configure will be re-run.
536
537``config-meson.cross``
538
539  A Meson "cross file" (or native file) used to communicate the paths to
540  the toolchain and other configuration options.
541
542``config.status``
543
544  A small shell script that will invoke configure again with the same
545  environment variables that were set during the first run.  It's used to
546  rerun configure after changes to the source code, but it can also be
547  inspected manually to check the contents of the environment.
548
549``Makefile.prereqs``
550
551  A set of Makefile dependencies that order the build and execution of
552  firmware and tests after the container images and emulators that they
553  need.
554
555``pc-bios/*/config.mak``, ``tests/tcg/config-host.mak``, ``tests/tcg/*/config-target.mak``
556
557  Configuration variables used to build the firmware and TCG tests,
558  including paths to cross compilation toolchains.
559
560``pyvenv``
561
562  A Python virtual environment that is used for all Python code running
563  during the build.  Using a virtual environment ensures that even code
564  that is run via ``sphinx-build``, ``meson`` etc. uses the same interpreter
565  and packages.
566
567Built by Meson:
568
569``config-host.h``
570  Used by C code to determine the properties of the build environment
571  and the set of enabled features for the entire build.
572
573``${TARGET-NAME}-config-devices.mak``
574  TARGET-NAME is the name of a system emulator. The file is
575  generated by Meson using files under ``configs/devices`` as input.
576
577``${TARGET-NAME}-config-target.mak``
578  TARGET-NAME is the name of a system or usermode emulator. The file is
579  generated by Meson using files under ``configs/targets`` as input.
580
581``$TARGET_NAME-config-target.h``, ``$TARGET_NAME-config-devices.h``
582  Used by C code to determine the properties and enabled
583  features for each target.  enabled.  They are generated from
584  the contents of the corresponding ``*.mak`` files using Meson's
585  ``configure_file()`` function; each target can include them using
586  the ``CONFIG_TARGET`` and ``CONFIG_DEVICES`` macro respectively.
587
588``build.ninja``
589  The build rules.
590
591
592Built by Makefile:
593
594``Makefile.ninja``
595  A Makefile include that bridges to ninja for the actual build.  The
596  Makefile is mostly a list of targets that Meson included in build.ninja.
597
598``Makefile.mtest``
599  The Makefile definitions that let "make check" run tests defined in
600  meson.build.  The rules are produced from Meson's JSON description of
601  tests (obtained with "meson introspect --tests") through the script
602  scripts/mtest2make.py.
603