xref: /openbmc/qemu/docs/devel/build-system.rst (revision 646b5378)
1==================================
2The QEMU build system architecture
3==================================
4
5This document aims to help developers understand the architecture of the
6QEMU build system. As with projects using GNU autotools, the QEMU build
7system has two stages; first the developer runs the "configure" script
8to determine the local build environment characteristics, then they run
9"make" to build the project.  This is about where the similarities with
10GNU autotools end, so try to forget what you know about them.
11
12The two general ways to perform a build are as follows:
13
14 - build artifacts outside of QEMU source tree entirely::
15
16     cd ../
17     mkdir build
18     cd build
19     ../qemu/configure
20     make
21
22 - build artifacts in a subdir of QEMU source tree::
23
24     mkdir build
25     cd build
26     ../configure
27     make
28
29Most of the actual build process uses Meson under the hood, therefore
30build artifacts cannot be placed in the source tree itself.
31
32
33Stage 1: configure
34==================
35
36The configure script has five tasks:
37
38 - detect the host architecture
39
40 - list the targets for which to build emulators; the list of
41   targets also affects which firmware binaries and tests to build
42
43 - find the compilers (native and cross) used to build executables,
44   firmware and tests.  The results are written as either Makefile
45   fragments (``config-host.mak``) or a Meson machine file
46   (``config-meson.cross``)
47
48 - create a virtual environment in which all Python code runs during
49   the build, and possibly install packages into it from PyPI
50
51 - invoke Meson in the virtual environment, to perform the actual
52   configuration step for the emulator build
53
54The configure script automatically recognizes command line options for
55which a same-named Meson option exists; dashes in the command line are
56replaced with underscores.
57
58Almost all QEMU developers that need to modify the build system will
59only be concerned with Meson, and therefore can skip the rest of this
60section.
61
62
63Modifying ``configure``
64-----------------------
65
66``configure`` is a shell script; it uses ``#!/bin/sh`` and therefore
67should be compatible with any POSIX shell. It is important to avoid
68using bash-isms to avoid breaking development platforms where bash is
69the primary host.
70
71The configure script provides a variety of functions to help writing
72portable shell code and providing consistent behavior across architectures
73and operating systems:
74
75``error_exit $MESSAGE $MORE...``
76   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
77   configure script with non-zero status.
78
79``has $COMMAND``
80   Determine if $COMMAND exists in the current environment, either as a
81   shell builtin, or executable binary, returning 0 on success.  The
82   replacement in Meson is ``find_program()``.
83
84``probe_target_compiler $TARGET``
85  Detect a cross compiler and cross tools for the QEMU target $TARGET (e.g.,
86  ``$CPU-softmmu``, ``$CPU-linux-user``, ``$CPU-bsd-user``).  If a working
87  compiler is present, return success and set variables ``$target_cc``,
88  ``$target_ar``, etc. to non-empty values.
89
90``write_target_makefile``
91  Write a Makefile fragment to stdout, exposing the result of the most
92  ``probe_target_compiler`` call as the usual Make variables (``CC``,
93  ``AR``, ``LD``, etc.).
94
95
96Configure does not generally perform tests for compiler options beyond
97basic checks to detect the host platform and ensure the compiler is
98functioning.  These are performed using a few more helper functions:
99
100``compile_object $CFLAGS``
101   Attempt to compile a test program with the system C compiler using
102   $CFLAGS. The test program must have been previously written to a file
103   called $TMPC.
104
105``compile_prog $CFLAGS $LDFLAGS``
106   Attempt to compile a test program with the system C compiler using
107   $CFLAGS and link it with the system linker using $LDFLAGS. The test
108   program must have been previously written to a file called $TMPC.
109
110``check_define $NAME``
111   Determine if the macro $NAME is defined by the system C compiler.
112
113``do_compiler $CC $ARGS...``
114   Attempt to run the C compiler $CC, passing it $ARGS...  This function
115   does not use flags passed via options such as ``--extra-cflags``, and
116   therefore can be used to check for cross compilers.  However, most
117   such checks are done at ``make`` time instead (see for example the
118   ``cc-option`` macro in ``pc-bios/option-rom/Makefile``).
119
120``write_c_skeleton``
121   Write a minimal C program main() function to the temporary file
122   indicated by $TMPC.
123
124
125Python virtual environments and the build process
126-------------------------------------------------
127
128An important step in ``configure`` is to create a Python virtual
129environment (venv) during the configuration phase.  The Python interpreter
130comes from the ``--python`` command line option, the ``$PYTHON`` variable
131from the environment, or the system PATH, in this order.  The venv resides
132in the ``pyvenv`` directory in the build tree, and provides consistency
133in how the build process runs Python code.
134
135At this stage, ``configure`` also queries the chosen Python interpreter
136about QEMU's build dependencies.  Note that the build process does  *not*
137look for ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
138likewise, there are no options such as ``--meson`` or ``--sphinx-build``.
139This avoids a potential mismatch, where Meson and Sphinx binaries on the
140PATH might operate in a different Python environment than the one chosen
141by the user during the build process.  On the other hand, it introduces
142a potential source of confusion where the user installs a dependency but
143``configure`` is not able to find it.  When this happens, the dependency
144was installed in the ``site-packages`` directory of another interpreter,
145or with the wrong ``pip`` program.
146
147If a package is available for the chosen interpreter, ``configure``
148prepares a small script that invokes it from the venv itself\ [#distlib]_.
149If not, ``configure`` can also optionally install dependencies in the
150virtual environment with ``pip``, either from wheels in ``python/wheels``
151or by downloading the package with PyPI.  Downloading can be disabled with
152``--disable-download``; and anyway, it only happens when a ``configure``
153option (currently, only ``--enable-docs``) is explicitly enabled but
154the dependencies are not present\ [#pip]_.
155
156.. [#distlib] The scripts are created based on the package's metadata,
157              specifically the ``console_script`` entry points.  This is the
158              same mechanism that ``pip`` uses when installing a package.
159              Currently, in all cases it would be possible to use ``python -m``
160              instead of an entry point script, which makes this approach a
161              bit overkill.  On the other hand, creating the scripts is
162              future proof and it makes the contents of the ``pyvenv/bin``
163              directory more informative.  Portability is also not an issue,
164              because the Python Packaging Authority provides a package
165              ``distlib.scripts`` to perform this task.
166
167.. [#pip] ``pip`` might also be used when running ``make check-avocado``
168           if downloading is enabled, to ensure that Avocado is
169           available.
170
171The required versions of the packages are stored in a configuration file
172``pythondeps.toml``.  The format is custom to QEMU, but it is documented
173at the top of the file itself and it should be easy to understand.  The
174requirements should make it possible to use the version that is packaged
175that is provided by supported distros.
176
177When dependencies are downloaded, instead, ``configure`` uses a "known
178good" version that is also listed in ``pythondeps.toml``.  In this
179scenario, ``pythondeps.toml`` behaves like the "lock file" used by
180``cargo``, ``poetry`` or other dependency management systems.
181
182
183Bundled Python packages
184-----------------------
185
186Python packages that are **mandatory** dependencies to build QEMU,
187but are not available in all supported distros, are bundled with the
188QEMU sources.  The only one is currently Meson (outdated in Ubuntu
18922.04 and openSUSE Leap).
190
191In order to include a new or updated wheel, modify and rerun the
192``python/scripts/vendor.py`` script.  The script embeds the
193sha256 hash of package sources and checks it.  The pypi.org web site
194provides an easy way to retrieve the sha256 hash of the sources.
195
196
197Stage 2: Meson
198==============
199
200The Meson build system describes the build and install process for:
201
2021) executables, which include:
203
204   - Tools - ``qemu-img``, ``qemu-nbd``, ``qemu-ga`` (guest agent), etc
205
206   - System emulators - ``qemu-system-$ARCH``
207
208   - Userspace emulators - ``qemu-$ARCH``
209
210   - Unit tests
211
2122) documentation
213
2143) ROMs, whether provided as binary blobs in the QEMU distributions
215   or cross compiled under the direction of the configure script
216
2174) other data files, such as icons or desktop files
218
219All executables are built by default, except for some ``contrib/``
220binaries that are known to fail to build on some platforms (for example
22132-bit or big-endian platforms).  Tests are also built by default,
222though that might change in the future.
223
224The source code is highly modularized, split across many files to
225facilitate building of all of these components with as little duplicated
226compilation as possible. Using the Meson "sourceset" functionality,
227``meson.build`` files group the source files in rules that are
228enabled according to the available system libraries and to various
229configuration symbols.  Sourcesets belong to one of four groups:
230
231Subsystem sourcesets:
232  Various subsystems that are common to both tools and emulators have
233  their own sourceset, for example ``block_ss`` for the block device subsystem,
234  ``chardev_ss`` for the character device subsystem, etc.  These sourcesets
235  are then turned into static libraries as follows::
236
237    libchardev = static_library('chardev', chardev_ss.sources(),
238                                build_by_default: false)
239
240    chardev = declare_dependency(objects: libchardev.extract_all_objects(recursive: false),
241                                 dependencies: chardev_ss.dependencies())
242
243Target-independent emulator sourcesets:
244  Various general purpose helper code is compiled only once and
245  the .o files are linked into all output binaries that need it.
246  This includes error handling infrastructure, standard data structures,
247  platform portability wrapper functions, etc.
248
249  Target-independent code lives in the ``common_ss``, ``system_ss`` and
250  ``user_ss`` sourcesets.  ``common_ss`` is linked into all emulators,
251  ``system_ss`` only in system emulators, ``user_ss`` only in user-mode
252  emulators.
253
254Target-dependent emulator sourcesets:
255  In the target-dependent set lives CPU emulation, some device emulation and
256  much glue code. This sometimes also has to be compiled multiple times,
257  once for each target being built.  Target-dependent files are included
258  in the ``specific_ss`` sourceset.
259
260  Each emulator also includes sources for files in the ``hw/`` and ``target/``
261  subdirectories.  The subdirectory used for each emulator comes
262  from the target's definition of ``TARGET_BASE_ARCH`` or (if missing)
263  ``TARGET_ARCH``, as found in ``default-configs/targets/*.mak``.
264
265  Each subdirectory in ``hw/`` adds one sourceset to the ``hw_arch`` dictionary,
266  for example::
267
268    arm_ss = ss.source_set()
269    arm_ss.add(files('boot.c'), fdt)
270    ...
271    hw_arch += {'arm': arm_ss}
272
273  The sourceset is only used for system emulators.
274
275  Each subdirectory in ``target/`` instead should add one sourceset to each
276  of the ``target_arch`` and ``target_system_arch``, which are used respectively
277  for all emulators and for system emulators only.  For example::
278
279    arm_ss = ss.source_set()
280    arm_system_ss = ss.source_set()
281    ...
282    target_arch += {'arm': arm_ss}
283    target_system_arch += {'arm': arm_system_ss}
284
285Module sourcesets:
286  There are two dictionaries for modules: ``modules`` is used for
287  target-independent modules and ``target_modules`` is used for
288  target-dependent modules.  When modules are disabled the ``module``
289  source sets are added to ``system_ss`` and the ``target_modules``
290  source sets are added to ``specific_ss``.
291
292  Both dictionaries are nested.  One dictionary is created per
293  subdirectory, and these per-subdirectory dictionaries are added to
294  the toplevel dictionaries.  For example::
295
296    hw_display_modules = {}
297    qxl_ss = ss.source_set()
298    ...
299    hw_display_modules += { 'qxl': qxl_ss }
300    modules += { 'hw-display': hw_display_modules }
301
302Utility sourcesets:
303  All binaries link with a static library ``libqemuutil.a``.  This library
304  is built from several sourcesets; most of them however host generated
305  code, and the only two of general interest are ``util_ss`` and ``stub_ss``.
306
307  The separation between these two is purely for documentation purposes.
308  ``util_ss`` contains generic utility files.  Even though this code is only
309  linked in some binaries, sometimes it requires hooks only in some of
310  these and depend on other functions that are not fully implemented by
311  all QEMU binaries.  ``stub_ss`` links dummy stubs that will only be linked
312  into the binary if the real implementation is not present.  In a way,
313  the stubs can be thought of as a portable implementation of the weak
314  symbols concept.
315
316
317The following files concur in the definition of which files are linked
318into each emulator:
319
320``default-configs/devices/*.mak``
321  The files under ``default-configs/devices/`` control the boards and devices
322  that are built into each QEMU system emulation targets. They merely contain
323  a list of config variable definitions such as::
324
325    include arm-softmmu.mak
326    CONFIG_XLNX_ZYNQMP_ARM=y
327    CONFIG_XLNX_VERSAL=y
328
329``*/Kconfig``
330  These files are processed together with ``default-configs/devices/*.mak`` and
331  describe the dependencies between various features, subsystems and
332  device models.  They are described in :ref:`kconfig`
333
334``default-configs/targets/*.mak``
335  These files mostly define symbols that appear in the ``*-config-target.h``
336  file for each emulator\ [#cfgtarget]_.  However, the ``TARGET_ARCH``
337  and ``TARGET_BASE_ARCH`` will also be used to select the ``hw/`` and
338  ``target/`` subdirectories that are compiled into each target.
339
340.. [#cfgtarget] This header is included by ``qemu/osdep.h`` when
341                compiling files from the target-specific sourcesets.
342
343These files rarely need changing unless you are adding a completely
344new target, or enabling new devices or hardware for a particular
345system/userspace emulation target
346
347
348Adding checks
349-------------
350
351Compiler checks can be as simple as the following::
352
353  config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h'))
354
355A more complex task such as adding a new dependency usually
356comprises the following tasks:
357
358 - Add a Meson build option to meson_options.txt.
359
360 - Add code to perform the actual feature check.
361
362 - Add code to include the feature status in ``config-host.h``
363
364 - Add code to print out the feature status in the configure summary
365   upon completion.
366
367Taking the probe for SDL2_Image as an example, we have the following
368in ``meson_options.txt``::
369
370  option('sdl_image', type : 'feature', value : 'auto',
371         description: 'SDL Image support for icons')
372
373Unless the option was given a non-``auto`` value (on the configure
374command line), the detection code must be performed only if the
375dependency will be used::
376
377  sdl_image = not_found
378  if not get_option('sdl_image').auto() or have_system
379    sdl_image = dependency('SDL2_image', required: get_option('sdl_image'),
380                           method: 'pkg-config')
381  endif
382
383This avoids warnings on static builds of user-mode emulators, for example.
384Most of the libraries used by system-mode emulators are not available for
385static linking.
386
387The other supporting code is generally simple::
388
389  # Create config-host.h (if applicable)
390  config_host_data.set('CONFIG_SDL_IMAGE', sdl_image.found())
391
392  # Summary
393  summary_info += {'SDL image support': sdl_image.found()}
394
395For the configure script to parse the new option, the
396``scripts/meson-buildoptions.sh`` file must be up-to-date; ``make
397update-buildoptions`` (or just ``make``) will take care of updating it.
398
399
400Support scripts
401---------------
402
403Meson has a special convention for invoking Python scripts: if their
404first line is ``#! /usr/bin/env python3`` and the file is *not* executable,
405find_program() arranges to invoke the script under the same Python
406interpreter that was used to invoke Meson.  This is the most common
407and preferred way to invoke support scripts from Meson build files,
408because it automatically uses the value of configure's --python= option.
409
410In case the script is not written in Python, use a ``#! /usr/bin/env ...``
411line and make the script executable.
412
413Scripts written in Python, where it is desirable to make the script
414executable (for example for test scripts that developers may want to
415invoke from the command line, such as tests/qapi-schema/test-qapi.py),
416should be invoked through the ``python`` variable in meson.build. For
417example::
418
419  test('QAPI schema regression tests', python,
420       args: files('test-qapi.py'),
421       env: test_env, suite: ['qapi-schema', 'qapi-frontend'])
422
423This is needed to obey the --python= option passed to the configure
424script, which may point to something other than the first python3
425binary on the path.
426
427By the time Meson runs, Python dependencies are available in the virtual
428environment and should be invoked through the scripts that ``configure``
429places under ``pyvenv``.  One way to do so is as follows, using Meson's
430``find_program`` function::
431
432  sphinx_build = find_program(
433       fs.parent(python.full_path()) / 'sphinx-build',
434       required: get_option('docs'))
435
436
437Stage 3: Make
438=============
439
440The next step in building QEMU is to invoke make.  GNU Make is required
441to build QEMU, and may be installed as ``gmake`` on some hosts.
442
443The output of Meson is a ``build.ninja`` file, which is used with the
444Ninja build tool.  However, QEMU's build comprises other components than
445just the emulators (namely firmware and the tests in ``tests/tcg``) which
446need different cross compilers.  The QEMU Makefile wraps both Ninja and
447the smaller build systems for firmware and tests; it also takes care of
448running ``configure`` again when the script changes.  Apart from invoking
449these sub-Makefiles, the resulting build is largely non-recursive.
450
451Tests, whether defined in ``meson.build`` or not, are also ran by the
452Makefile with the traditional ``make check`` phony target, while benchmarks
453are run with ``make bench``.  Meson test suites such as ``unit`` can be ran
454with ``make check-unit``, and ``make check-tcg`` builds and runs "non-Meson"
455tests for all targets.
456
457If desired, it is also possible to use ``ninja`` and ``meson test``,
458respectively to build emulators and run tests defined in meson.build.
459The main difference is that ``make`` needs the ``-jN`` flag in order to
460enable parallel builds or tests.
461
462Useful make targets
463-------------------
464
465``help``
466  Print a help message for the most common build targets.
467
468``print-VAR``
469  Print the value of the variable VAR. Useful for debugging the build
470  system.
471
472
473Important files for the build system
474====================================
475
476Statically defined files
477------------------------
478
479The following key files are statically defined in the source tree, with
480the rules needed to build QEMU. Their behaviour is influenced by a
481number of dynamically created files listed later.
482
483``Makefile``
484  The main entry point used when invoking make to build all the components
485  of QEMU. The default 'all' target will naturally result in the build of
486  every component.
487
488``*/meson.build``
489  The meson.build file in the root directory is the main entry point for the
490  Meson build system, and it coordinates the configuration and build of all
491  executables.  Build rules for various subdirectories are included in
492  other meson.build files spread throughout the QEMU source tree.
493
494``python/scripts/mkvenv.py``
495  A wrapper for the Python ``venv`` and ``distlib.scripts`` packages.
496  It handles creating the virtual environment, creating scripts in
497  ``pyvenv/bin``, and calling ``pip`` to install dependencies.
498
499``tests/Makefile.include``
500  Rules for external test harnesses. These include the TCG tests
501  and the Avocado-based integration tests.
502
503``tests/docker/Makefile.include``
504  Rules for Docker tests. Like ``tests/Makefile.include``, this file is
505  included directly by the top level Makefile, anything defined in this
506  file will influence the entire build system.
507
508``tests/vm/Makefile.include``
509  Rules for VM-based tests. Like ``tests/Makefile.include``, this file is
510  included directly by the top level Makefile, anything defined in this
511  file will influence the entire build system.
512
513Dynamically created files
514-------------------------
515
516The following files are generated at run-time in order to control the
517behaviour of the Makefiles. This avoids the need for QEMU makefiles to
518go through any pre-processing as seen with autotools, where configure
519generates ``Makefile`` from ``Makefile.in``.
520
521Built by configure:
522
523``config-host.mak``
524  When configure has determined the characteristics of the build host it
525  will write the paths to various tools to this file, for use in ``Makefile``
526  and to a smaller extent ``meson.build``.
527
528  ``config-host.mak`` is also used as a dependency checking mechanism. If make
529  sees that the modification timestamp on configure is newer than that on
530  ``config-host.mak``, then configure will be re-run.
531
532``config-meson.cross``
533
534  A Meson "cross file" (or native file) used to communicate the paths to
535  the toolchain and other configuration options.
536
537``config.status``
538
539  A small shell script that will invoke configure again with the same
540  environment variables that were set during the first run.  It's used to
541  rerun configure after changes to the source code, but it can also be
542  inspected manually to check the contents of the environment.
543
544``Makefile.prereqs``
545
546  A set of Makefile dependencies that order the build and execution of
547  firmware and tests after the container images and emulators that they
548  need.
549
550``pc-bios/*/config.mak``, ``tests/tcg/config-host.mak``, ``tests/tcg/*/config-target.mak``
551
552  Configuration variables used to build the firmware and TCG tests,
553  including paths to cross compilation toolchains.
554
555``pyvenv``
556
557  A Python virtual environment that is used for all Python code running
558  during the build.  Using a virtual environment ensures that even code
559  that is run via ``sphinx-build``, ``meson`` etc. uses the same interpreter
560  and packages.
561
562Built by Meson:
563
564``config-host.h``
565  Used by C code to determine the properties of the build environment
566  and the set of enabled features for the entire build.
567
568``${TARGET-NAME}-config-devices.mak``
569  TARGET-NAME is the name of a system emulator. The file is
570  generated by Meson using files under ``configs/devices`` as input.
571
572``${TARGET-NAME}-config-target.mak``
573  TARGET-NAME is the name of a system or usermode emulator. The file is
574  generated by Meson using files under ``configs/targets`` as input.
575
576``$TARGET_NAME-config-target.h``, ``$TARGET_NAME-config-devices.h``
577  Used by C code to determine the properties and enabled
578  features for each target.  enabled.  They are generated from
579  the contents of the corresponding ``*.mak`` files using Meson's
580  ``configure_file()`` function; each target can include them using
581  the ``CONFIG_TARGET`` and ``CONFIG_DEVICES`` macro respectively.
582
583``build.ninja``
584  The build rules.
585
586
587Built by Makefile:
588
589``Makefile.ninja``
590  A Makefile include that bridges to ninja for the actual build.  The
591  Makefile is mostly a list of targets that Meson included in build.ninja.
592
593``Makefile.mtest``
594  The Makefile definitions that let "make check" run tests defined in
595  meson.build.  The rules are produced from Meson's JSON description of
596  tests (obtained with "meson introspect --tests") through the script
597  scripts/mtest2make.py.
598