Lines Matching +full:docker +full:- +full:test +full:- +full:clang
7 everything from unit testing and exercising specific sub-systems all
9 tests you can run ``make check-help`` from either the source or build
17 [./pyvenv/bin/]meson test --suite qemu:softfloat
21 The rest of this document will cover the details for specific test
25 -------------------------
36 Different sub-types of "make check" tests will be explained below.
45 Unit tests, which can be invoked with ``make check-unit``, are simple C tests
49 If you are writing new code in QEMU, consider adding a unit test, especially
51 add a new unit test:
53 1. Create a new source file. For example, ``tests/unit/foo-test.c``.
55 2. Write the test. Normally you would include the header file which exports
57 test. The test code should be organized with the glib testing framework.
58 Copying and modifying an existing test is usually a good idea.
60 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
62 dependencies to be linked with the test. For a simple test whose source
63 is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
67 'foo-test': [],
72 a unit test failure is often directly invoking it or even running it under
80 make check-unit V=1
82 and copy the actual command line which executes the unit test, then run
88 QTest is a device emulation testing framework. It can be very useful to test
97 make check-qtest
99 Writing portable test cases
102 Care must be taken when writing portable test cases that can be built and run
110 stdin/stdout/stderr and null devices. For example if your test case uses
114 * If your test cases uses the blkdebug feature, use relative path to pass
117 * Use double quotes in your extra QEMU command line in your test cases
122 test cases opens a file to write some data and later wants to compare the
125 * If a certain test case can only run on POSIX or Linux hosts, use a proper
126 #ifdef in the codes. If the whole test suite cannot run on Windows, disable
136 The input/output data is managed under the ``tests/qapi-schema`` directory.
137 Each test case includes four files that have a common base name:
139 * ``${casename}.json`` - the file contains the JSON input for feeding the
141 * ``${casename}.out`` - the file contains the expected stdout from the parser
142 * ``${casename}.err`` - the file contains the expected stderr from the parser
143 * ``${casename}.exit`` - the expected error code
145 Consider adding a new QAPI schema test when you are making a change on the QAPI
150 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
152 2. Add the new test in ``tests/Makefile.include``. For example:
154 ``qapi-schema += foo.json``
156 check-block
159 ``make check-block`` runs a subset of the block layer iotests (the tests that
164 ------------
166 QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
167 framework widely used to test block layer related features. It is higher level
170 test files are named with numbers.
173 ``tests/qemu-iotests`` directory under the build directory, and run ``./check``
182 # test with qcow2 format
183 ./check -qcow2
184 # or test a different protocol
185 ./check -nbd
187 It's also possible to list test numbers explicitly:
192 ./check -qcow2 001 030 153
194 Cache mode can be selected with the "-c" option, which may help reveal bugs
197 More options are supported by the ``./check`` script, run ``./check -h`` for
200 Writing a new test case
205 test cases, so it is possible that extending one of them may achieve the goal
212 output. They are given the same number in file names. E.g. Test script ``055``
220 There isn't a hard rule about how to write a test script, but a new test is
222 commonly used ways to create a test:
243 Both Python and Bash frameworks in iotests provide helpers to manage test
244 images. They can be used to create and clean up images under the test
246 more convenient to use the pseudo block driver, ``null-co://``, as the test
247 image, which doesn't require image creation or cleaning up. Avoid system-wide
251 test failure. If using such devices are explicitly desired, consider adding
254 Debugging a test case
258 a failing test:
260 * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
266 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
268 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
271 * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
274 The final command line will be ``valgrind --log-file=$TEST_DIR/
275 <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
277 * ``-d`` (debug) just increases the logging verbosity, showing
280 * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
282 ``$TEST_DIR/qemu-machine-<random_string>``.
284 Test case groups
287 "Tests may belong to one or more test groups, which are defined in the form
288 of a comment in the test source file. By convention, test groups are listed
289 in the second line of the test file, after the "#!/..." line, like this:
298 Another way of defining groups is creating the tests/qemu-iotests/group.local
300 in upstream). This file may be used for defining some downstream test groups
307 # ci - tests to run on build
308 # down - our downstream tests, not for upstream
316 our-ugly-workaround-test down ci
320 - quick: Tests in this group should finish within a few seconds.
322 - auto: Tests in this group are used during "make check" and should be
324 (also non-x86), with every QEMU configuration (i.e. must not fail if
325 an optional feature is not compiled in - but reporting a "skip" is ok),
330 - disabled: Tests in this group are disabled and ignored by check.
332 .. _container-ref:
335 ---------------------
341 build and test QEMU in predefined and widely accessible Linux
342 environments. This makes it possible to expand the test coverage
344 was originally written for Docker although we also support Podman as
346 names and scripts are prefixed with "docker" the system will
350 for testing TCG. See :ref:`checktcg-ref` for more details.
352 Docker Prerequisites
355 Install "docker" with the system package manager and start the Docker service
357 Docker commands. Typically it means setting up passwordless ``sudo docker``
362 $ sudo yum install docker
363 $ # or `apt-get install docker` for Ubuntu, etc.
364 $ sudo systemctl start docker
365 $ sudo docker ps
370 "docker" group and making the docker daemon socket file (by default
371 ``/var/run/docker.sock``) accessible to the group:
375 $ sudo groupadd docker
376 $ sudo usermod $USER -a -G docker
377 $ sudo chown :docker /var/run/docker.sock
380 exploit the whole host with Docker bind mounting or other privileged
398 From source tree, type ``make docker-help`` to see the help. Testing
405 make docker-test-build@debian
408 is downloaded and initialized automatically), in which the ``test-build`` job
415 ``registry.gitlab.com/qemu-project/qemu`` which will automatically be
416 used to pull in pre-built layers. This avoids unnecessary strain on
423 make docker-image-debian-arm64-cross NOCACHE=1
429 in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help``
432 A ``.pre`` script can be added beside the ``.docker`` file, which will be
435 for example, to make qemu-user powered cross build containers work.
438 a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
443 using the ``lcitool`` program provided by the ``libvirt-ci`` project:
445 https://gitlab.com/libvirt/libvirt-ci
447 ``libvirt-ci`` contains an ``lcitool`` program as well as a list of
450 pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
456 Adding new build pre-requisites
460 pre-requisite to QEMU, the prerequisites should to be added to
464 In the simple case where the pre-requisite is already known to ``libvirt-ci``
467 * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
469 * Run ``make lcitool-refresh`` to re-generate all relevant build environment
472 It may be that ``libvirt-ci`` does not know about the new pre-requisite.
474 first to contribute the mapping to the ``libvirt-ci`` project:
476 * Fork the ``libvirt-ci`` project on gitlab
480 many OS distros as practical. Run ``python -m pytest --regenerate-output``
483 * Commit the ``mappings.yml`` change together with the regenerated test
484 files, and submit a merge request to the ``libvirt-ci`` project.
485 Please note in the description that this is a new build pre-requisite
490 all OS distributions supported by ``libvirt-ci``.
493 the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
495 run ``make lcitool-refresh``.
499 obvious breakages when adding the new pre-requisite. Please see
503 For enterprise distros that default to old, end-of-life versions of the
506 Modifying this file should not be necessary unless the new pre-requisite
513 In some cases ``libvirt-ci`` will not know about the OS distro that is
517 * Send a mail to qemu-devel, copying people listed in the
518 MAINTAINERS file for ``Build and test automation``.
523 * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
524 pointing to the qemu-devel mail thread in the archives.
527 to avoid duplication, as well as to get feedback from libvirt-ci
532 * Fork the ``libvirt-ci`` project on gitlab
536 uses a package format not currently known. The ``libvirt-ci``
541 as practical. Run ``python -m pytest --regenerate-output`` and
544 * Commit the changes to ``lcitool/facts`` and the regenerated test
545 files, and submit a merge request to the ``libvirt-ci`` project.
546 Please note in the description that this is a new build pre-requisite
551 all OS distributions supported by ``libvirt-ci``.
554 the ``libvirt-ci`` submodule to point to a commit that contains
561 Different tests are added to cover various configurations to build and test
562 QEMU. Docker tests are the executables under ``tests/docker`` named
563 ``test-*``. They are typically shell scripts and are built on top of a shell
564 library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
567 The full list of tests is printed in the ``make docker-help`` help.
569 Debugging a Docker test failure
572 When CI tasks, maintainers or yourself report a Docker test failure, follow the
576 ``make docker-test-mingw@fedora-win64-cross J=8``.
580 build QEMU and run tests from there, or press Ctrl-D to let the Docker
582 4. If you press Ctrl-D, the same building and testing procedure will begin, and
589 Various options can be used to affect how Docker tests are done. The full
590 list is in the ``make docker`` help text. The frequently used ones are:
595 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
597 * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
601 ----------------
610 Thread Sanitizer in Docker
612 TSan is currently supported in the ubuntu2204 docker.
614 The test-tsan test will build using TSan and then run make check.
618 make docker-test-tsan@ubuntu2204
620 TSan warnings under docker are placed in files located at build/tsan/.
622 We recommend using DEBUG=1 to allow launching the test from inside the docker,
628 It is possible to build and test with TSan, with a few additional steps.
629 These steps are normally done automatically in the docker.
631 TSan is supported for clang and gcc.
641 $ git clone --depth=1 --branch=2.81.0 https://github.com/GNOME/glib.git
643 $ CFLAGS="-O2 -g -fsanitize=thread" meson build
644 $ ninja -C build
650 ../configure --enable-tsan \
651 --disable-werror --extra-cflags="-O0"
660 $ ldd build/qemu-x86_64 | grep glib
661 $ qemu-system-x86_64 ...
679 This allows for running the test and then checking the warnings afterwards.
692 tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
698 tests/tsan/ignore.tsan - Has TSan warnings we wish to disable
699 at compile time for test or debug.
702 "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan"
717 https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
723 https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
725 docker-binfmt-image-debian-% targets
726 ------------------------------------
731 simplify setting up a rootfs by using docker to contain the foreign
737 You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
741 potential complications when copying into the docker image. Modern
743 executable on setup and avoids the need to find and re-open in the
744 chroot environment. This is triggered with the ``--persistent`` flag.
751 make docker-binfmt-image-debian-sid-hppa \
753 DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
754 DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
755 EXECUTABLE=(pwd)/qemu-hppa V=1
758 ``debian-bootstrap.pre`` which is called to do the initial debootstrap
761 ``qemu/debian-sid-hppa``.
764 ----------
766 This test suite contains scripts that bootstrap various guest images that have
768 help which is displayed with ``make vm-help``.
773 Run ``make vm-help`` to list available make targets. Invoke a specific make
774 command to run build test in an image. For example, ``make vm-build-freebsd``
777 not needed. The command will then generate the test image in ``./tests/vm/``
780 Note: images created by the scripts accept a well-known RSA key pair for SSH
788 By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
789 there isn't one, or if it is older than 2.10, the test won't work. In this case,
790 provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
792 Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
797 The ``-j$X`` option in the make command line is not propagated into the VM,
818 $ ./netbsd --build-image --image /var/tmp/netbsd.img
822 # --debug is added)
823 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
826 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
829 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
847 - Root password set to ``BaseVM.ROOT_PASS``
848 - User ``BaseVM.GUEST_USER`` is created, and password set to
850 - SSH service is enabled and started on boot,
853 - DHCP client service is enabled and started on boot, so that it can
854 automatically configure the virtio-net-pci NIC and communicate with QEMU
856 - Necessary packages are installed to untar the source tarball and build
860 untars a raw virtio-blk block device, which is the tarball data blob of the
865 --------------------
872 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
874 Alternatively, some command different from ``qemu-img info`` can be tested, by
875 changing the ``-c`` option.
878 -----------------------------
885 make check-functional
887 See :ref:`checkfunctional-ref` for more details.
890 ---------------------------------------------
900 make check-avocado
902 See :ref:`checkavocado-ref` for more details.
905 .. _checktcg-ref:
907 Testing with "make check-tcg"
908 -----------------------------
910 The check-tcg tests are intended for simple smoke tests of both
911 linux-user and softmmu TCG functionality. However to build test
916 apt install gcc-aarch64-linux-gnu
923 $(configure) --cross-cc-aarch64=aarch64-cc
925 There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
932 additional containers defined that have a minimal cross-build
933 environment that is only suitable for building test cases. Sometimes
935 for test cases that aren't yet in the LTS distros we support for QEMU
938 See :ref:`container-ref` for more details.
945 make build-tcg-tests-$TARGET
949 make run-tcg-tests-$TARGET
952 invoke QEMU for the test which is useful for debugging tests.
957 Tests can also be run directly from the test build directory. If you
958 run ``make help`` from the test build directory you will get a list of
962 make run-plugin-test-mmap-with-libinline.so
964 will run the mmap test with the ``libinline.so`` TCG plugin. The
965 gdbstub tests also re-use the test binaries but while exercising gdb.
967 TCG test dependencies
971 either totally bare with minimal gcc lib support (for system-mode tests)
972 or just glibc (for linux-user tests). This is because getting a cross
976 ---------------
978 There are a number of out-of-tree test suites that are used for more
987 as reporting test results via a special device::
989 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
991 Linux Test Project
996 exercise as many corner cases as possible. It is a useful test suite
997 to run to exercise QEMU's linux-user code::
999 https://linux-test-project.github.io/
1002 ----------------
1006 ``--enable-gcov`` option and build. Then run the tests as usual.
1008 If you want to gather coverage information on a single test the ``make
1009 clean-gcda`` target can be used to delete any existing coverage
1010 information before running a single test.
1013 coverage-html`` which will create
1014 ``meson-logs/coveragereport/index.html``.