devel/testing/avocado.rst

1 .. _checkavocado-ref:
4 Integration testing with Avocado
7 The ``tests/avocado`` directory hosts integration tests. They're usually
11 These tests are written using the Avocado Testing Framework (which must be
13 class, implemented at ``tests/avocado/avocado_qemu``.
34    ``avocado.utils.vmimage`` library)
39    - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
40    - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
43 -------------
45 You can run the avocado tests simply by executing:
49   make check-avocado
52 necessary avocado-framework dependencies into the QEMU venv within the
58 ``configure`` is called with ``--python=`` and that those modules are
60 specific version, they may be on packages named ``python3-venv`` and
61 ``python3-pip``.
64 ``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
69    make check-avocado AVOCADO_TAGS=quick
72 separated by spaces have an OR behavior. For more information on Avocado
75  https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
78 using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
84   make check-avocado AVOCADO_TESTS=$FILEPATH
90   make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
96   make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
102 …make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNA…
105 without an "activation".  For instance, the Avocado test runner
110   pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
112 Note that if ``make check-avocado`` was not executed before, it is
118   make check-venv
125   pyvenv/bin/avocado run tests/avocado/$TESTFILE
131   pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
134 of Avocado or ``make check-avocado``, and can also be queried using:
138   pyvenv/bin/avocado list tests/avocado
141 -------------------
143 To manually install Avocado and its dependencies, run:
147   pip install --user avocado-framework
151   https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
154 --------
156 The ``tests/avocado/avocado_qemu`` directory provides the
167       :avocado: tags=quick
171           res = self.vm.cmd('human-monitor-command',
179   avocado run version.py
182 directives such as ``:avocado: tags=TAG1,TAG2``.  To run all tests
187   avocado run -t quick .
223               'human-monitor-command',
227               'human-monitor-command',
231               'human-monitor-command',
248 time, it uses a Fedora 31 guest image.  The most basic example looks
263 ``tests/avocado`` for more examples.
266 -----------
269 device-crash-test and other Python scripts.  It's a wrapper around the
297    "qemu-system-${arch}", either in the current
304 -------------------
310 ``avocado.Test`` class, the following attributes are available on any
334 ``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
345 ``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
356 ``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
376 of images supported by the avocado.utils.vmimage library:
378 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
385 of images supported by the avocado.utils.vmimage library:
387 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
399 -------------------
401 To understand how Avocado parameters are accessed by tests, and how
404 …https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-t…
411   PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
457 of images supported by the avocado.utils.vmimage library:
459 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
466 of images supported by the avocado.utils.vmimage library:
468 https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
479 --------------
481 The Avocado framework provides Python decorators which allow for easily skip
483 on the test system or when the running environment is a CI system. For further
486 …https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-te…
527 The Avocado framework has a timeout mechanism which interrupts tests to avoid the
531 …https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-t…
534 that may not have a well-defined limit of time to finish under certain
550   # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
555 You can also add ``:avocado: tags=flaky`` to the test meta-data so
560    env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
561       run tests/avocado -filter-by-tags=flaky
567 Uninstalling Avocado
568 --------------------
571 easily uninstall Avocado.  Start by listing the packages you have
574   pip list --user
580 If you've used ``make check-avocado``, the Python virtual environment where
581 Avocado is installed will be cleaned up as part of ``make check-clean``.