devel/testing/ci-definitions.rst.inc

8 ---------------
25 ------------
34 On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
37 ------------------
46 On QEMU, functional testing is represented by the 'check-qtest' target from
49 System testing
50 --------------
52 System tests ensure all application elements mesh properly while the overall
53 functionality and performance are achieved [4]_. Some or all system components
54 are integrated to create a complete system to be tested as a whole. System
56 transfer the right data at the right time across their interfaces. As system
57 testing focuses on interactions, use case-based testing is a practical approach
58 to system testing [5]_. Note that, in some cases, system testing may require
59 interaction with third-party software, like operating system images, databases,
62 On QEMU, system testing is represented by the 'check-avocado' target from
65 Flaky tests
66 -----------
68 A flaky test is defined as a test that exhibits both a passing and a failing
70 intermittent/flaky test are async wait, concurrency, and test order dependency
74 ------
85 ---------------------------
90 the unit, functional, system, and other tests.
94 1. System tests may depend on external software (operating system images,
96 2. It may take a long time to build and test. It may be impractical to build
97    the system being developed several times per day.
99    not be possible to run system tests in the developer’s private workspace.
100    There may be differences in hardware, operating system, or installed
101    software. Therefore, more time is required for testing the system.
104 ----------
114 .. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests.
118        Reliable Software Releases Through Build, Test, and Deployment, p. 122.
120        Reliable Software Releases Through Build, Test, and Deployment, p. 55.