xref: /openbmc/qemu/docs/devel/testing/acpi-bits.rst (revision ca5aa28e)
1==================================
2ACPI/SMBIOS testing using biosbits
3==================================
4************
5Introduction
6************
7Biosbits is a software written by Josh Triplett that can be downloaded
8from https://biosbits.org/. The github codebase can be found
9`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that
10executes the bios components such as acpi and smbios tables directly through
11acpica bios interpreter (a freely available C based library written by Intel,
12downloadable from https://acpica.org/ and is included with biosbits) without an
13operating system getting involved in between. Bios-bits has python integration
14with grub so actual routines that executes bios components can be written in
15python instead of bash-ish (grub's native scripting language).
16There are several advantages to directly testing the bios in a real physical
17machine or in a VM as opposed to indirectly discovering bios issues through the
18operating system (the OS). Operating systems tend to bypass bios problems and
19hide them from the end user. We have more control of what we wanted to test and
20how by being as close to the bios on a running system as possible without a
21complicated software component such as an operating system coming in between.
22Another issue is that we cannot exercise bios components such as ACPI and
23SMBIOS without being in the highest hardware privilege level, ring 0 for
24example in case of x86. Since the OS executes from ring 0 whereas normal user
25land software resides in unprivileged ring 3, operating system must be modified
26in order to write our test routines that exercise and test the bios. This is
27not possible in all cases. Lastly, test frameworks and routines are preferably
28written using a high level scripting language such as python. OSes and
29OS modules are generally written using low level languages such as C and
30low level assembly machine language. Writing test routines in a low level
31language makes things more cumbersome. These and other reasons makes using
32bios-bits very attractive for testing bioses. More details on the inspiration
33for developing biosbits and its real life uses were presented `at Plumbers
34in 2011 <Plumbers_>`__ and `at Linux.conf.au in 2012 <Linux.conf.au_>`__.
35
36For QEMU, we maintain a fork of bios bits in `gitlab`_, along with all
37the dependent submodules.  This fork contains numerous fixes, a newer
38acpica and changes specific to running these functional QEMU tests using
39bits. The author of this document is the current maintainer of the QEMU
40fork of bios bits repository. For more information, please see `the
41author's FOSDEM presentation <FOSDEM_>`__ on this bios-bits based test framework.
42
43.. _Plumbers: https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
44.. _Linux.conf.au: https://www.youtube.com/watch?v=36QIepyUuhg
45.. _gitlab: https://gitlab.com/qemu-project/biosbits-bits
46.. _FOSDEM: https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/
47
48*********************************
49Description of the test framework
50*********************************
51
52Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU
53functional test that drives all this.
54
55A brief description of the various test files follows.
56
57Under ``tests/functional/`` as the root we have:
58
59::
60
61   ├── acpi-bits
62   │ ├── bits-config
63   │ │ └── bits-cfg.txt
64   │ ├── bits-tests
65   │   ├── smbios.py2
66   │   ├── testacpi.py2
67   │   └── testcpuid.py2
68   ├── test_acpi_bits.py
69
70* ``tests/functional``:
71
72   ``test_acpi_bits.py``:
73   This is the main python functional test script that generates a
74   biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
75   test failures. This is the script one would be interested in if they wanted
76   to add or change some component of the log parsing, add a new command line
77   to alter how QEMU is spawned etc. Test writers typically would not need to
78   modify this script unless they wanted to enhance or change the log parsing
79   for their tests. In order to enable debugging, you can set **V=1**
80   environment variable. This enables verbose mode for the test and also dumps
81   the entire log from bios bits and more information in case failure happens.
82   You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
83   verbose logs and also retain the temporary work directory the test used for
84   you to inspect and run the specific commands manually.
85
86   In order to run this test, please perform the following steps from the QEMU
87   build directory (assuming that the sources are in ".."):
88   ::
89
90     $ export PYTHONPATH=../python:../tests/functional
91     $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
92     $ python3 ../tests/functional/test_acpi_bits.py
93
94   The above will run all acpi-bits functional tests (producing output in
95   tap format).
96
97   You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/
98   for more information about the run or in order to diagnoze issues.
99   If you pass V=1 in the environment, more diagnostic logs will be put into
100   the test log.
101
102* ``tests/functional/acpi-bits/bits-config``:
103
104   This location contains biosbits configuration files that determine how the
105   software runs the tests.
106
107   ``bits-config.txt``:
108   This is the biosbits config file that determines what tests
109   or actions are performed by bits. The description of the config options are
110   provided in the file itself.
111
112* ``tests/functional/acpi-bits/bits-tests``:
113
114   This directory contains biosbits python based tests that are run from within
115   the biosbits environment in the spawned VM. New additions of test cases can
116   be made in the appropriate test file. For example, new acpi tests can go
117   into testacpi.py2 and one would call testsuite.add_test() to register the new
118   test so that it gets executed as a part of the ACPI tests.
119   It might be occasionally necessary to disable some subtests or add a new
120   test that belongs to a test suite not already present in this directory. To
121   do this, please clone the bits source from
122   https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
123   Note that this is the "qemu-bits" branch and not the "bits" branch of the
124   repository. "qemu-bits" is the branch where we have made all the QEMU
125   specific enhancements and we must use the source from this branch only.
126   Copy the test suite/script that needs modification (addition of new tests
127   or disabling them) from python directory into this directory. For
128   example, in order to change cpuid related tests, copy the following
129   file into this directory and rename it with .py2 extension:
130   https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
131   Then make your additions and changes here. Therefore, the steps are:
132
133       (a) Copy unmodified test script to this directory from bits source.
134       (b) Add a SPDX license header.
135       (c) Perform modifications to the test.
136
137   Commits (a), (b) and (c) preferably should go under separate commits so that
138   the original test script and the changes we have made are separated and
139   clear. (a) and (b) can sometimes be combined into a single step.
140
141   The test framework will then use your modified test script to run the test.
142   No further changes would be needed. Please check the logs to make sure that
143   appropriate changes have taken effect.
144
145   The tests have an extension .py2 in order to indicate that:
146
147   (a) They are python2.7 based scripts and not python 3 scripts.
148   (b) They are run from within the bios bits VM and is not subjected to QEMU
149       build/test python script maintenance and dependency resolutions.
150   (c) They need not be loaded by the test framework by accident when running
151       tests.
152
153
154Author: Ani Sinha <anisinha@redhat.com>
155
156