1================================== 2ACPI/SMBIOS testing using biosbits 3================================== 4************ 5Introduction 6************ 7Biosbits is a software written by Josh Triplett that can be downloaded 8from https://biosbits.org/. The github codebase can be found 9`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that 10executes the bios components such as acpi and smbios tables directly through 11acpica bios interpreter (a freely available C based library written by Intel, 12downloadable from https://acpica.org/ and is included with biosbits) without an 13operating system getting involved in between. Bios-bits has python integration 14with grub so actual routines that executes bios components can be written in 15python instead of bash-ish (grub's native scripting language). 16There are several advantages to directly testing the bios in a real physical 17machine or in a VM as opposed to indirectly discovering bios issues through the 18operating system (the OS). Operating systems tend to bypass bios problems and 19hide them from the end user. We have more control of what we wanted to test and 20how by being as close to the bios on a running system as possible without a 21complicated software component such as an operating system coming in between. 22Another issue is that we cannot exercise bios components such as ACPI and 23SMBIOS without being in the highest hardware privilege level, ring 0 for 24example in case of x86. Since the OS executes from ring 0 whereas normal user 25land software resides in unprivileged ring 3, operating system must be modified 26in order to write our test routines that exercise and test the bios. This is 27not possible in all cases. Lastly, test frameworks and routines are preferably 28written using a high level scripting language such as python. OSes and 29OS modules are generally written using low level languages such as C and 30low level assembly machine language. Writing test routines in a low level 31language makes things more cumbersome. These and other reasons makes using 32bios-bits very attractive for testing bioses. More details on the inspiration 33for developing biosbits and its real life uses can be found in [#a]_ and [#b]_. 34 35For QEMU, we maintain a fork of bios bits in gitlab along with all the 36dependent submodules `here <https://gitlab.com/qemu-project/biosbits-bits>`__. 37This fork contains numerous fixes, a newer acpica and changes specific to 38running these functional QEMU tests using bits. The author of this document 39is the sole maintainer of the QEMU fork of bios bits repository. For more 40information, please see author's `FOSDEM talk on this bios-bits based test 41framework <https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/>`__. 42 43********************************* 44Description of the test framework 45********************************* 46 47Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU 48functional test that drives all this. 49 50A brief description of the various test files follows. 51 52Under ``tests/functional/`` as the root we have: 53 54:: 55 56 ├── acpi-bits 57 │ ├── bits-config 58 │ │ └── bits-cfg.txt 59 │ ├── bits-tests 60 │ ├── smbios.py2 61 │ ├── testacpi.py2 62 │ └── testcpuid.py2 63 ├── test_acpi_bits.py 64 65* ``tests/functional``: 66 67 ``test_acpi_bits.py``: 68 This is the main python functional test script that generates a 69 biosbits iso. It then spawns a QEMU VM with it, collects the log and reports 70 test failures. This is the script one would be interested in if they wanted 71 to add or change some component of the log parsing, add a new command line 72 to alter how QEMU is spawned etc. Test writers typically would not need to 73 modify this script unless they wanted to enhance or change the log parsing 74 for their tests. In order to enable debugging, you can set **V=1** 75 environment variable. This enables verbose mode for the test and also dumps 76 the entire log from bios bits and more information in case failure happens. 77 You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable 78 verbose logs and also retain the temporary work directory the test used for 79 you to inspect and run the specific commands manually. 80 81 In order to run this test, please perform the following steps from the QEMU 82 build directory (assuming that the sources are in ".."): 83 :: 84 85 $ export PYTHONPATH=../python:../tests/functional 86 $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64 87 $ python3 ../tests/functional/test_acpi_bits.py 88 89 The above will run all acpi-bits functional tests (producing output in 90 tap format). 91 92 You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/ 93 for more information about the run or in order to diagnoze issues. 94 If you pass V=1 in the environment, more diagnostic logs will be put into 95 the test log. 96 97* ``tests/functional/acpi-bits/bits-config``: 98 99 This location contains biosbits configuration files that determine how the 100 software runs the tests. 101 102 ``bits-config.txt``: 103 This is the biosbits config file that determines what tests 104 or actions are performed by bits. The description of the config options are 105 provided in the file itself. 106 107* ``tests/functional/acpi-bits/bits-tests``: 108 109 This directory contains biosbits python based tests that are run from within 110 the biosbits environment in the spawned VM. New additions of test cases can 111 be made in the appropriate test file. For example, new acpi tests can go 112 into testacpi.py2 and one would call testsuite.add_test() to register the new 113 test so that it gets executed as a part of the ACPI tests. 114 It might be occasionally necessary to disable some subtests or add a new 115 test that belongs to a test suite not already present in this directory. To 116 do this, please clone the bits source from 117 https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits. 118 Note that this is the "qemu-bits" branch and not the "bits" branch of the 119 repository. "qemu-bits" is the branch where we have made all the QEMU 120 specific enhancements and we must use the source from this branch only. 121 Copy the test suite/script that needs modification (addition of new tests 122 or disabling them) from python directory into this directory. For 123 example, in order to change cpuid related tests, copy the following 124 file into this directory and rename it with .py2 extension: 125 https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py 126 Then make your additions and changes here. Therefore, the steps are: 127 128 (a) Copy unmodified test script to this directory from bits source. 129 (b) Add a SPDX license header. 130 (c) Perform modifications to the test. 131 132 Commits (a), (b) and (c) preferably should go under separate commits so that 133 the original test script and the changes we have made are separated and 134 clear. (a) and (b) can sometimes be combined into a single step. 135 136 The test framework will then use your modified test script to run the test. 137 No further changes would be needed. Please check the logs to make sure that 138 appropriate changes have taken effect. 139 140 The tests have an extension .py2 in order to indicate that: 141 142 (a) They are python2.7 based scripts and not python 3 scripts. 143 (b) They are run from within the bios bits VM and is not subjected to QEMU 144 build/test python script maintenance and dependency resolutions. 145 (c) They need not be loaded by the test framework by accident when running 146 tests. 147 148 149Author: Ani Sinha <anisinha@redhat.com> 150 151References: 152----------- 153.. [#a] https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf 154.. [#b] https://www.youtube.com/watch?v=36QIepyUuhg 155.. [#c] https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/ 156