1======== 2Fuzzing 3======== 4 5This document describes the virtual-device fuzzing infrastructure in QEMU and 6how to use it to implement additional fuzzers. 7 8Basics 9------ 10 11Fuzzing operates by passing inputs to an entry point/target function. The 12fuzzer tracks the code coverage triggered by the input. Based on these 13findings, the fuzzer mutates the input and repeats the fuzzing. 14 15To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer 16is an *in-process* fuzzer. For the developer, this means that it is their 17responsibility to ensure that state is reset between fuzzing-runs. 18 19Building the fuzzers 20-------------------- 21 22To build the fuzzers, install a recent version of clang: 23Configure with (substitute the clang binaries with the version you installed). 24Here, enable-asan and enable-ubsan are optional but they allow us to reliably 25detect bugs such as out-of-bounds accesses, uses-after-free, double-frees 26etc.:: 27 28 CC=clang-8 CXX=clang++-8 /path/to/configure \ 29 --enable-fuzzing --enable-asan --enable-ubsan 30 31Fuzz targets are built similarly to system targets:: 32 33 make qemu-fuzz-i386 34 35This builds ``./qemu-fuzz-i386`` 36 37The first option to this command is: ``--fuzz-target=FUZZ_NAME`` 38To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments. 39 40For example:: 41 42 ./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz 43 44Internally, libfuzzer parses all arguments that do not begin with ``"--"``. 45Information about these is available by passing ``-help=1`` 46 47Now the only thing left to do is wait for the fuzzer to trigger potential 48crashes. 49 50Useful libFuzzer flags 51---------------------- 52 53As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will 54list the available arguments. In particular, these arguments might be helpful: 55 56* ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer. 57 libFuzzer stores each "interesting" input in this corpus directory. The next 58 time you run libFuzzer, it will read all of the inputs from the corpus, and 59 continue fuzzing from there. You can also specify multiple directories. 60 libFuzzer loads existing inputs from all specified directories, but will only 61 write new ones to the first one specified. 62 63* ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer 64 will generate. 65 66* ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that 67 trigger many debug/error messages, or create output on the serial console. 68 69* ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in 70 parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only 71 ``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal 72 to half the available CPU cores. Replace 4 with a number appropriate for your 73 machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel 74 fuzzers to share information about the interesting inputs they find. 75 76* ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes 77 ``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the 78 coverage table. Useful for targets with "magic" constants. If Arg1 came from 79 the fuzzer's input and Arg2 is a magic constant, then each time the Hamming 80 distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the 81 corpus. 82 83* ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to 84 better coverage performance, depending on the target. 85 86Note that libFuzzer's exact behavior will depend on the version of 87clang and libFuzzer used to build the device fuzzers. 88 89Generating Coverage Reports 90--------------------------- 91 92Code coverage is a crucial metric for evaluating a fuzzer's performance. 93libFuzzer's output provides a "cov: " column that provides a total number of 94unique blocks/edges covered. To examine coverage on a line-by-line basis we 95can use Clang coverage: 96 97 1. Configure libFuzzer to store a corpus of all interesting inputs (see 98 CORPUS_DIR above) 99 2. ``./configure`` the QEMU build with :: 100 101 --enable-fuzzing \ 102 --extra-cflags="-fprofile-instr-generate -fcoverage-mapping" 103 104 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer 105 to execute all of the inputs in $CORPUS_DIR and exit. Once the process 106 exits, you should find a file, "default.profraw" in the working directory. 107 4. Execute these commands to generate a detailed HTML coverage-report:: 108 109 llvm-profdata merge -output=default.profdata default.profraw 110 llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \ 111 --format html -output-dir=/path/to/output/report 112 113Adding a new fuzzer 114------------------- 115 116Coverage over virtual devices can be improved by adding additional fuzzers. 117Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to 118``tests/qtest/fuzz/meson.build`` 119 120Fuzzers can rely on both qtest and libqos to communicate with virtual devices. 121 1221. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``. 123 1242. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers 125 for reference. 126 1273. Add the fuzzer to ``tests/qtest/fuzz/meson.build``. 128 129Fuzzers can be more-or-less thought of as special qtest programs which can 130modify the qtest commands and/or qtest command arguments based on inputs 131provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the 132fuzzer loops over the byte-array interpreting it as a list of qtest commands, 133addresses, or values. 134 135The Generic Fuzzer 136------------------ 137 138Writing a fuzz target can be a lot of effort (especially if a device driver has 139not be built-out within libqos). Many devices can be fuzzed to some degree, 140without any device-specific code, using the generic-fuzz target. 141 142The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO, 143and DMA input-spaces. To apply the generic-fuzz to a device, we need to define 144two env-variables, at minimum: 145 146* ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with 147 the device attached. For example, if we want to fuzz the virtio-net device 148 attached to a pc-i440fx machine, we can specify:: 149 150 QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \ 151 -device virtio-net,netdev=user0" 152 153* ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify 154 the MemoryRegions that will be fuzzed. These strings are compared against 155 MemoryRegion names and MemoryRegion owner names, to decide whether each 156 MemoryRegion should be fuzzed. These strings support globbing. For the 157 virtio-net example, we could use one of :: 158 159 QEMU_FUZZ_OBJECTS='virtio-net' 160 QEMU_FUZZ_OBJECTS='virtio*' 161 QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker 162 QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine`` 163 164The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially 165useful for identifying the ``MemoryRegion`` and ``Object`` names used for 166matching. 167 168As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the 169greater the input-space, and the smaller the probability of finding crashing 170inputs for individual devices. As such, it is usually a good idea to limit the 171fuzzer to only a few ``MemoryRegions``. 172 173To ensure that these env variables have been configured correctly, we can use:: 174 175 ./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0 176 177The output should contain a complete list of matched MemoryRegions. 178 179OSS-Fuzz 180-------- 181QEMU is continuously fuzzed on `OSS-Fuzz 182<https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build 183will try to fuzz every fuzz-target. Since the generic-fuzz target 184requires additional information provided in environment variables, we 185pre-define some generic-fuzz configs in 186``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify: 187 188- ``.name``: To identify the fuzzer config 189 190- ``.args`` OR ``.argfunc``: A string or pointer to a function returning a 191 string. These strings are used to specify the ``QEMU_FUZZ_ARGS`` 192 environment variable. ``argfunc`` is useful when the config relies on e.g. 193 a dynamically created temp directory, or a free tcp/udp port. 194 195- ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment 196 variable. 197 198To fuzz additional devices/device configuration on OSS-Fuzz, send patches for 199either a new device-specific fuzzer or a new generic-fuzz config. 200 201Build details: 202 203- The Dockerfile that sets up the environment for building QEMU's 204 fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository 205 __(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile) 206 207- The script responsible for building the fuzzers can be found in the 208 QEMU source tree at ``scripts/oss-fuzz/build.sh`` 209 210Building Crash Reproducers 211----------------------------------------- 212When we find a crash, we should try to create an independent reproducer, that 213can be used on a non-fuzzer build of QEMU. This filters out any potential 214false-positives, and improves the debugging experience for developers. 215Here are the steps for building a reproducer for a crash found by the 216generic-fuzz target. 217 218- Ensure the crash reproduces:: 219 220 qemu-fuzz-i386 --fuzz-target... ./crash-... 221 222- Gather the QTest output for the crash:: 223 224 QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \ 225 qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace 226 227- Reorder and clean-up the resulting trace:: 228 229 scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer 230 231- Get the arguments needed to start qemu, and provide a path to qemu:: 232 233 less /tmp/trace # The args should be logged at the top of this file 234 export QEMU_ARGS="-machine ..." 235 export QEMU_PATH="path/to/qemu-system" 236 237- Ensure the crash reproduces in qemu-system:: 238 239 $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer 240 241- From the crash output, obtain some string that identifies the crash. This 242 can be a line in the stack-trace, for example:: 243 244 export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865" 245 246- Minimize the reproducer:: 247 248 scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \ 249 /tmp/reproducer /tmp/reproducer-minimized 250 251- Confirm that the minimized reproducer still crashes:: 252 253 $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized 254 255- Create a one-liner reproducer that can be sent over email:: 256 257 ./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized 258 259- Output the C source code for a test case that will reproduce the bug:: 260 261 ./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <john@smith.com>"\ 262 -name "test_function_name" /tmp/reproducer-minimized 263 264- Report the bug and send a patch with the C reproducer upstream 265 266Implementation Details / Fuzzer Lifecycle 267----------------------------------------- 268 269The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's 270own ``main()``, which performs some setup, and calls the entrypoints: 271 272``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the 273necessary state 274 275``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and 276resets the state at the end of each run. 277 278In more detail: 279 280``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two 281dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments 282select the fuzz target. Then, the qtest client is initialized. If the target 283requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized. 284Then the QGraph is walked and the QEMU cmd_line is determined and saved. 285 286After this, the ``vl.c:main`` is called to set up the guest. There are 287target-specific hooks that can be called before and after main, for 288additional setup(e.g. PCI setup, or VM snapshotting). 289 290``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz 291input. It is also responsible for manually calling ``main_loop_wait`` to ensure 292that bottom halves are executed and any cleanup required before the next input. 293 294Since the same process is reused for many fuzzing runs, QEMU state needs to 295be reset at the end of each run. For example, this can be done by rebooting the 296VM, after each run. 297 298 - *Pros*: Straightforward and fast for simple fuzz targets. 299 300 - *Cons*: Depending on the device, does not reset all device state. If the 301 device requires some initialization prior to being ready for fuzzing (common 302 for QOS-based targets), this initialization needs to be done after each 303 reboot. 304 305 - *Example target*: ``i440fx-qtest-reboot-fuzz`` 306