1======== 2Fuzzing 3======== 4 5This document describes the virtual-device fuzzing infrastructure in QEMU and 6how to use it to implement additional fuzzers. 7 8Basics 9------ 10 11Fuzzing operates by passing inputs to an entry point/target function. The 12fuzzer tracks the code coverage triggered by the input. Based on these 13findings, the fuzzer mutates the input and repeats the fuzzing. 14 15To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer 16is an *in-process* fuzzer. For the developer, this means that it is their 17responsibility to ensure that state is reset between fuzzing-runs. 18 19Building the fuzzers 20-------------------- 21 22To build the fuzzers, install a recent version of clang: 23Configure with (substitute the clang binaries with the version you installed). 24Here, enable-sanitizers, is optional but it allows us to reliably detect bugs 25such as out-of-bounds accesses, use-after-frees, double-frees etc.:: 26 27 CC=clang-8 CXX=clang++-8 /path/to/configure \ 28 --enable-fuzzing --enable-asan --enable-ubsan 29 30Fuzz targets are built similarly to system targets:: 31 32 make qemu-fuzz-i386 33 34This builds ``./qemu-fuzz-i386`` 35 36The first option to this command is: ``--fuzz-target=FUZZ_NAME`` 37To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments. 38 39For example:: 40 41 ./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz 42 43Internally, libfuzzer parses all arguments that do not begin with ``"--"``. 44Information about these is available by passing ``-help=1`` 45 46Now the only thing left to do is wait for the fuzzer to trigger potential 47crashes. 48 49Useful libFuzzer flags 50---------------------- 51 52As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will 53list the available arguments. In particular, these arguments might be helpful: 54 55* ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer. 56 libFuzzer stores each "interesting" input in this corpus directory. The next 57 time you run libFuzzer, it will read all of the inputs from the corpus, and 58 continue fuzzing from there. You can also specify multiple directories. 59 libFuzzer loads existing inputs from all specified directories, but will only 60 write new ones to the first one specified. 61 62* ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer 63 will generate. 64 65* ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that 66 trigger many debug/error messages, or create output on the serial console. 67 68* ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in 69 parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only 70 ``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal 71 to half the available CPU cores. Replace 4 with a number appropriate for your 72 machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel 73 fuzzers to share information about the interesting inputs they find. 74 75* ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes 76 ``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the 77 coverage table. Useful for targets with "magic" constants. If Arg1 came from 78 the fuzzer's input and Arg2 is a magic constant, then each time the Hamming 79 distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the 80 corpus. 81 82* ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to 83 better coverage performance, depending on the target. 84 85Note that libFuzzer's exact behavior will depend on the version of 86clang and libFuzzer used to build the device fuzzers. 87 88Generating Coverage Reports 89--------------------------- 90 91Code coverage is a crucial metric for evaluating a fuzzer's performance. 92libFuzzer's output provides a "cov: " column that provides a total number of 93unique blocks/edges covered. To examine coverage on a line-by-line basis we 94can use Clang coverage: 95 96 1. Configure libFuzzer to store a corpus of all interesting inputs (see 97 CORPUS_DIR above) 98 2. ``./configure`` the QEMU build with :: 99 100 --enable-fuzzing \ 101 --extra-cflags="-fprofile-instr-generate -fcoverage-mapping" 102 103 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer 104 to execute all of the inputs in $CORPUS_DIR and exit. Once the process 105 exits, you should find a file, "default.profraw" in the working directory. 106 4. Execute these commands to generate a detailed HTML coverage-report:: 107 108 llvm-profdata merge -output=default.profdata default.profraw 109 llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \ 110 --format html -output-dir=/path/to/output/report 111 112Adding a new fuzzer 113------------------- 114 115Coverage over virtual devices can be improved by adding additional fuzzers. 116Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to 117``tests/qtest/fuzz/meson.build`` 118 119Fuzzers can rely on both qtest and libqos to communicate with virtual devices. 120 1211. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``. 122 1232. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers 124 for reference. 125 1263. Add the fuzzer to ``tests/qtest/fuzz/meson.build``. 127 128Fuzzers can be more-or-less thought of as special qtest programs which can 129modify the qtest commands and/or qtest command arguments based on inputs 130provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the 131fuzzer loops over the byte-array interpreting it as a list of qtest commands, 132addresses, or values. 133 134The Generic Fuzzer 135------------------ 136 137Writing a fuzz target can be a lot of effort (especially if a device driver has 138not be built-out within libqos). Many devices can be fuzzed to some degree, 139without any device-specific code, using the generic-fuzz target. 140 141The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO, 142and DMA input-spaces. To apply the generic-fuzz to a device, we need to define 143two env-variables, at minimum: 144 145* ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with 146 the device attached. For example, if we want to fuzz the virtio-net device 147 attached to a pc-i440fx machine, we can specify:: 148 149 QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \ 150 -device virtio-net,netdev=user0" 151 152* ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify 153 the MemoryRegions that will be fuzzed. These strings are compared against 154 MemoryRegion names and MemoryRegion owner names, to decide whether each 155 MemoryRegion should be fuzzed. These strings support globbing. For the 156 virtio-net example, we could use one of :: 157 158 QEMU_FUZZ_OBJECTS='virtio-net' 159 QEMU_FUZZ_OBJECTS='virtio*' 160 QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker 161 QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine`` 162 163The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially 164useful for identifying the ``MemoryRegion`` and ``Object`` names used for 165matching. 166 167As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the 168greater the input-space, and the smaller the probability of finding crashing 169inputs for individual devices. As such, it is usually a good idea to limit the 170fuzzer to only a few ``MemoryRegions``. 171 172To ensure that these env variables have been configured correctly, we can use:: 173 174 ./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0 175 176The output should contain a complete list of matched MemoryRegions. 177 178OSS-Fuzz 179-------- 180QEMU is continuously fuzzed on `OSS-Fuzz 181<https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build 182will try to fuzz every fuzz-target. Since the generic-fuzz target 183requires additional information provided in environment variables, we 184pre-define some generic-fuzz configs in 185``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify: 186 187- ``.name``: To identify the fuzzer config 188 189- ``.args`` OR ``.argfunc``: A string or pointer to a function returning a 190 string. These strings are used to specify the ``QEMU_FUZZ_ARGS`` 191 environment variable. ``argfunc`` is useful when the config relies on e.g. 192 a dynamically created temp directory, or a free tcp/udp port. 193 194- ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment 195 variable. 196 197To fuzz additional devices/device configuration on OSS-Fuzz, send patches for 198either a new device-specific fuzzer or a new generic-fuzz config. 199 200Build details: 201 202- The Dockerfile that sets up the environment for building QEMU's 203 fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository 204 __(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile) 205 206- The script responsible for building the fuzzers can be found in the 207 QEMU source tree at ``scripts/oss-fuzz/build.sh`` 208 209Building Crash Reproducers 210----------------------------------------- 211When we find a crash, we should try to create an independent reproducer, that 212can be used on a non-fuzzer build of QEMU. This filters out any potential 213false-positives, and improves the debugging experience for developers. 214Here are the steps for building a reproducer for a crash found by the 215generic-fuzz target. 216 217- Ensure the crash reproduces:: 218 219 qemu-fuzz-i386 --fuzz-target... ./crash-... 220 221- Gather the QTest output for the crash:: 222 223 QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \ 224 qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace 225 226- Reorder and clean-up the resulting trace:: 227 228 scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer 229 230- Get the arguments needed to start qemu, and provide a path to qemu:: 231 232 less /tmp/trace # The args should be logged at the top of this file 233 export QEMU_ARGS="-machine ..." 234 export QEMU_PATH="path/to/qemu-system" 235 236- Ensure the crash reproduces in qemu-system:: 237 238 $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer 239 240- From the crash output, obtain some string that identifies the crash. This 241 can be a line in the stack-trace, for example:: 242 243 export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865" 244 245- Minimize the reproducer:: 246 247 scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \ 248 /tmp/reproducer /tmp/reproducer-minimized 249 250- Confirm that the minimized reproducer still crashes:: 251 252 $QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized 253 254- Create a one-liner reproducer that can be sent over email:: 255 256 ./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized 257 258- Output the C source code for a test case that will reproduce the bug:: 259 260 ./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <john@smith.com>"\ 261 -name "test_function_name" /tmp/reproducer-minimized 262 263- Report the bug and send a patch with the C reproducer upstream 264 265Implementation Details / Fuzzer Lifecycle 266----------------------------------------- 267 268The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's 269own ``main()``, which performs some setup, and calls the entrypoints: 270 271``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the 272necessary state 273 274``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and 275resets the state at the end of each run. 276 277In more detail: 278 279``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two 280dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments 281select the fuzz target. Then, the qtest client is initialized. If the target 282requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized. 283Then the QGraph is walked and the QEMU cmd_line is determined and saved. 284 285After this, the ``vl.c:main`` is called to set up the guest. There are 286target-specific hooks that can be called before and after main, for 287additional setup(e.g. PCI setup, or VM snapshotting). 288 289``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz 290input. It is also responsible for manually calling ``main_loop_wait`` to ensure 291that bottom halves are executed and any cleanup required before the next input. 292 293Since the same process is reused for many fuzzing runs, QEMU state needs to 294be reset at the end of each run. For example, this can be done by rebooting the 295VM, after each run. 296 297 - *Pros*: Straightforward and fast for simple fuzz targets. 298 299 - *Cons*: Depending on the device, does not reset all device state. If the 300 device requires some initialization prior to being ready for fuzzing (common 301 for QOS-based targets), this initialization needs to be done after each 302 reboot. 303 304 - *Example target*: ``i440fx-qtest-reboot-fuzz`` 305