1Block I/O error injection using ``blkdebug`` 2============================================ 3 4.. 5 Copyright (C) 2014-2015 Red Hat Inc 6 7 This work is licensed under the terms of the GNU GPL, version 2 or later. See 8 the COPYING file in the top-level directory. 9 10The ``blkdebug`` block driver is a rule-based error injection engine. It can be 11used to exercise error code paths in block drivers including ``ENOSPC`` (out of 12space) and ``EIO``. 13 14This document gives an overview of the features available in ``blkdebug``. 15 16Background 17---------- 18Block drivers have many error code paths that handle I/O errors. Image formats 19are especially complex since metadata I/O errors during cluster allocation or 20while updating tables happen halfway through request processing and require 21discipline to keep image files consistent. 22 23Error injection allows test cases to trigger I/O errors at specific points. 24This way, all error paths can be tested to make sure they are correct. 25 26Rules 27----- 28The ``blkdebug`` block driver takes a list of "rules" that tell the error injection 29engine when to fail an I/O request. 30 31Each I/O request is evaluated against the rules. If a rule matches the request 32then its "action" is executed. 33 34Rules can be placed in a configuration file; the configuration file 35follows the same .ini-like format used by QEMU's ``-readconfig`` option, and 36each section of the file represents a rule. 37 38The following configuration file defines a single rule:: 39 40 $ cat blkdebug.conf 41 [inject-error] 42 event = "read_aio" 43 errno = "28" 44 45This rule fails all aio read requests with ``ENOSPC`` (28). Note that the errno 46value depends on the host. On Linux, see 47``/usr/include/asm-generic/errno-base.h`` for errno values. 48 49Invoke QEMU as follows:: 50 51 $ qemu-system-x86_64 52 -drive if=none,cache=none,file=blkdebug:blkdebug.conf:test.img,id=drive0 \ 53 -device virtio-blk-pci,drive=drive0,id=virtio-blk-pci0 54 55Rules support the following attributes: 56 57``event`` 58 which type of operation to match (e.g. ``read_aio``, ``write_aio``, 59 ``flush_to_os``, ``flush_to_disk``). See `Events`_ for 60 information on events. 61 62``state`` 63 (optional) the engine must be in this state number in order for this 64 rule to match. See `State transitions`_ for information 65 on states. 66 67``errno`` 68 the numeric errno value to return when a request matches this rule. 69 The errno values depend on the host since the numeric values are not 70 standardized in the POSIX specification. 71 72``sector`` 73 (optional) a sector number that the request must overlap in order to 74 match this rule 75 76``once`` 77 (optional, default ``off``) only execute this action on the first 78 matching request 79 80``immediately`` 81 (optional, default ``off``) return a NULL ``BlockAIOCB`` 82 pointer and fail without an errno instead. This 83 exercises the code path where ``BlockAIOCB`` fails and the 84 caller's ``BlockCompletionFunc`` is not invoked. 85 86Events 87------ 88Block drivers provide information about the type of I/O request they are about 89to make so rules can match specific types of requests. For example, the ``qcow2`` 90block driver tells ``blkdebug`` when it accesses the L1 table so rules can match 91only L1 table accesses and not other metadata or guest data requests. 92 93The core events are: 94 95``read_aio`` 96 guest data read 97 98``write_aio`` 99 guest data write 100 101``flush_to_os`` 102 write out unwritten block driver state (e.g. cached metadata) 103 104``flush_to_disk`` 105 flush the host block device's disk cache 106 107See ``qapi/block-core.json:BlkdebugEvent`` for the full list of events. 108You may need to grep block driver source code to understand the 109meaning of specific events. 110 111State transitions 112----------------- 113There are cases where more power is needed to match a particular I/O request in 114a longer sequence of requests. For example:: 115 116 write_aio 117 flush_to_disk 118 write_aio 119 120How do we match the 2nd ``write_aio`` but not the first? This is where state 121transitions come in. 122 123The error injection engine has an integer called the "state" that always starts 124initialized to 1. The state integer is internal to ``blkdebug`` and cannot be 125observed from outside but rules can interact with it for powerful matching 126behavior. 127 128Rules can be conditional on the current state and they can transition to a new 129state. 130 131When a rule's "state" attribute is non-zero then the current state must equal 132the attribute in order for the rule to match. 133 134For example, to match the 2nd write_aio:: 135 136 [set-state] 137 event = "write_aio" 138 state = "1" 139 new_state = "2" 140 141 [inject-error] 142 event = "write_aio" 143 state = "2" 144 errno = "5" 145 146The first ``write_aio`` request matches the ``set-state`` rule and transitions from 147state 1 to state 2. Once state 2 has been entered, the ``set-state`` rule no 148longer matches since it requires state 1. But the ``inject-error`` rule now 149matches the next ``write_aio`` request and injects ``EIO`` (5). 150 151State transition rules support the following attributes: 152 153``event`` 154 which type of operation to match (e.g. ``read_aio``, ``write_aio``, 155 ``flush_to_os`, ``flush_to_disk``). See `Events`_ for 156 information on events. 157 158``state`` 159 (optional) the engine must be in this state number in order for this 160 rule to match 161 162``new_state`` 163 transition to this state number 164 165Suspend and resume 166------------------ 167Exercising code paths in block drivers may require specific ordering amongst 168concurrent requests. The "breakpoint" feature allows requests to be halted on 169a ``blkdebug`` event and resumed later. This makes it possible to achieve 170deterministic ordering when multiple requests are in flight. 171 172Breakpoints on ``blkdebug`` events are associated with a user-defined ``tag`` string. 173This tag serves as an identifier by which the request can be resumed at a later 174point. 175 176See the ``qemu-io(1)`` ``break``, ``resume``, ``remove_break``, and ``wait_break`` 177commands for details. 178