1The QEMU throttling infrastructure 2================================== 3Copyright (C) 2016 Igalia, S.L. 4Author: Alberto Garcia <berto@igalia.com> 5 6This work is licensed under the terms of the GNU GPL, version 2 or 7later. See the COPYING file in the top-level directory. 8 9Introduction 10------------ 11QEMU includes a throttling module that can be used to set limits to 12I/O operations. The code itself is generic and independent of the I/O 13units, but it is currently used to limit the number of bytes per second 14and operations per second (IOPS) when performing disk I/O. 15 16This document explains how to use the throttling code in QEMU, and how 17it works internally. The implementation is in throttle.c. 18 19 20Using throttling to limit disk I/O 21---------------------------------- 22Two aspects of the disk I/O can be limited: the number of bytes per 23second and the number of operations per second (IOPS). For each one of 24them the user can set a global limit or separate limits for read and 25write operations. This gives us a total of six different parameters. 26 27I/O limits can be set using the throttling.* parameters of -drive, or 28using the QMP 'block_set_io_throttle' command. These are the names of 29the parameters for both cases: 30 31|-----------------------+-----------------------| 32| -drive | block_set_io_throttle | 33|-----------------------+-----------------------| 34| throttling.iops-total | iops | 35| throttling.iops-read | iops_rd | 36| throttling.iops-write | iops_wr | 37| throttling.bps-total | bps | 38| throttling.bps-read | bps_rd | 39| throttling.bps-write | bps_wr | 40|-----------------------+-----------------------| 41 42It is possible to set limits for both IOPS and bps at the same time, 43and for each case we can decide whether to have separate read and 44write limits or not, but note that if iops-total is set then neither 45iops-read nor iops-write can be set. The same applies to bps-total and 46bps-read/write. 47 48The default value of these parameters is 0, and it means 'unlimited'. 49 50In its most basic usage, the user can add a drive to QEMU with a limit 51of 100 IOPS with the following -drive line: 52 53 -drive file=hd0.qcow2,throttling.iops-total=100 54 55We can do the same using QMP. In this case all these parameters are 56mandatory, so we must set to 0 the ones that we don't want to limit: 57 58 { "execute": "block_set_io_throttle", 59 "arguments": { 60 "device": "virtio0", 61 "iops": 100, 62 "iops_rd": 0, 63 "iops_wr": 0, 64 "bps": 0, 65 "bps_rd": 0, 66 "bps_wr": 0 67 } 68 } 69 70 71I/O bursts 72---------- 73In addition to the basic limits we have just seen, QEMU allows the 74user to do bursts of I/O for a configurable amount of time. A burst is 75an amount of I/O that can exceed the basic limit. Bursts are useful to 76allow better performance when there are peaks of activity (the OS 77boots, a service needs to be restarted) while keeping the average 78limits lower the rest of the time. 79 80Two parameters control bursts: their length and the maximum amount of 81I/O they allow. These two can be configured separately for each one of 82the six basic parameters described in the previous section, but in 83this section we'll use 'iops-total' as an example. 84 85The I/O limit during bursts is set using 'iops-total-max', and the 86maximum length (in seconds) is set with 'iops-total-max-length'. So if 87we want to configure a drive with a basic limit of 100 IOPS and allow 88bursts of 2000 IOPS for 60 seconds, we would do it like this (the line 89is split for clarity): 90 91 -drive file=hd0.qcow2, 92 throttling.iops-total=100, 93 throttling.iops-total-max=2000, 94 throttling.iops-total-max-length=60 95 96Or, with QMP: 97 98 { "execute": "block_set_io_throttle", 99 "arguments": { 100 "device": "virtio0", 101 "iops": 100, 102 "iops_rd": 0, 103 "iops_wr": 0, 104 "bps": 0, 105 "bps_rd": 0, 106 "bps_wr": 0, 107 "iops_max": 2000, 108 "iops_max_length": 60, 109 } 110 } 111 112With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 113IOPS for 1 minute before it's throttled down to 100 IOPS. 114 115The user will be able to do bursts again if there's a sufficiently 116long period of time with unused I/O (see below for details). 117 118The default value for 'iops-total-max' is 0 and it means that bursts 119are not allowed. 'iops-total-max-length' can only be set if 120'iops-total-max' is set as well, and its default value is 1 second. 121 122Here's the complete list of parameters for configuring bursts: 123 124|----------------------------------+-----------------------| 125| -drive | block_set_io_throttle | 126|----------------------------------+-----------------------| 127| throttling.iops-total-max | iops_max | 128| throttling.iops-total-max-length | iops_max_length | 129| throttling.iops-read-max | iops_rd_max | 130| throttling.iops-read-max-length | iops_rd_max_length | 131| throttling.iops-write-max | iops_wr_max | 132| throttling.iops-write-max-length | iops_wr_max_length | 133| throttling.bps-total-max | bps_max | 134| throttling.bps-total-max-length | bps_max_length | 135| throttling.bps-read-max | bps_rd_max | 136| throttling.bps-read-max-length | bps_rd_max_length | 137| throttling.bps-write-max | bps_wr_max | 138| throttling.bps-write-max-length | bps_wr_max_length | 139|----------------------------------+-----------------------| 140 141 142Controlling the size of I/O operations 143-------------------------------------- 144When applying IOPS limits all I/O operations are treated equally 145regardless of their size. This means that the user can take advantage 146of this in order to circumvent the limits and submit one huge I/O 147request instead of several smaller ones. 148 149QEMU provides a setting called throttling.iops-size to prevent this 150from happening. This setting specifies the size (in bytes) of an I/O 151request for accounting purposes. Larger requests will be counted 152proportionally to this size. 153 154For example, if iops-size is set to 4096 then an 8KB request will be 155counted as two, and a 6KB request will be counted as one and a 156half. This only applies to requests larger than iops-size: smaller 157requests will be always counted as one, no matter their size. 158 159The default value of iops-size is 0 and it means that the size of the 160requests is never taken into account when applying IOPS limits. 161 162 163Applying I/O limits to groups of disks 164-------------------------------------- 165In all the examples so far we have seen how to apply limits to the I/O 166performed on individual drives, but QEMU allows grouping drives so 167they all share the same limits. 168 169The way it works is that each drive with I/O limits is assigned to a 170group named using the throttling.group parameter. If this parameter is 171not specified, then the device name (i.e. 'virtio0', 'ide0-hd0') will 172be used as the group name. 173 174Limits set using the throttling.* parameters discussed earlier in this 175document apply to the combined I/O of all members of a group. 176 177Consider this example: 178 179 -drive file=hd1.qcow2,throttling.iops-total=6000,throttling.group=foo 180 -drive file=hd2.qcow2,throttling.iops-total=6000,throttling.group=foo 181 -drive file=hd3.qcow2,throttling.iops-total=3000,throttling.group=bar 182 -drive file=hd4.qcow2,throttling.iops-total=6000,throttling.group=foo 183 -drive file=hd5.qcow2,throttling.iops-total=3000,throttling.group=bar 184 -drive file=hd6.qcow2,throttling.iops-total=5000 185 186Here hd1, hd2 and hd4 are all members of a group named 'foo' with a 187combined IOPS limit of 6000, and hd3 and hd5 are members of 'bar'. hd6 188is left alone (technically it is part of a 1-member group). 189 190Limits are applied in a round-robin fashion so if there are concurrent 191I/O requests on several drives of the same group they will be 192distributed evenly. 193 194When I/O limits are applied to an existing drive using the QMP command 195'block_set_io_throttle', the following things need to be taken into 196account: 197 198 - I/O limits are shared within the same group, so new values will 199 affect all members and overwrite the previous settings. In other 200 words: if different limits are applied to members of the same 201 group, the last one wins. 202 203 - If 'group' is unset it is assumed to be the current group of that 204 drive. If the drive is not in a group yet, it will be added to a 205 group named after the device name. 206 207 - If 'group' is set then the drive will be moved to that group if 208 it was member of a different one. In this case the limits 209 specified in the parameters will be applied to the new group 210 only. 211 212 - I/O limits can be disabled by setting all of them to 0. In this 213 case the device will be removed from its group and the rest of 214 its members will not be affected. The 'group' parameter is 215 ignored. 216 217 218The Leaky Bucket algorithm 219-------------------------- 220I/O limits in QEMU are implemented using the leaky bucket algorithm 221(specifically the "Leaky bucket as a meter" variant). 222 223This algorithm uses the analogy of a bucket that leaks water 224constantly. The water that gets into the bucket represents the I/O 225that has been performed, and no more I/O is allowed once the bucket is 226full. 227 228To see the way this corresponds to the throttling parameters in QEMU, 229consider the following values: 230 231 iops-total=100 232 iops-total-max=2000 233 iops-total-max-length=60 234 235 - Water leaks from the bucket at a rate of 100 IOPS. 236 - Water can be added to the bucket at a rate of 2000 IOPS. 237 - The size of the bucket is 2000 x 60 = 120000 238 - If 'iops-total-max-length' is unset then it defaults to 1 and the 239 size of the bucket is 2000. 240 - If 'iops-total-max' is unset then 'iops-total-max-length' must be 241 unset as well. In this case the bucket size is 100. 242 243The bucket is initially empty, therefore water can be added until it's 244full at a rate of 2000 IOPS (the burst rate). Once the bucket is full 245we can only add as much water as it leaks, therefore the I/O rate is 246reduced to 100 IOPS. If we add less water than it leaks then the 247bucket will start to empty, allowing for bursts again. 248 249Note that since water is leaking from the bucket even during bursts, 250it will take a bit more than 60 seconds at 2000 IOPS to fill it 251up. After those 60 seconds the bucket will have leaked 60 x 100 = 2526000, allowing for 3 more seconds of I/O at 2000 IOPS. 253 254Also, due to the way the algorithm works, longer burst can be done at 255a lower I/O rate, e.g. 1000 IOPS during 120 seconds. 256