1898bd37aSMauro Carvalho Chehab==========================
2898bd37aSMauro Carvalho ChehabBFQ (Budget Fair Queueing)
3898bd37aSMauro Carvalho Chehab==========================
4898bd37aSMauro Carvalho Chehab
5898bd37aSMauro Carvalho ChehabBFQ is a proportional-share I/O scheduler, with some extra
6898bd37aSMauro Carvalho Chehablow-latency capabilities. In addition to cgroups support (blkio or io
7898bd37aSMauro Carvalho Chehabcontrollers), BFQ's main features are:
8898bd37aSMauro Carvalho Chehab
9898bd37aSMauro Carvalho Chehab- BFQ guarantees a high system and application responsiveness, and a
10898bd37aSMauro Carvalho Chehab  low latency for time-sensitive applications, such as audio or video
11898bd37aSMauro Carvalho Chehab  players;
12898bd37aSMauro Carvalho Chehab- BFQ distributes bandwidth, and not just time, among processes or
13898bd37aSMauro Carvalho Chehab  groups (switching back to time distribution when needed to keep
14898bd37aSMauro Carvalho Chehab  throughput high).
15898bd37aSMauro Carvalho Chehab
16898bd37aSMauro Carvalho ChehabIn its default configuration, BFQ privileges latency over
17898bd37aSMauro Carvalho Chehabthroughput. So, when needed for achieving a lower latency, BFQ builds
18898bd37aSMauro Carvalho Chehabschedules that may lead to a lower throughput. If your main or only
19898bd37aSMauro Carvalho Chehabgoal, for a given device, is to achieve the maximum-possible
20898bd37aSMauro Carvalho Chehabthroughput at all times, then do switch off all low-latency heuristics
21898bd37aSMauro Carvalho Chehabfor that device, by setting low_latency to 0. See Section 3 for
22898bd37aSMauro Carvalho Chehabdetails on how to configure BFQ for the desired tradeoff between
23898bd37aSMauro Carvalho Chehablatency and throughput, or on how to maximize throughput.
24898bd37aSMauro Carvalho Chehab
25898bd37aSMauro Carvalho ChehabAs every I/O scheduler, BFQ adds some overhead to per-I/O-request
26898bd37aSMauro Carvalho Chehabprocessing. To give an idea of this overhead, the total,
27898bd37aSMauro Carvalho Chehabsingle-lock-protected, per-request processing time of BFQ---i.e., the
28898bd37aSMauro Carvalho Chehabsum of the execution times of the request insertion, dispatch and
29898bd37aSMauro Carvalho Chehabcompletion hooks---is, e.g., 1.9 us on an Intel Core i7-2760QM@2.40GHz
30898bd37aSMauro Carvalho Chehab(dated CPU for notebooks; time measured with simple code
31898bd37aSMauro Carvalho Chehabinstrumentation, and using the throughput-sync.sh script of the S
32898bd37aSMauro Carvalho Chehabsuite [1], in performance-profiling mode). To put this result into
33898bd37aSMauro Carvalho Chehabcontext, the total, single-lock-protected, per-request execution time
34898bd37aSMauro Carvalho Chehabof the lightest I/O scheduler available in blk-mq, mq-deadline, is 0.7
35898bd37aSMauro Carvalho Chehabus (mq-deadline is ~800 LOC, against ~10500 LOC for BFQ).
36898bd37aSMauro Carvalho Chehab
37898bd37aSMauro Carvalho ChehabScheduling overhead further limits the maximum IOPS that a CPU can
38898bd37aSMauro Carvalho Chehabprocess (already limited by the execution of the rest of the I/O
39898bd37aSMauro Carvalho Chehabstack). To give an idea of the limits with BFQ, on slow or average
40898bd37aSMauro Carvalho ChehabCPUs, here are, first, the limits of BFQ for three different CPUs, on,
41898bd37aSMauro Carvalho Chehabrespectively, an average laptop, an old desktop, and a cheap embedded
42898bd37aSMauro Carvalho Chehabsystem, in case full hierarchical support is enabled (i.e.,
43898bd37aSMauro Carvalho ChehabCONFIG_BFQ_GROUP_IOSCHED is set), but CONFIG_BFQ_CGROUP_DEBUG is not
44898bd37aSMauro Carvalho Chehabset (Section 4-2):
45898bd37aSMauro Carvalho Chehab- Intel i7-4850HQ: 400 KIOPS
46898bd37aSMauro Carvalho Chehab- AMD A8-3850: 250 KIOPS
47898bd37aSMauro Carvalho Chehab- ARM CortexTM-A53 Octa-core: 80 KIOPS
48898bd37aSMauro Carvalho Chehab
49898bd37aSMauro Carvalho ChehabIf CONFIG_BFQ_CGROUP_DEBUG is set (and of course full hierarchical
50898bd37aSMauro Carvalho Chehabsupport is enabled), then the sustainable throughput with BFQ
51898bd37aSMauro Carvalho Chehabdecreases, because all blkio.bfq* statistics are created and updated
52898bd37aSMauro Carvalho Chehab(Section 4-2). For BFQ, this leads to the following maximum
53898bd37aSMauro Carvalho Chehabsustainable throughputs, on the same systems as above:
54898bd37aSMauro Carvalho Chehab- Intel i7-4850HQ: 310 KIOPS
55898bd37aSMauro Carvalho Chehab- AMD A8-3850: 200 KIOPS
56898bd37aSMauro Carvalho Chehab- ARM CortexTM-A53 Octa-core: 56 KIOPS
57898bd37aSMauro Carvalho Chehab
58898bd37aSMauro Carvalho ChehabBFQ works for multi-queue devices too.
59898bd37aSMauro Carvalho Chehab
60898bd37aSMauro Carvalho Chehab.. The table of contents follow. Impatients can just jump to Section 3.
61898bd37aSMauro Carvalho Chehab
62898bd37aSMauro Carvalho Chehab.. CONTENTS
63898bd37aSMauro Carvalho Chehab
64898bd37aSMauro Carvalho Chehab   1. When may BFQ be useful?
65898bd37aSMauro Carvalho Chehab    1-1 Personal systems
66898bd37aSMauro Carvalho Chehab    1-2 Server systems
67898bd37aSMauro Carvalho Chehab   2. How does BFQ work?
68898bd37aSMauro Carvalho Chehab   3. What are BFQ's tunables and how to properly configure BFQ?
69898bd37aSMauro Carvalho Chehab   4. BFQ group scheduling
70898bd37aSMauro Carvalho Chehab    4-1 Service guarantees provided
71898bd37aSMauro Carvalho Chehab    4-2 Interface
72898bd37aSMauro Carvalho Chehab
73898bd37aSMauro Carvalho Chehab1. When may BFQ be useful?
74898bd37aSMauro Carvalho Chehab==========================
75898bd37aSMauro Carvalho Chehab
76898bd37aSMauro Carvalho ChehabBFQ provides the following benefits on personal and server systems.
77898bd37aSMauro Carvalho Chehab
78898bd37aSMauro Carvalho Chehab1-1 Personal systems
79898bd37aSMauro Carvalho Chehab--------------------
80898bd37aSMauro Carvalho Chehab
81898bd37aSMauro Carvalho ChehabLow latency for interactive applications
82898bd37aSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
83898bd37aSMauro Carvalho Chehab
84898bd37aSMauro Carvalho ChehabRegardless of the actual background workload, BFQ guarantees that, for
85898bd37aSMauro Carvalho Chehabinteractive tasks, the storage device is virtually as responsive as if
86898bd37aSMauro Carvalho Chehabit was idle. For example, even if one or more of the following
87898bd37aSMauro Carvalho Chehabbackground workloads are being executed:
88898bd37aSMauro Carvalho Chehab
89898bd37aSMauro Carvalho Chehab- one or more large files are being read, written or copied,
90898bd37aSMauro Carvalho Chehab- a tree of source files is being compiled,
91898bd37aSMauro Carvalho Chehab- one or more virtual machines are performing I/O,
92898bd37aSMauro Carvalho Chehab- a software update is in progress,
93898bd37aSMauro Carvalho Chehab- indexing daemons are scanning filesystems and updating their
94898bd37aSMauro Carvalho Chehab  databases,
95898bd37aSMauro Carvalho Chehab
96898bd37aSMauro Carvalho Chehabstarting an application or loading a file from within an application
97898bd37aSMauro Carvalho Chehabtakes about the same time as if the storage device was idle. As a
98898bd37aSMauro Carvalho Chehabcomparison, with CFQ, NOOP or DEADLINE, and in the same conditions,
99898bd37aSMauro Carvalho Chehabapplications experience high latencies, or even become unresponsive
100898bd37aSMauro Carvalho Chehabuntil the background workload terminates (also on SSDs).
101898bd37aSMauro Carvalho Chehab
102898bd37aSMauro Carvalho ChehabLow latency for soft real-time applications
103898bd37aSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
104898bd37aSMauro Carvalho ChehabAlso soft real-time applications, such as audio and video
105898bd37aSMauro Carvalho Chehabplayers/streamers, enjoy a low latency and a low drop rate, regardless
106898bd37aSMauro Carvalho Chehabof the background I/O workload. As a consequence, these applications
107898bd37aSMauro Carvalho Chehabdo not suffer from almost any glitch due to the background workload.
108898bd37aSMauro Carvalho Chehab
109898bd37aSMauro Carvalho ChehabHigher speed for code-development tasks
110898bd37aSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
111898bd37aSMauro Carvalho Chehab
112898bd37aSMauro Carvalho ChehabIf some additional workload happens to be executed in parallel, then
113898bd37aSMauro Carvalho ChehabBFQ executes the I/O-related components of typical code-development
114898bd37aSMauro Carvalho Chehabtasks (compilation, checkout, merge, ...) much more quickly than CFQ,
115898bd37aSMauro Carvalho ChehabNOOP or DEADLINE.
116898bd37aSMauro Carvalho Chehab
117898bd37aSMauro Carvalho ChehabHigh throughput
118898bd37aSMauro Carvalho Chehab^^^^^^^^^^^^^^^
119898bd37aSMauro Carvalho Chehab
120898bd37aSMauro Carvalho ChehabOn hard disks, BFQ achieves up to 30% higher throughput than CFQ, and
121898bd37aSMauro Carvalho Chehabup to 150% higher throughput than DEADLINE and NOOP, with all the
122898bd37aSMauro Carvalho Chehabsequential workloads considered in our tests. With random workloads,
123898bd37aSMauro Carvalho Chehaband with all the workloads on flash-based devices, BFQ achieves,
124898bd37aSMauro Carvalho Chehabinstead, about the same throughput as the other schedulers.
125898bd37aSMauro Carvalho Chehab
126898bd37aSMauro Carvalho ChehabStrong fairness, bandwidth and delay guarantees
127898bd37aSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
128898bd37aSMauro Carvalho Chehab
129898bd37aSMauro Carvalho ChehabBFQ distributes the device throughput, and not just the device time,
130898bd37aSMauro Carvalho Chehabamong I/O-bound applications in proportion their weights, with any
131898bd37aSMauro Carvalho Chehabworkload and regardless of the device parameters. From these bandwidth
132898bd37aSMauro Carvalho Chehabguarantees, it is possible to compute tight per-I/O-request delay
133898bd37aSMauro Carvalho Chehabguarantees by a simple formula. If not configured for strict service
134898bd37aSMauro Carvalho Chehabguarantees, BFQ switches to time-based resource sharing (only) for
135898bd37aSMauro Carvalho Chehabapplications that would otherwise cause a throughput loss.
136898bd37aSMauro Carvalho Chehab
137898bd37aSMauro Carvalho Chehab1-2 Server systems
138898bd37aSMauro Carvalho Chehab------------------
139898bd37aSMauro Carvalho Chehab
140898bd37aSMauro Carvalho ChehabMost benefits for server systems follow from the same service
141898bd37aSMauro Carvalho Chehabproperties as above. In particular, regardless of whether additional,
142898bd37aSMauro Carvalho Chehabpossibly heavy workloads are being served, BFQ guarantees:
143898bd37aSMauro Carvalho Chehab
144898bd37aSMauro Carvalho Chehab* audio and video-streaming with zero or very low jitter and drop
145898bd37aSMauro Carvalho Chehab  rate;
146898bd37aSMauro Carvalho Chehab
147898bd37aSMauro Carvalho Chehab* fast retrieval of WEB pages and embedded objects;
148898bd37aSMauro Carvalho Chehab
149898bd37aSMauro Carvalho Chehab* real-time recording of data in live-dumping applications (e.g.,
150898bd37aSMauro Carvalho Chehab  packet logging);
151898bd37aSMauro Carvalho Chehab
152898bd37aSMauro Carvalho Chehab* responsiveness in local and remote access to a server.
153898bd37aSMauro Carvalho Chehab
154898bd37aSMauro Carvalho Chehab
155898bd37aSMauro Carvalho Chehab2. How does BFQ work?
156898bd37aSMauro Carvalho Chehab=====================
157898bd37aSMauro Carvalho Chehab
158898bd37aSMauro Carvalho ChehabBFQ is a proportional-share I/O scheduler, whose general structure,
159898bd37aSMauro Carvalho Chehabplus a lot of code, are borrowed from CFQ.
160898bd37aSMauro Carvalho Chehab
161898bd37aSMauro Carvalho Chehab- Each process doing I/O on a device is associated with a weight and a
162898bd37aSMauro Carvalho Chehab  `(bfq_)queue`.
163898bd37aSMauro Carvalho Chehab
164898bd37aSMauro Carvalho Chehab- BFQ grants exclusive access to the device, for a while, to one queue
165898bd37aSMauro Carvalho Chehab  (process) at a time, and implements this service model by
166898bd37aSMauro Carvalho Chehab  associating every queue with a budget, measured in number of
167898bd37aSMauro Carvalho Chehab  sectors.
168898bd37aSMauro Carvalho Chehab
169898bd37aSMauro Carvalho Chehab  - After a queue is granted access to the device, the budget of the
170898bd37aSMauro Carvalho Chehab    queue is decremented, on each request dispatch, by the size of the
171898bd37aSMauro Carvalho Chehab    request.
172898bd37aSMauro Carvalho Chehab
173898bd37aSMauro Carvalho Chehab  - The in-service queue is expired, i.e., its service is suspended,
174898bd37aSMauro Carvalho Chehab    only if one of the following events occurs: 1) the queue finishes
175898bd37aSMauro Carvalho Chehab    its budget, 2) the queue empties, 3) a "budget timeout" fires.
176898bd37aSMauro Carvalho Chehab
177898bd37aSMauro Carvalho Chehab    - The budget timeout prevents processes doing random I/O from
178898bd37aSMauro Carvalho Chehab      holding the device for too long and dramatically reducing
179898bd37aSMauro Carvalho Chehab      throughput.
180898bd37aSMauro Carvalho Chehab
181898bd37aSMauro Carvalho Chehab    - Actually, as in CFQ, a queue associated with a process issuing
182898bd37aSMauro Carvalho Chehab      sync requests may not be expired immediately when it empties. In
183898bd37aSMauro Carvalho Chehab      contrast, BFQ may idle the device for a short time interval,
184898bd37aSMauro Carvalho Chehab      giving the process the chance to go on being served if it issues
185898bd37aSMauro Carvalho Chehab      a new request in time. Device idling typically boosts the
186898bd37aSMauro Carvalho Chehab      throughput on rotational devices and on non-queueing flash-based
187898bd37aSMauro Carvalho Chehab      devices, if processes do synchronous and sequential I/O. In
188898bd37aSMauro Carvalho Chehab      addition, under BFQ, device idling is also instrumental in
189898bd37aSMauro Carvalho Chehab      guaranteeing the desired throughput fraction to processes
190898bd37aSMauro Carvalho Chehab      issuing sync requests (see the description of the slice_idle
191898bd37aSMauro Carvalho Chehab      tunable in this document, or [1, 2], for more details).
192898bd37aSMauro Carvalho Chehab
193898bd37aSMauro Carvalho Chehab      - With respect to idling for service guarantees, if several
194898bd37aSMauro Carvalho Chehab	processes are competing for the device at the same time, but
195898bd37aSMauro Carvalho Chehab	all processes and groups have the same weight, then BFQ
196898bd37aSMauro Carvalho Chehab	guarantees the expected throughput distribution without ever
197898bd37aSMauro Carvalho Chehab	idling the device. Throughput is thus as high as possible in
198898bd37aSMauro Carvalho Chehab	this common scenario.
199898bd37aSMauro Carvalho Chehab
200898bd37aSMauro Carvalho Chehab     - On flash-based storage with internal queueing of commands
201898bd37aSMauro Carvalho Chehab       (typically NCQ), device idling happens to be always detrimental
202898bd37aSMauro Carvalho Chehab       for throughput. So, with these devices, BFQ performs idling
203898bd37aSMauro Carvalho Chehab       only when strictly needed for service guarantees, i.e., for
204898bd37aSMauro Carvalho Chehab       guaranteeing low latency or fairness. In these cases, overall
205898bd37aSMauro Carvalho Chehab       throughput may be sub-optimal. No solution currently exists to
206898bd37aSMauro Carvalho Chehab       provide both strong service guarantees and optimal throughput
207898bd37aSMauro Carvalho Chehab       on devices with internal queueing.
208898bd37aSMauro Carvalho Chehab
209898bd37aSMauro Carvalho Chehab  - If low-latency mode is enabled (default configuration), BFQ
210898bd37aSMauro Carvalho Chehab    executes some special heuristics to detect interactive and soft
211898bd37aSMauro Carvalho Chehab    real-time applications (e.g., video or audio players/streamers),
212898bd37aSMauro Carvalho Chehab    and to reduce their latency. The most important action taken to
213898bd37aSMauro Carvalho Chehab    achieve this goal is to give to the queues associated with these
214898bd37aSMauro Carvalho Chehab    applications more than their fair share of the device
215898bd37aSMauro Carvalho Chehab    throughput. For brevity, we call just "weight-raising" the whole
216898bd37aSMauro Carvalho Chehab    sets of actions taken by BFQ to privilege these queues. In
217898bd37aSMauro Carvalho Chehab    particular, BFQ provides a milder form of weight-raising for
218898bd37aSMauro Carvalho Chehab    interactive applications, and a stronger form for soft real-time
219898bd37aSMauro Carvalho Chehab    applications.
220898bd37aSMauro Carvalho Chehab
221898bd37aSMauro Carvalho Chehab  - BFQ automatically deactivates idling for queues born in a burst of
222898bd37aSMauro Carvalho Chehab    queue creations. In fact, these queues are usually associated with
223898bd37aSMauro Carvalho Chehab    the processes of applications and services that benefit mostly
224898bd37aSMauro Carvalho Chehab    from a high throughput. Examples are systemd during boot, or git
225898bd37aSMauro Carvalho Chehab    grep.
226898bd37aSMauro Carvalho Chehab
227898bd37aSMauro Carvalho Chehab  - As CFQ, BFQ merges queues performing interleaved I/O, i.e.,
228898bd37aSMauro Carvalho Chehab    performing random I/O that becomes mostly sequential if
229898bd37aSMauro Carvalho Chehab    merged. Differently from CFQ, BFQ achieves this goal with a more
230898bd37aSMauro Carvalho Chehab    reactive mechanism, called Early Queue Merge (EQM). EQM is so
231898bd37aSMauro Carvalho Chehab    responsive in detecting interleaved I/O (cooperating processes),
232898bd37aSMauro Carvalho Chehab    that it enables BFQ to achieve a high throughput, by queue
233898bd37aSMauro Carvalho Chehab    merging, even for queues for which CFQ needs a different
234898bd37aSMauro Carvalho Chehab    mechanism, preemption, to get a high throughput. As such EQM is a
235898bd37aSMauro Carvalho Chehab    unified mechanism to achieve a high throughput with interleaved
236898bd37aSMauro Carvalho Chehab    I/O.
237898bd37aSMauro Carvalho Chehab
238898bd37aSMauro Carvalho Chehab  - Queues are scheduled according to a variant of WF2Q+, named
239898bd37aSMauro Carvalho Chehab    B-WF2Q+, and implemented using an augmented rb-tree to preserve an
240898bd37aSMauro Carvalho Chehab    O(log N) overall complexity.  See [2] for more details. B-WF2Q+ is
241898bd37aSMauro Carvalho Chehab    also ready for hierarchical scheduling, details in Section 4.
242898bd37aSMauro Carvalho Chehab
243898bd37aSMauro Carvalho Chehab  - B-WF2Q+ guarantees a tight deviation with respect to an ideal,
244898bd37aSMauro Carvalho Chehab    perfectly fair, and smooth service. In particular, B-WF2Q+
245898bd37aSMauro Carvalho Chehab    guarantees that each queue receives a fraction of the device
246898bd37aSMauro Carvalho Chehab    throughput proportional to its weight, even if the throughput
247898bd37aSMauro Carvalho Chehab    fluctuates, and regardless of: the device parameters, the current
248898bd37aSMauro Carvalho Chehab    workload and the budgets assigned to the queue.
249898bd37aSMauro Carvalho Chehab
250898bd37aSMauro Carvalho Chehab  - The last, budget-independence, property (although probably
251898bd37aSMauro Carvalho Chehab    counterintuitive in the first place) is definitely beneficial, for
252898bd37aSMauro Carvalho Chehab    the following reasons:
253898bd37aSMauro Carvalho Chehab
254898bd37aSMauro Carvalho Chehab    - First, with any proportional-share scheduler, the maximum
255898bd37aSMauro Carvalho Chehab      deviation with respect to an ideal service is proportional to
256898bd37aSMauro Carvalho Chehab      the maximum budget (slice) assigned to queues. As a consequence,
257898bd37aSMauro Carvalho Chehab      BFQ can keep this deviation tight not only because of the
258898bd37aSMauro Carvalho Chehab      accurate service of B-WF2Q+, but also because BFQ *does not*
259898bd37aSMauro Carvalho Chehab      need to assign a larger budget to a queue to let the queue
260898bd37aSMauro Carvalho Chehab      receive a higher fraction of the device throughput.
261898bd37aSMauro Carvalho Chehab
262898bd37aSMauro Carvalho Chehab    - Second, BFQ is free to choose, for every process (queue), the
263898bd37aSMauro Carvalho Chehab      budget that best fits the needs of the process, or best
264898bd37aSMauro Carvalho Chehab      leverages the I/O pattern of the process. In particular, BFQ
265898bd37aSMauro Carvalho Chehab      updates queue budgets with a simple feedback-loop algorithm that
266898bd37aSMauro Carvalho Chehab      allows a high throughput to be achieved, while still providing
267898bd37aSMauro Carvalho Chehab      tight latency guarantees to time-sensitive applications. When
268898bd37aSMauro Carvalho Chehab      the in-service queue expires, this algorithm computes the next
269898bd37aSMauro Carvalho Chehab      budget of the queue so as to:
270898bd37aSMauro Carvalho Chehab
271898bd37aSMauro Carvalho Chehab      - Let large budgets be eventually assigned to the queues
272898bd37aSMauro Carvalho Chehab	associated with I/O-bound applications performing sequential
273898bd37aSMauro Carvalho Chehab	I/O: in fact, the longer these applications are served once
274898bd37aSMauro Carvalho Chehab	got access to the device, the higher the throughput is.
275898bd37aSMauro Carvalho Chehab
276898bd37aSMauro Carvalho Chehab      - Let small budgets be eventually assigned to the queues
277898bd37aSMauro Carvalho Chehab	associated with time-sensitive applications (which typically
278898bd37aSMauro Carvalho Chehab	perform sporadic and short I/O), because, the smaller the
279898bd37aSMauro Carvalho Chehab	budget assigned to a queue waiting for service is, the sooner
280898bd37aSMauro Carvalho Chehab	B-WF2Q+ will serve that queue (Subsec 3.3 in [2]).
281898bd37aSMauro Carvalho Chehab
282898bd37aSMauro Carvalho Chehab- If several processes are competing for the device at the same time,
283898bd37aSMauro Carvalho Chehab  but all processes and groups have the same weight, then BFQ
284898bd37aSMauro Carvalho Chehab  guarantees the expected throughput distribution without ever idling
285898bd37aSMauro Carvalho Chehab  the device. It uses preemption instead. Throughput is then much
286898bd37aSMauro Carvalho Chehab  higher in this common scenario.
287898bd37aSMauro Carvalho Chehab
288898bd37aSMauro Carvalho Chehab- ioprio classes are served in strict priority order, i.e.,
289898bd37aSMauro Carvalho Chehab  lower-priority queues are not served as long as there are
290898bd37aSMauro Carvalho Chehab  higher-priority queues.  Among queues in the same class, the
291898bd37aSMauro Carvalho Chehab  bandwidth is distributed in proportion to the weight of each
292898bd37aSMauro Carvalho Chehab  queue. A very thin extra bandwidth is however guaranteed to
293898bd37aSMauro Carvalho Chehab  the Idle class, to prevent it from starving.
294898bd37aSMauro Carvalho Chehab
295898bd37aSMauro Carvalho Chehab
296898bd37aSMauro Carvalho Chehab3. What are BFQ's tunables and how to properly configure BFQ?
297898bd37aSMauro Carvalho Chehab=============================================================
298898bd37aSMauro Carvalho Chehab
299898bd37aSMauro Carvalho ChehabMost BFQ tunables affect service guarantees (basically latency and
300898bd37aSMauro Carvalho Chehabfairness) and throughput. For full details on how to choose the
301898bd37aSMauro Carvalho Chehabdesired tradeoff between service guarantees and throughput, see the
302898bd37aSMauro Carvalho Chehabparameters slice_idle, strict_guarantees and low_latency. For details
303898bd37aSMauro Carvalho Chehabon how to maximise throughput, see slice_idle, timeout_sync and
304898bd37aSMauro Carvalho Chehabmax_budget. The other performance-related parameters have been
305898bd37aSMauro Carvalho Chehabinherited from, and have been preserved mostly for compatibility with
306898bd37aSMauro Carvalho ChehabCFQ. So far, no performance improvement has been reported after
307898bd37aSMauro Carvalho Chehabchanging the latter parameters in BFQ.
308898bd37aSMauro Carvalho Chehab
309898bd37aSMauro Carvalho ChehabIn particular, the tunables back_seek-max, back_seek_penalty,
310898bd37aSMauro Carvalho Chehabfifo_expire_async and fifo_expire_sync below are the same as in
311898bd37aSMauro Carvalho ChehabCFQ. Their description is just copied from that for CFQ. Some
312898bd37aSMauro Carvalho Chehabconsiderations in the description of slice_idle are copied from CFQ
313898bd37aSMauro Carvalho Chehabtoo.
314898bd37aSMauro Carvalho Chehab
315898bd37aSMauro Carvalho Chehabper-process ioprio and weight
316898bd37aSMauro Carvalho Chehab-----------------------------
317898bd37aSMauro Carvalho Chehab
318898bd37aSMauro Carvalho ChehabUnless the cgroups interface is used (see "4. BFQ group scheduling"),
319898bd37aSMauro Carvalho Chehabweights can be assigned to processes only indirectly, through I/O
320898bd37aSMauro Carvalho Chehabpriorities, and according to the relation:
321898bd37aSMauro Carvalho Chehabweight = (IOPRIO_BE_NR - ioprio) * 10.
322898bd37aSMauro Carvalho Chehab
323898bd37aSMauro Carvalho ChehabBeware that, if low-latency is set, then BFQ automatically raises the
324898bd37aSMauro Carvalho Chehabweight of the queues associated with interactive and soft real-time
325898bd37aSMauro Carvalho Chehabapplications. Unset this tunable if you need/want to control weights.
326898bd37aSMauro Carvalho Chehab
327898bd37aSMauro Carvalho Chehabslice_idle
328898bd37aSMauro Carvalho Chehab----------
329898bd37aSMauro Carvalho Chehab
330898bd37aSMauro Carvalho ChehabThis parameter specifies how long BFQ should idle for next I/O
331898bd37aSMauro Carvalho Chehabrequest, when certain sync BFQ queues become empty. By default
332898bd37aSMauro Carvalho Chehabslice_idle is a non-zero value. Idling has a double purpose: boosting
333898bd37aSMauro Carvalho Chehabthroughput and making sure that the desired throughput distribution is
334898bd37aSMauro Carvalho Chehabrespected (see the description of how BFQ works, and, if needed, the
335898bd37aSMauro Carvalho Chehabpapers referred there).
336898bd37aSMauro Carvalho Chehab
337898bd37aSMauro Carvalho ChehabAs for throughput, idling can be very helpful on highly seeky media
338898bd37aSMauro Carvalho Chehablike single spindle SATA/SAS disks where we can cut down on overall
339898bd37aSMauro Carvalho Chehabnumber of seeks and see improved throughput.
340898bd37aSMauro Carvalho Chehab
341898bd37aSMauro Carvalho ChehabSetting slice_idle to 0 will remove all the idling on queues and one
342898bd37aSMauro Carvalho Chehabshould see an overall improved throughput on faster storage devices
343898bd37aSMauro Carvalho Chehablike multiple SATA/SAS disks in hardware RAID configuration, as well
344898bd37aSMauro Carvalho Chehabas flash-based storage with internal command queueing (and
345898bd37aSMauro Carvalho Chehabparallelism).
346898bd37aSMauro Carvalho Chehab
347898bd37aSMauro Carvalho ChehabSo depending on storage and workload, it might be useful to set
348898bd37aSMauro Carvalho Chehabslice_idle=0.  In general for SATA/SAS disks and software RAID of
349898bd37aSMauro Carvalho ChehabSATA/SAS disks keeping slice_idle enabled should be useful. For any
350898bd37aSMauro Carvalho Chehabconfigurations where there are multiple spindles behind single LUN
351898bd37aSMauro Carvalho Chehab(Host based hardware RAID controller or for storage arrays), or with
352898bd37aSMauro Carvalho Chehabflash-based fast storage, setting slice_idle=0 might end up in better
353898bd37aSMauro Carvalho Chehabthroughput and acceptable latencies.
354898bd37aSMauro Carvalho Chehab
355898bd37aSMauro Carvalho ChehabIdling is however necessary to have service guarantees enforced in
356898bd37aSMauro Carvalho Chehabcase of differentiated weights or differentiated I/O-request lengths.
357898bd37aSMauro Carvalho ChehabTo see why, suppose that a given BFQ queue A must get several I/O
358898bd37aSMauro Carvalho Chehabrequests served for each request served for another queue B. Idling
359898bd37aSMauro Carvalho Chehabensures that, if A makes a new I/O request slightly after becoming
360898bd37aSMauro Carvalho Chehabempty, then no request of B is dispatched in the middle, and thus A
361898bd37aSMauro Carvalho Chehabdoes not lose the possibility to get more than one request dispatched
362898bd37aSMauro Carvalho Chehabbefore the next request of B is dispatched. Note that idling
363898bd37aSMauro Carvalho Chehabguarantees the desired differentiated treatment of queues only in
364898bd37aSMauro Carvalho Chehabterms of I/O-request dispatches. To guarantee that the actual service
365898bd37aSMauro Carvalho Chehaborder then corresponds to the dispatch order, the strict_guarantees
366898bd37aSMauro Carvalho Chehabtunable must be set too.
367898bd37aSMauro Carvalho Chehab
368898bd37aSMauro Carvalho ChehabThere is an important flipside for idling: apart from the above cases
369898bd37aSMauro Carvalho Chehabwhere it is beneficial also for throughput, idling can severely impact
370898bd37aSMauro Carvalho Chehabthroughput. One important case is random workload. Because of this
371898bd37aSMauro Carvalho Chehabissue, BFQ tends to avoid idling as much as possible, when it is not
372898bd37aSMauro Carvalho Chehabbeneficial also for throughput (as detailed in Section 2). As a
373898bd37aSMauro Carvalho Chehabconsequence of this behavior, and of further issues described for the
374898bd37aSMauro Carvalho Chehabstrict_guarantees tunable, short-term service guarantees may be
375898bd37aSMauro Carvalho Chehaboccasionally violated. And, in some cases, these guarantees may be
376898bd37aSMauro Carvalho Chehabmore important than guaranteeing maximum throughput. For example, in
377898bd37aSMauro Carvalho Chehabvideo playing/streaming, a very low drop rate may be more important
378898bd37aSMauro Carvalho Chehabthan maximum throughput. In these cases, consider setting the
379898bd37aSMauro Carvalho Chehabstrict_guarantees parameter.
380898bd37aSMauro Carvalho Chehab
381898bd37aSMauro Carvalho Chehabslice_idle_us
382898bd37aSMauro Carvalho Chehab-------------
383898bd37aSMauro Carvalho Chehab
384898bd37aSMauro Carvalho ChehabControls the same tuning parameter as slice_idle, but in microseconds.
385898bd37aSMauro Carvalho ChehabEither tunable can be used to set idling behavior.  Afterwards, the
386898bd37aSMauro Carvalho Chehabother tunable will reflect the newly set value in sysfs.
387898bd37aSMauro Carvalho Chehab
388898bd37aSMauro Carvalho Chehabstrict_guarantees
389898bd37aSMauro Carvalho Chehab-----------------
390898bd37aSMauro Carvalho Chehab
391898bd37aSMauro Carvalho ChehabIf this parameter is set (default: unset), then BFQ
392898bd37aSMauro Carvalho Chehab
393898bd37aSMauro Carvalho Chehab- always performs idling when the in-service queue becomes empty;
394898bd37aSMauro Carvalho Chehab
395898bd37aSMauro Carvalho Chehab- forces the device to serve one I/O request at a time, by dispatching a
396898bd37aSMauro Carvalho Chehab  new request only if there is no outstanding request.
397898bd37aSMauro Carvalho Chehab
398898bd37aSMauro Carvalho ChehabIn the presence of differentiated weights or I/O-request sizes, both
399898bd37aSMauro Carvalho Chehabthe above conditions are needed to guarantee that every BFQ queue
400898bd37aSMauro Carvalho Chehabreceives its allotted share of the bandwidth. The first condition is
401898bd37aSMauro Carvalho Chehabneeded for the reasons explained in the description of the slice_idle
402898bd37aSMauro Carvalho Chehabtunable.  The second condition is needed because all modern storage
403898bd37aSMauro Carvalho Chehabdevices reorder internally-queued requests, which may trivially break
404898bd37aSMauro Carvalho Chehabthe service guarantees enforced by the I/O scheduler.
405898bd37aSMauro Carvalho Chehab
406898bd37aSMauro Carvalho ChehabSetting strict_guarantees may evidently affect throughput.
407898bd37aSMauro Carvalho Chehab
408898bd37aSMauro Carvalho Chehabback_seek_max
409898bd37aSMauro Carvalho Chehab-------------
410898bd37aSMauro Carvalho Chehab
411898bd37aSMauro Carvalho ChehabThis specifies, given in Kbytes, the maximum "distance" for backward seeking.
412898bd37aSMauro Carvalho ChehabThe distance is the amount of space from the current head location to the
413898bd37aSMauro Carvalho Chehabsectors that are backward in terms of distance.
414898bd37aSMauro Carvalho Chehab
415898bd37aSMauro Carvalho ChehabThis parameter allows the scheduler to anticipate requests in the "backward"
416898bd37aSMauro Carvalho Chehabdirection and consider them as being the "next" if they are within this
417898bd37aSMauro Carvalho Chehabdistance from the current head location.
418898bd37aSMauro Carvalho Chehab
419898bd37aSMauro Carvalho Chehabback_seek_penalty
420898bd37aSMauro Carvalho Chehab-----------------
421898bd37aSMauro Carvalho Chehab
422898bd37aSMauro Carvalho ChehabThis parameter is used to compute the cost of backward seeking. If the
423898bd37aSMauro Carvalho Chehabbackward distance of request is just 1/back_seek_penalty from a "front"
424898bd37aSMauro Carvalho Chehabrequest, then the seeking cost of two requests is considered equivalent.
425898bd37aSMauro Carvalho Chehab
426898bd37aSMauro Carvalho ChehabSo scheduler will not bias toward one or the other request (otherwise scheduler
427898bd37aSMauro Carvalho Chehabwill bias toward front request). Default value of back_seek_penalty is 2.
428898bd37aSMauro Carvalho Chehab
429898bd37aSMauro Carvalho Chehabfifo_expire_async
430898bd37aSMauro Carvalho Chehab-----------------
431898bd37aSMauro Carvalho Chehab
432898bd37aSMauro Carvalho ChehabThis parameter is used to set the timeout of asynchronous requests. Default
4334168a8d2SJoseph Qivalue of this is 250ms.
434898bd37aSMauro Carvalho Chehab
435898bd37aSMauro Carvalho Chehabfifo_expire_sync
436898bd37aSMauro Carvalho Chehab----------------
437898bd37aSMauro Carvalho Chehab
438898bd37aSMauro Carvalho ChehabThis parameter is used to set the timeout of synchronous requests. Default
4394168a8d2SJoseph Qivalue of this is 125ms. In case to favor synchronous requests over asynchronous
440898bd37aSMauro Carvalho Chehabone, this value should be decreased relative to fifo_expire_async.
441898bd37aSMauro Carvalho Chehab
442898bd37aSMauro Carvalho Chehablow_latency
443898bd37aSMauro Carvalho Chehab-----------
444898bd37aSMauro Carvalho Chehab
445898bd37aSMauro Carvalho ChehabThis parameter is used to enable/disable BFQ's low latency mode. By
446898bd37aSMauro Carvalho Chehabdefault, low latency mode is enabled. If enabled, interactive and soft
447898bd37aSMauro Carvalho Chehabreal-time applications are privileged and experience a lower latency,
448898bd37aSMauro Carvalho Chehabas explained in more detail in the description of how BFQ works.
449898bd37aSMauro Carvalho Chehab
450898bd37aSMauro Carvalho ChehabDISABLE this mode if you need full control on bandwidth
451898bd37aSMauro Carvalho Chehabdistribution. In fact, if it is enabled, then BFQ automatically
452898bd37aSMauro Carvalho Chehabincreases the bandwidth share of privileged applications, as the main
453898bd37aSMauro Carvalho Chehabmeans to guarantee a lower latency to them.
454898bd37aSMauro Carvalho Chehab
455898bd37aSMauro Carvalho ChehabIn addition, as already highlighted at the beginning of this document,
456898bd37aSMauro Carvalho ChehabDISABLE this mode if your only goal is to achieve a high throughput.
457898bd37aSMauro Carvalho ChehabIn fact, privileging the I/O of some application over the rest may
458898bd37aSMauro Carvalho Chehabentail a lower throughput. To achieve the highest-possible throughput
459898bd37aSMauro Carvalho Chehabon a non-rotational device, setting slice_idle to 0 may be needed too
460898bd37aSMauro Carvalho Chehab(at the cost of giving up any strong guarantee on fairness and low
461898bd37aSMauro Carvalho Chehablatency).
462898bd37aSMauro Carvalho Chehab
463898bd37aSMauro Carvalho Chehabtimeout_sync
464898bd37aSMauro Carvalho Chehab------------
465898bd37aSMauro Carvalho Chehab
466898bd37aSMauro Carvalho ChehabMaximum amount of device time that can be given to a task (queue) once
467898bd37aSMauro Carvalho Chehabit has been selected for service. On devices with costly seeks,
468898bd37aSMauro Carvalho Chehabincreasing this time usually increases maximum throughput. On the
469898bd37aSMauro Carvalho Chehabopposite end, increasing this time coarsens the granularity of the
470898bd37aSMauro Carvalho Chehabshort-term bandwidth and latency guarantees, especially if the
471898bd37aSMauro Carvalho Chehabfollowing parameter is set to zero.
472898bd37aSMauro Carvalho Chehab
473898bd37aSMauro Carvalho Chehabmax_budget
474898bd37aSMauro Carvalho Chehab----------
475898bd37aSMauro Carvalho Chehab
476898bd37aSMauro Carvalho ChehabMaximum amount of service, measured in sectors, that can be provided
477898bd37aSMauro Carvalho Chehabto a BFQ queue once it is set in service (of course within the limits
478898bd37aSMauro Carvalho Chehabof the above timeout). According to what said in the description of
479898bd37aSMauro Carvalho Chehabthe algorithm, larger values increase the throughput in proportion to
480898bd37aSMauro Carvalho Chehabthe percentage of sequential I/O requests issued. The price of larger
481898bd37aSMauro Carvalho Chehabvalues is that they coarsen the granularity of short-term bandwidth
482898bd37aSMauro Carvalho Chehaband latency guarantees.
483898bd37aSMauro Carvalho Chehab
484898bd37aSMauro Carvalho ChehabThe default value is 0, which enables auto-tuning: BFQ sets max_budget
485898bd37aSMauro Carvalho Chehabto the maximum number of sectors that can be served during
486898bd37aSMauro Carvalho Chehabtimeout_sync, according to the estimated peak rate.
487898bd37aSMauro Carvalho Chehab
488898bd37aSMauro Carvalho ChehabFor specific devices, some users have occasionally reported to have
489898bd37aSMauro Carvalho Chehabreached a higher throughput by setting max_budget explicitly, i.e., by
490898bd37aSMauro Carvalho Chehabsetting max_budget to a higher value than 0. In particular, they have
491898bd37aSMauro Carvalho Chehabset max_budget to higher values than those to which BFQ would have set
492898bd37aSMauro Carvalho Chehabit with auto-tuning. An alternative way to achieve this goal is to
493898bd37aSMauro Carvalho Chehabjust increase the value of timeout_sync, leaving max_budget equal to 0.
494898bd37aSMauro Carvalho Chehab
495898bd37aSMauro Carvalho Chehab4. Group scheduling with BFQ
496898bd37aSMauro Carvalho Chehab============================
497898bd37aSMauro Carvalho Chehab
498898bd37aSMauro Carvalho ChehabBFQ supports both cgroups-v1 and cgroups-v2 io controllers, namely
499898bd37aSMauro Carvalho Chehabblkio and io. In particular, BFQ supports weight-based proportional
500898bd37aSMauro Carvalho Chehabshare. To activate cgroups support, set BFQ_GROUP_IOSCHED.
501898bd37aSMauro Carvalho Chehab
502898bd37aSMauro Carvalho Chehab4-1 Service guarantees provided
503898bd37aSMauro Carvalho Chehab-------------------------------
504898bd37aSMauro Carvalho Chehab
505898bd37aSMauro Carvalho ChehabWith BFQ, proportional share means true proportional share of the
506898bd37aSMauro Carvalho Chehabdevice bandwidth, according to group weights. For example, a group
507898bd37aSMauro Carvalho Chehabwith weight 200 gets twice the bandwidth, and not just twice the time,
508898bd37aSMauro Carvalho Chehabof a group with weight 100.
509898bd37aSMauro Carvalho Chehab
510898bd37aSMauro Carvalho ChehabBFQ supports hierarchies (group trees) of any depth. Bandwidth is
511898bd37aSMauro Carvalho Chehabdistributed among groups and processes in the expected way: for each
512898bd37aSMauro Carvalho Chehabgroup, the children of the group share the whole bandwidth of the
513898bd37aSMauro Carvalho Chehabgroup in proportion to their weights. In particular, this implies
514898bd37aSMauro Carvalho Chehabthat, for each leaf group, every process of the group receives the
515898bd37aSMauro Carvalho Chehabsame share of the whole group bandwidth, unless the ioprio of the
516898bd37aSMauro Carvalho Chehabprocess is modified.
517898bd37aSMauro Carvalho Chehab
518898bd37aSMauro Carvalho ChehabThe resource-sharing guarantee for a group may partially or totally
519898bd37aSMauro Carvalho Chehabswitch from bandwidth to time, if providing bandwidth guarantees to
520898bd37aSMauro Carvalho Chehabthe group lowers the throughput too much. This switch occurs on a
521898bd37aSMauro Carvalho Chehabper-process basis: if a process of a leaf group causes throughput loss
522898bd37aSMauro Carvalho Chehabif served in such a way to receive its share of the bandwidth, then
523898bd37aSMauro Carvalho ChehabBFQ switches back to just time-based proportional share for that
524898bd37aSMauro Carvalho Chehabprocess.
525898bd37aSMauro Carvalho Chehab
526898bd37aSMauro Carvalho Chehab4-2 Interface
527898bd37aSMauro Carvalho Chehab-------------
528898bd37aSMauro Carvalho Chehab
529898bd37aSMauro Carvalho ChehabTo get proportional sharing of bandwidth with BFQ for a given device,
530898bd37aSMauro Carvalho ChehabBFQ must of course be the active scheduler for that device.
531898bd37aSMauro Carvalho Chehab
532898bd37aSMauro Carvalho ChehabWithin each group directory, the names of the files associated with
533898bd37aSMauro Carvalho ChehabBFQ-specific cgroup parameters and stats begin with the "bfq."
534898bd37aSMauro Carvalho Chehabprefix. So, with cgroups-v1 or cgroups-v2, the full prefix for
535898bd37aSMauro Carvalho ChehabBFQ-specific files is "blkio.bfq." or "io.bfq." For example, the group
536898bd37aSMauro Carvalho Chehabparameter to set the weight of a group with BFQ is blkio.bfq.weight
537898bd37aSMauro Carvalho Chehabor io.bfq.weight.
538898bd37aSMauro Carvalho Chehab
539898bd37aSMauro Carvalho ChehabAs for cgroups-v1 (blkio controller), the exact set of stat files
540898bd37aSMauro Carvalho Chehabcreated, and kept up-to-date by bfq, depends on whether
541898bd37aSMauro Carvalho ChehabCONFIG_BFQ_CGROUP_DEBUG is set. If it is set, then bfq creates all
542898bd37aSMauro Carvalho Chehabthe stat files documented in
543da82c92fSMauro Carvalho ChehabDocumentation/admin-guide/cgroup-v1/blkio-controller.rst. If, instead,
544898bd37aSMauro Carvalho ChehabCONFIG_BFQ_CGROUP_DEBUG is not set, then bfq creates only the files::
545898bd37aSMauro Carvalho Chehab
546898bd37aSMauro Carvalho Chehab  blkio.bfq.io_service_bytes
547898bd37aSMauro Carvalho Chehab  blkio.bfq.io_service_bytes_recursive
548898bd37aSMauro Carvalho Chehab  blkio.bfq.io_serviced
549898bd37aSMauro Carvalho Chehab  blkio.bfq.io_serviced_recursive
550898bd37aSMauro Carvalho Chehab
551898bd37aSMauro Carvalho ChehabThe value of CONFIG_BFQ_CGROUP_DEBUG greatly influences the maximum
552898bd37aSMauro Carvalho Chehabthroughput sustainable with bfq, because updating the blkio.bfq.*
553898bd37aSMauro Carvalho Chehabstats is rather costly, especially for some of the stats enabled by
554898bd37aSMauro Carvalho ChehabCONFIG_BFQ_CGROUP_DEBUG.
555898bd37aSMauro Carvalho Chehab
556*fda0b5baSKir KolyshkinParameters
557*fda0b5baSKir Kolyshkin----------
558898bd37aSMauro Carvalho Chehab
559*fda0b5baSKir KolyshkinFor each group, the following parameters can be set:
560898bd37aSMauro Carvalho Chehab
561*fda0b5baSKir Kolyshkin  weight
562*fda0b5baSKir Kolyshkin        This specifies the default weight for the cgroup inside its parent.
563*fda0b5baSKir Kolyshkin        Available values: 1..1000 (default: 100).
564*fda0b5baSKir Kolyshkin
565*fda0b5baSKir Kolyshkin        For cgroup v1, it is set by writing the value to `blkio.bfq.weight`.
566*fda0b5baSKir Kolyshkin
567*fda0b5baSKir Kolyshkin        For cgroup v2, it is set by writing the value to `io.bfq.weight`.
568*fda0b5baSKir Kolyshkin        (with an optional prefix of `default` and a space).
569*fda0b5baSKir Kolyshkin
570*fda0b5baSKir Kolyshkin        The linear mapping between ioprio and weights, described at the beginning
571898bd37aSMauro Carvalho Chehab        of the tunable section, is still valid, but all weights higher than
572898bd37aSMauro Carvalho Chehab        IOPRIO_BE_NR*10 are mapped to ioprio 0.
573898bd37aSMauro Carvalho Chehab
574898bd37aSMauro Carvalho Chehab        Recall that, if low-latency is set, then BFQ automatically raises the
575898bd37aSMauro Carvalho Chehab        weight of the queues associated with interactive and soft real-time
576898bd37aSMauro Carvalho Chehab        applications. Unset this tunable if you need/want to control weights.
577898bd37aSMauro Carvalho Chehab
578*fda0b5baSKir Kolyshkin  weight_device
579*fda0b5baSKir Kolyshkin        This specifies a per-device weight for the cgroup. The syntax is
580*fda0b5baSKir Kolyshkin        `minor:major weight`. A weight of `0` may be used to reset to the default
581*fda0b5baSKir Kolyshkin        weight.
582*fda0b5baSKir Kolyshkin
583*fda0b5baSKir Kolyshkin        For cgroup v1, it is set by writing the value to `blkio.bfq.weight_device`.
584*fda0b5baSKir Kolyshkin
585*fda0b5baSKir Kolyshkin        For cgroup v2, the file name is `io.bfq.weight`.
586*fda0b5baSKir Kolyshkin
587898bd37aSMauro Carvalho Chehab
588898bd37aSMauro Carvalho Chehab[1]
589898bd37aSMauro Carvalho Chehab    P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O
590898bd37aSMauro Carvalho Chehab    Scheduler", Proceedings of the First Workshop on Mobile System
591898bd37aSMauro Carvalho Chehab    Technologies (MST-2015), May 2015.
592898bd37aSMauro Carvalho Chehab
593898bd37aSMauro Carvalho Chehab    http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf
594898bd37aSMauro Carvalho Chehab
595898bd37aSMauro Carvalho Chehab[2]
596898bd37aSMauro Carvalho Chehab    P. Valente and M. Andreolini, "Improving Application
597898bd37aSMauro Carvalho Chehab    Responsiveness with the BFQ Disk I/O Scheduler", Proceedings of
598898bd37aSMauro Carvalho Chehab    the 5th Annual International Systems and Storage Conference
599898bd37aSMauro Carvalho Chehab    (SYSTOR '12), June 2012.
600898bd37aSMauro Carvalho Chehab
601898bd37aSMauro Carvalho Chehab    Slightly extended version:
602898bd37aSMauro Carvalho Chehab
603898bd37aSMauro Carvalho Chehab    http://algogroup.unimore.it/people/paolo/disk_sched/bfq-v1-suite-results.pdf
604898bd37aSMauro Carvalho Chehab
605898bd37aSMauro Carvalho Chehab[3]
606898bd37aSMauro Carvalho Chehab   https://github.com/Algodev-github/S
607