1===========================================
2Fault injection capabilities infrastructure
3===========================================
4
5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
6
7
8Available fault injection capabilities
9--------------------------------------
10
11- failslab
12
13  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
14
15- fail_page_alloc
16
17  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
18
19- fail_usercopy
20
21  injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
22
23- fail_futex
24
25  injects futex deadlock and uaddr fault errors.
26
27- fail_make_request
28
29  injects disk IO errors on devices permitted by setting
30  /sys/block/<device>/make-it-fail or
31  /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
32
33- fail_mmc_request
34
35  injects MMC data errors on devices permitted by setting
36  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
37
38- fail_function
39
40  injects error return on specific functions, which are marked by
41  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
42  under /sys/kernel/debug/fail_function. No boot option supported.
43
44- NVMe fault injection
45
46  inject NVMe status code and retry flag on devices permitted by setting
47  debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
48  status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
49  retry flag can be set via the debugfs.
50
51
52Configure fault-injection capabilities behavior
53-----------------------------------------------
54
55debugfs entries
56^^^^^^^^^^^^^^^
57
58fault-inject-debugfs kernel module provides some debugfs entries for runtime
59configuration of fault-injection capabilities.
60
61- /sys/kernel/debug/fail*/probability:
62
63	likelihood of failure injection, in percent.
64
65	Format: <percent>
66
67	Note that one-failure-per-hundred is a very high error rate
68	for some testcases.  Consider setting probability=100 and configure
69	/sys/kernel/debug/fail*/interval for such testcases.
70
71- /sys/kernel/debug/fail*/interval:
72
73	specifies the interval between failures, for calls to
74	should_fail() that pass all the other tests.
75
76	Note that if you enable this, by setting interval>1, you will
77	probably want to set probability=100.
78
79- /sys/kernel/debug/fail*/times:
80
81	specifies how many times failures may happen at most.
82	A value of -1 means "no limit".
83
84- /sys/kernel/debug/fail*/space:
85
86	specifies an initial resource "budget", decremented by "size"
87	on each call to should_fail(,size).  Failure injection is
88	suppressed until "space" reaches zero.
89
90- /sys/kernel/debug/fail*/verbose
91
92	Format: { 0 | 1 | 2 }
93
94	specifies the verbosity of the messages when failure is
95	injected.  '0' means no messages; '1' will print only a single
96	log line per failure; '2' will print a call trace too -- useful
97	to debug the problems revealed by fault injection.
98
99- /sys/kernel/debug/fail*/task-filter:
100
101	Format: { 'Y' | 'N' }
102
103	A value of 'N' disables filtering by process (default).
104	Any positive value limits failures to only processes indicated by
105	/proc/<pid>/make-it-fail==1.
106
107- /sys/kernel/debug/fail*/require-start,
108  /sys/kernel/debug/fail*/require-end,
109  /sys/kernel/debug/fail*/reject-start,
110  /sys/kernel/debug/fail*/reject-end:
111
112	specifies the range of virtual addresses tested during
113	stacktrace walking.  Failure is injected only if some caller
114	in the walked stacktrace lies within the required range, and
115	none lies within the rejected range.
116	Default required range is [0,ULONG_MAX) (whole of virtual address space).
117	Default rejected range is [0,0).
118
119- /sys/kernel/debug/fail*/stacktrace-depth:
120
121	specifies the maximum stacktrace depth walked during search
122	for a caller within [require-start,require-end) OR
123	[reject-start,reject-end).
124
125- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
126
127	Format: { 'Y' | 'N' }
128
129	default is 'N', setting it to 'Y' won't inject failures into
130	highmem/user allocations.
131
132- /sys/kernel/debug/failslab/ignore-gfp-wait:
133- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
134
135	Format: { 'Y' | 'N' }
136
137	default is 'N', setting it to 'Y' will inject failures
138	only into non-sleep allocations (GFP_ATOMIC allocations).
139
140- /sys/kernel/debug/fail_page_alloc/min-order:
141
142	specifies the minimum page allocation order to be injected
143	failures.
144
145- /sys/kernel/debug/fail_futex/ignore-private:
146
147	Format: { 'Y' | 'N' }
148
149	default is 'N', setting it to 'Y' will disable failure injections
150	when dealing with private (address space) futexes.
151
152- /sys/kernel/debug/fail_function/inject:
153
154	Format: { 'function-name' | '!function-name' | '' }
155
156	specifies the target function of error injection by name.
157	If the function name leads '!' prefix, given function is
158	removed from injection list. If nothing specified ('')
159	injection list is cleared.
160
161- /sys/kernel/debug/fail_function/injectable:
162
163	(read only) shows error injectable functions and what type of
164	error values can be specified. The error type will be one of
165	below;
166	- NULL:	retval must be 0.
167	- ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
168	- ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
169
170- /sys/kernel/debug/fail_function/<functiuon-name>/retval:
171
172	specifies the "error" return value to inject to the given
173	function for given function. This will be created when
174	user specifies new injection entry.
175
176Boot option
177^^^^^^^^^^^
178
179In order to inject faults while debugfs is not available (early boot time),
180use the boot option::
181
182	failslab=
183	fail_page_alloc=
184	fail_usercopy=
185	fail_make_request=
186	fail_futex=
187	mmc_core.fail_request=<interval>,<probability>,<space>,<times>
188
189proc entries
190^^^^^^^^^^^^
191
192- /proc/<pid>/fail-nth,
193  /proc/self/task/<tid>/fail-nth:
194
195	Write to this file of integer N makes N-th call in the task fail.
196	Read from this file returns a integer value. A value of '0' indicates
197	that the fault setup with a previous write to this file was injected.
198	A positive integer N indicates that the fault wasn't yet injected.
199	Note that this file enables all types of faults (slab, futex, etc).
200	This setting takes precedence over all other generic debugfs settings
201	like probability, interval, times, etc. But per-capability settings
202	(e.g. fail_futex/ignore-private) take precedence over it.
203
204	This feature is intended for systematic testing of faults in a single
205	system call. See an example below.
206
207How to add new fault injection capability
208-----------------------------------------
209
210- #include <linux/fault-inject.h>
211
212- define the fault attributes
213
214  DECLARE_FAULT_ATTR(name);
215
216  Please see the definition of struct fault_attr in fault-inject.h
217  for details.
218
219- provide a way to configure fault attributes
220
221- boot option
222
223  If you need to enable the fault injection capability from boot time, you can
224  provide boot option to configure it. There is a helper function for it:
225
226	setup_fault_attr(attr, str);
227
228- debugfs entries
229
230  failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
231  Helper functions:
232
233	fault_create_debugfs_attr(name, parent, attr);
234
235- module parameters
236
237  If the scope of the fault injection capability is limited to a
238  single kernel module, it is better to provide module parameters to
239  configure the fault attributes.
240
241- add a hook to insert failures
242
243  Upon should_fail() returning true, client code should inject a failure:
244
245	should_fail(attr, size);
246
247Application Examples
248--------------------
249
250- Inject slab allocation failures into module init/exit code::
251
252    #!/bin/bash
253
254    FAILTYPE=failslab
255    echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
256    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
257    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
258    echo -1 > /sys/kernel/debug/$FAILTYPE/times
259    echo 0 > /sys/kernel/debug/$FAILTYPE/space
260    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
261    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
262
263    faulty_system()
264    {
265	bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
266    }
267
268    if [ $# -eq 0 ]
269    then
270	echo "Usage: $0 modulename [ modulename ... ]"
271	exit 1
272    fi
273
274    for m in $*
275    do
276	echo inserting $m...
277	faulty_system modprobe $m
278
279	echo removing $m...
280	faulty_system modprobe -r $m
281    done
282
283------------------------------------------------------------------------------
284
285- Inject page allocation failures only for a specific module::
286
287    #!/bin/bash
288
289    FAILTYPE=fail_page_alloc
290    module=$1
291
292    if [ -z $module ]
293    then
294	echo "Usage: $0 <modulename>"
295	exit 1
296    fi
297
298    modprobe $module
299
300    if [ ! -d /sys/module/$module/sections ]
301    then
302	echo Module $module is not loaded
303	exit 1
304    fi
305
306    cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
307    cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
308
309    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
310    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
311    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
312    echo -1 > /sys/kernel/debug/$FAILTYPE/times
313    echo 0 > /sys/kernel/debug/$FAILTYPE/space
314    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
315    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
316    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
317    echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
318
319    trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
320
321    echo "Injecting errors into the module $module... (interrupt to stop)"
322    sleep 1000000
323
324------------------------------------------------------------------------------
325
326- Inject open_ctree error while btrfs mount::
327
328    #!/bin/bash
329
330    rm -f testfile.img
331    dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
332    DEVICE=$(losetup --show -f testfile.img)
333    mkfs.btrfs -f $DEVICE
334    mkdir -p tmpmnt
335
336    FAILTYPE=fail_function
337    FAILFUNC=open_ctree
338    echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
339    echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
340    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
341    echo 100 > /sys/kernel/debug/$FAILTYPE/probability
342    echo 0 > /sys/kernel/debug/$FAILTYPE/interval
343    echo -1 > /sys/kernel/debug/$FAILTYPE/times
344    echo 0 > /sys/kernel/debug/$FAILTYPE/space
345    echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
346
347    mount -t btrfs $DEVICE tmpmnt
348    if [ $? -ne 0 ]
349    then
350	echo "SUCCESS!"
351    else
352	echo "FAILED!"
353	umount tmpmnt
354    fi
355
356    echo > /sys/kernel/debug/$FAILTYPE/inject
357
358    rmdir tmpmnt
359    losetup -d $DEVICE
360    rm testfile.img
361
362
363Tool to run command with failslab or fail_page_alloc
364----------------------------------------------------
365In order to make it easier to accomplish the tasks mentioned above, we can use
366tools/testing/fault-injection/failcmd.sh.  Please run a command
367"./tools/testing/fault-injection/failcmd.sh --help" for more information and
368see the following examples.
369
370Examples:
371
372Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
373allocation failure::
374
375	# ./tools/testing/fault-injection/failcmd.sh \
376		-- make -C tools/testing/selftests/ run_tests
377
378Same as above except to specify 100 times failures at most instead of one time
379at most by default::
380
381	# ./tools/testing/fault-injection/failcmd.sh --times=100 \
382		-- make -C tools/testing/selftests/ run_tests
383
384Same as above except to inject page allocation failure instead of slab
385allocation failure::
386
387	# env FAILCMD_TYPE=fail_page_alloc \
388		./tools/testing/fault-injection/failcmd.sh --times=100 \
389		-- make -C tools/testing/selftests/ run_tests
390
391Systematic faults using fail-nth
392---------------------------------
393
394The following code systematically faults 0-th, 1-st, 2-nd and so on
395capabilities in the socketpair() system call::
396
397  #include <sys/types.h>
398  #include <sys/stat.h>
399  #include <sys/socket.h>
400  #include <sys/syscall.h>
401  #include <fcntl.h>
402  #include <unistd.h>
403  #include <string.h>
404  #include <stdlib.h>
405  #include <stdio.h>
406  #include <errno.h>
407
408  int main()
409  {
410	int i, err, res, fail_nth, fds[2];
411	char buf[128];
412
413	system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
414	sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
415	fail_nth = open(buf, O_RDWR);
416	for (i = 1;; i++) {
417		sprintf(buf, "%d", i);
418		write(fail_nth, buf, strlen(buf));
419		res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
420		err = errno;
421		pread(fail_nth, buf, sizeof(buf), 0);
422		if (res == 0) {
423			close(fds[0]);
424			close(fds[1]);
425		}
426		printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
427			res, err);
428		if (atoi(buf))
429			break;
430	}
431	return 0;
432  }
433
434An example output::
435
436	1-th fault Y: res=-1/23
437	2-th fault Y: res=-1/23
438	3-th fault Y: res=-1/12
439	4-th fault Y: res=-1/12
440	5-th fault Y: res=-1/23
441	6-th fault Y: res=-1/23
442	7-th fault Y: res=-1/23
443	8-th fault Y: res=-1/12
444	9-th fault Y: res=-1/12
445	10-th fault Y: res=-1/12
446	11-th fault Y: res=-1/12
447	12-th fault Y: res=-1/12
448	13-th fault Y: res=-1/12
449	14-th fault Y: res=-1/12
450	15-th fault Y: res=-1/12
451	16-th fault N: res=0/12
452