1===========================================
2Fault injection capabilities infrastructure
3===========================================
4
5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
6
7
8Available fault injection capabilities
9--------------------------------------
10
11- failslab
12
13  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
14
15- fail_page_alloc
16
17  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
18
19- fail_usercopy
20
21  injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
22
23- fail_futex
24
25  injects futex deadlock and uaddr fault errors.
26
27- fail_sunrpc
28
29  injects kernel RPC client and server failures.
30
31- fail_make_request
32
33  injects disk IO errors on devices permitted by setting
34  /sys/block/<device>/make-it-fail or
35  /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
36
37- fail_mmc_request
38
39  injects MMC data errors on devices permitted by setting
40  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
41
42- fail_function
43
44  injects error return on specific functions, which are marked by
45  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
46  under /sys/kernel/debug/fail_function. No boot option supported.
47
48- NVMe fault injection
49
50  inject NVMe status code and retry flag on devices permitted by setting
51  debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
52  status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
53  retry flag can be set via the debugfs.
54
55
56Configure fault-injection capabilities behavior
57-----------------------------------------------
58
59debugfs entries
60^^^^^^^^^^^^^^^
61
62fault-inject-debugfs kernel module provides some debugfs entries for runtime
63configuration of fault-injection capabilities.
64
65- /sys/kernel/debug/fail*/probability:
66
67	likelihood of failure injection, in percent.
68
69	Format: <percent>
70
71	Note that one-failure-per-hundred is a very high error rate
72	for some testcases.  Consider setting probability=100 and configure
73	/sys/kernel/debug/fail*/interval for such testcases.
74
75- /sys/kernel/debug/fail*/interval:
76
77	specifies the interval between failures, for calls to
78	should_fail() that pass all the other tests.
79
80	Note that if you enable this, by setting interval>1, you will
81	probably want to set probability=100.
82
83- /sys/kernel/debug/fail*/times:
84
85	specifies how many times failures may happen at most. A value of -1
86	means "no limit".
87
88- /sys/kernel/debug/fail*/space:
89
90	specifies an initial resource "budget", decremented by "size"
91	on each call to should_fail(,size).  Failure injection is
92	suppressed until "space" reaches zero.
93
94- /sys/kernel/debug/fail*/verbose
95
96	Format: { 0 | 1 | 2 }
97
98	specifies the verbosity of the messages when failure is
99	injected.  '0' means no messages; '1' will print only a single
100	log line per failure; '2' will print a call trace too -- useful
101	to debug the problems revealed by fault injection.
102
103- /sys/kernel/debug/fail*/task-filter:
104
105	Format: { 'Y' | 'N' }
106
107	A value of 'N' disables filtering by process (default).
108	Any positive value limits failures to only processes indicated by
109	/proc/<pid>/make-it-fail==1.
110
111- /sys/kernel/debug/fail*/require-start,
112  /sys/kernel/debug/fail*/require-end,
113  /sys/kernel/debug/fail*/reject-start,
114  /sys/kernel/debug/fail*/reject-end:
115
116	specifies the range of virtual addresses tested during
117	stacktrace walking.  Failure is injected only if some caller
118	in the walked stacktrace lies within the required range, and
119	none lies within the rejected range.
120	Default required range is [0,ULONG_MAX) (whole of virtual address space).
121	Default rejected range is [0,0).
122
123- /sys/kernel/debug/fail*/stacktrace-depth:
124
125	specifies the maximum stacktrace depth walked during search
126	for a caller within [require-start,require-end) OR
127	[reject-start,reject-end).
128
129- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
130
131	Format: { 'Y' | 'N' }
132
133	default is 'Y', setting it to 'N' will also inject failures into
134	highmem/user allocations (__GFP_HIGHMEM allocations).
135
136- /sys/kernel/debug/failslab/ignore-gfp-wait:
137- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
138
139	Format: { 'Y' | 'N' }
140
141	default is 'Y', setting it to 'N' will also inject failures
142	into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations).
143
144- /sys/kernel/debug/fail_page_alloc/min-order:
145
146	specifies the minimum page allocation order to be injected
147	failures.
148
149- /sys/kernel/debug/fail_futex/ignore-private:
150
151	Format: { 'Y' | 'N' }
152
153	default is 'N', setting it to 'Y' will disable failure injections
154	when dealing with private (address space) futexes.
155
156- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
157
158	Format: { 'Y' | 'N' }
159
160	default is 'N', setting it to 'Y' will disable disconnect
161	injection on the RPC client.
162
163- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
164
165	Format: { 'Y' | 'N' }
166
167	default is 'N', setting it to 'Y' will disable disconnect
168	injection on the RPC server.
169
170- /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
171
172	Format: { 'Y' | 'N' }
173
174	default is 'N', setting it to 'Y' will disable cache wait
175	injection on the RPC server.
176
177- /sys/kernel/debug/fail_function/inject:
178
179	Format: { 'function-name' | '!function-name' | '' }
180
181	specifies the target function of error injection by name.
182	If the function name leads '!' prefix, given function is
183	removed from injection list. If nothing specified ('')
184	injection list is cleared.
185
186- /sys/kernel/debug/fail_function/injectable:
187
188	(read only) shows error injectable functions and what type of
189	error values can be specified. The error type will be one of
190	below;
191	- NULL:	retval must be 0.
192	- ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
193	- ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
194
195- /sys/kernel/debug/fail_function/<function-name>/retval:
196
197	specifies the "error" return value to inject to the given function.
198	This will be created when the user specifies a new injection entry.
199	Note that this file only accepts unsigned values. So, if you want to
200	use a negative errno, you better use 'printf' instead of 'echo', e.g.:
201	$ printf %#x -12 > retval
202
203Boot option
204^^^^^^^^^^^
205
206In order to inject faults while debugfs is not available (early boot time),
207use the boot option::
208
209	failslab=
210	fail_page_alloc=
211	fail_usercopy=
212	fail_make_request=
213	fail_futex=
214	mmc_core.fail_request=<interval>,<probability>,<space>,<times>
215
216proc entries
217^^^^^^^^^^^^
218
219- /proc/<pid>/fail-nth,
220  /proc/self/task/<tid>/fail-nth:
221
222	Write to this file of integer N makes N-th call in the task fail.
223	Read from this file returns a integer value. A value of '0' indicates
224	that the fault setup with a previous write to this file was injected.
225	A positive integer N indicates that the fault wasn't yet injected.
226	Note that this file enables all types of faults (slab, futex, etc).
227	This setting takes precedence over all other generic debugfs settings
228	like probability, interval, times, etc. But per-capability settings
229	(e.g. fail_futex/ignore-private) take precedence over it.
230
231	This feature is intended for systematic testing of faults in a single
232	system call. See an example below.
233
234How to add new fault injection capability
235-----------------------------------------
236
237- #include <linux/fault-inject.h>
238
239- define the fault attributes
240
241  DECLARE_FAULT_ATTR(name);
242
243  Please see the definition of struct fault_attr in fault-inject.h
244  for details.
245
246- provide a way to configure fault attributes
247
248- boot option
249
250  If you need to enable the fault injection capability from boot time, you can
251  provide boot option to configure it. There is a helper function for it:
252
253	setup_fault_attr(attr, str);
254
255- debugfs entries
256
257  failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
258  Helper functions:
259
260	fault_create_debugfs_attr(name, parent, attr);
261
262- module parameters
263
264  If the scope of the fault injection capability is limited to a
265  single kernel module, it is better to provide module parameters to
266  configure the fault attributes.
267
268- add a hook to insert failures
269
270  Upon should_fail() returning true, client code should inject a failure:
271
272	should_fail(attr, size);
273
274Application Examples
275--------------------
276
277- Inject slab allocation failures into module init/exit code::
278
279    #!/bin/bash
280
281    FAILTYPE=failslab
282    echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
283    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
284    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
285    echo -1 > /sys/kernel/debug/$FAILTYPE/times
286    echo 0 > /sys/kernel/debug/$FAILTYPE/space
287    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
288    echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
289
290    faulty_system()
291    {
292	bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
293    }
294
295    if [ $# -eq 0 ]
296    then
297	echo "Usage: $0 modulename [ modulename ... ]"
298	exit 1
299    fi
300
301    for m in $*
302    do
303	echo inserting $m...
304	faulty_system modprobe $m
305
306	echo removing $m...
307	faulty_system modprobe -r $m
308    done
309
310------------------------------------------------------------------------------
311
312- Inject page allocation failures only for a specific module::
313
314    #!/bin/bash
315
316    FAILTYPE=fail_page_alloc
317    module=$1
318
319    if [ -z $module ]
320    then
321	echo "Usage: $0 <modulename>"
322	exit 1
323    fi
324
325    modprobe $module
326
327    if [ ! -d /sys/module/$module/sections ]
328    then
329	echo Module $module is not loaded
330	exit 1
331    fi
332
333    cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
334    cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
335
336    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
337    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
338    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
339    echo -1 > /sys/kernel/debug/$FAILTYPE/times
340    echo 0 > /sys/kernel/debug/$FAILTYPE/space
341    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
342    echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
343    echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
344    echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
345
346    trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
347
348    echo "Injecting errors into the module $module... (interrupt to stop)"
349    sleep 1000000
350
351------------------------------------------------------------------------------
352
353- Inject open_ctree error while btrfs mount::
354
355    #!/bin/bash
356
357    rm -f testfile.img
358    dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
359    DEVICE=$(losetup --show -f testfile.img)
360    mkfs.btrfs -f $DEVICE
361    mkdir -p tmpmnt
362
363    FAILTYPE=fail_function
364    FAILFUNC=open_ctree
365    echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
366    printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
367    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
368    echo 100 > /sys/kernel/debug/$FAILTYPE/probability
369    echo 0 > /sys/kernel/debug/$FAILTYPE/interval
370    echo -1 > /sys/kernel/debug/$FAILTYPE/times
371    echo 0 > /sys/kernel/debug/$FAILTYPE/space
372    echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
373
374    mount -t btrfs $DEVICE tmpmnt
375    if [ $? -ne 0 ]
376    then
377	echo "SUCCESS!"
378    else
379	echo "FAILED!"
380	umount tmpmnt
381    fi
382
383    echo > /sys/kernel/debug/$FAILTYPE/inject
384
385    rmdir tmpmnt
386    losetup -d $DEVICE
387    rm testfile.img
388
389
390Tool to run command with failslab or fail_page_alloc
391----------------------------------------------------
392In order to make it easier to accomplish the tasks mentioned above, we can use
393tools/testing/fault-injection/failcmd.sh.  Please run a command
394"./tools/testing/fault-injection/failcmd.sh --help" for more information and
395see the following examples.
396
397Examples:
398
399Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
400allocation failure::
401
402	# ./tools/testing/fault-injection/failcmd.sh \
403		-- make -C tools/testing/selftests/ run_tests
404
405Same as above except to specify 100 times failures at most instead of one time
406at most by default::
407
408	# ./tools/testing/fault-injection/failcmd.sh --times=100 \
409		-- make -C tools/testing/selftests/ run_tests
410
411Same as above except to inject page allocation failure instead of slab
412allocation failure::
413
414	# env FAILCMD_TYPE=fail_page_alloc \
415		./tools/testing/fault-injection/failcmd.sh --times=100 \
416		-- make -C tools/testing/selftests/ run_tests
417
418Systematic faults using fail-nth
419---------------------------------
420
421The following code systematically faults 0-th, 1-st, 2-nd and so on
422capabilities in the socketpair() system call::
423
424  #include <sys/types.h>
425  #include <sys/stat.h>
426  #include <sys/socket.h>
427  #include <sys/syscall.h>
428  #include <fcntl.h>
429  #include <unistd.h>
430  #include <string.h>
431  #include <stdlib.h>
432  #include <stdio.h>
433  #include <errno.h>
434
435  int main()
436  {
437	int i, err, res, fail_nth, fds[2];
438	char buf[128];
439
440	system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
441	sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
442	fail_nth = open(buf, O_RDWR);
443	for (i = 1;; i++) {
444		sprintf(buf, "%d", i);
445		write(fail_nth, buf, strlen(buf));
446		res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
447		err = errno;
448		pread(fail_nth, buf, sizeof(buf), 0);
449		if (res == 0) {
450			close(fds[0]);
451			close(fds[1]);
452		}
453		printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
454			res, err);
455		if (atoi(buf))
456			break;
457	}
458	return 0;
459  }
460
461An example output::
462
463	1-th fault Y: res=-1/23
464	2-th fault Y: res=-1/23
465	3-th fault Y: res=-1/12
466	4-th fault Y: res=-1/12
467	5-th fault Y: res=-1/23
468	6-th fault Y: res=-1/23
469	7-th fault Y: res=-1/23
470	8-th fault Y: res=-1/12
471	9-th fault Y: res=-1/12
472	10-th fault Y: res=-1/12
473	11-th fault Y: res=-1/12
474	12-th fault Y: res=-1/12
475	13-th fault Y: res=-1/12
476	14-th fault Y: res=-1/12
477	15-th fault Y: res=-1/12
478	16-th fault N: res=0/12
479