1===========================================
2Fault injection capabilities infrastructure
3===========================================
4
5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
6
7
8Available fault injection capabilities
9--------------------------------------
10
11- failslab
12
13  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
14
15- fail_page_alloc
16
17  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
18
19- fail_futex
20
21  injects futex deadlock and uaddr fault errors.
22
23- fail_make_request
24
25  injects disk IO errors on devices permitted by setting
26  /sys/block/<device>/make-it-fail or
27  /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
28
29- fail_mmc_request
30
31  injects MMC data errors on devices permitted by setting
32  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
33
34- fail_function
35
36  injects error return on specific functions, which are marked by
37  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
38  under /sys/kernel/debug/fail_function. No boot option supported.
39
40- NVMe fault injection
41
42  inject NVMe status code and retry flag on devices permitted by setting
43  debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
44  status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
45  retry flag can be set via the debugfs.
46
47
48Configure fault-injection capabilities behavior
49-----------------------------------------------
50
51debugfs entries
52^^^^^^^^^^^^^^^
53
54fault-inject-debugfs kernel module provides some debugfs entries for runtime
55configuration of fault-injection capabilities.
56
57- /sys/kernel/debug/fail*/probability:
58
59	likelihood of failure injection, in percent.
60
61	Format: <percent>
62
63	Note that one-failure-per-hundred is a very high error rate
64	for some testcases.  Consider setting probability=100 and configure
65	/sys/kernel/debug/fail*/interval for such testcases.
66
67- /sys/kernel/debug/fail*/interval:
68
69	specifies the interval between failures, for calls to
70	should_fail() that pass all the other tests.
71
72	Note that if you enable this, by setting interval>1, you will
73	probably want to set probability=100.
74
75- /sys/kernel/debug/fail*/times:
76
77	specifies how many times failures may happen at most.
78	A value of -1 means "no limit".
79
80- /sys/kernel/debug/fail*/space:
81
82	specifies an initial resource "budget", decremented by "size"
83	on each call to should_fail(,size).  Failure injection is
84	suppressed until "space" reaches zero.
85
86- /sys/kernel/debug/fail*/verbose
87
88	Format: { 0 | 1 | 2 }
89
90	specifies the verbosity of the messages when failure is
91	injected.  '0' means no messages; '1' will print only a single
92	log line per failure; '2' will print a call trace too -- useful
93	to debug the problems revealed by fault injection.
94
95- /sys/kernel/debug/fail*/task-filter:
96
97	Format: { 'Y' | 'N' }
98
99	A value of 'N' disables filtering by process (default).
100	Any positive value limits failures to only processes indicated by
101	/proc/<pid>/make-it-fail==1.
102
103- /sys/kernel/debug/fail*/require-start,
104  /sys/kernel/debug/fail*/require-end,
105  /sys/kernel/debug/fail*/reject-start,
106  /sys/kernel/debug/fail*/reject-end:
107
108	specifies the range of virtual addresses tested during
109	stacktrace walking.  Failure is injected only if some caller
110	in the walked stacktrace lies within the required range, and
111	none lies within the rejected range.
112	Default required range is [0,ULONG_MAX) (whole of virtual address space).
113	Default rejected range is [0,0).
114
115- /sys/kernel/debug/fail*/stacktrace-depth:
116
117	specifies the maximum stacktrace depth walked during search
118	for a caller within [require-start,require-end) OR
119	[reject-start,reject-end).
120
121- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
122
123	Format: { 'Y' | 'N' }
124
125	default is 'N', setting it to 'Y' won't inject failures into
126	highmem/user allocations.
127
128- /sys/kernel/debug/failslab/ignore-gfp-wait:
129- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
130
131	Format: { 'Y' | 'N' }
132
133	default is 'N', setting it to 'Y' will inject failures
134	only into non-sleep allocations (GFP_ATOMIC allocations).
135
136- /sys/kernel/debug/fail_page_alloc/min-order:
137
138	specifies the minimum page allocation order to be injected
139	failures.
140
141- /sys/kernel/debug/fail_futex/ignore-private:
142
143	Format: { 'Y' | 'N' }
144
145	default is 'N', setting it to 'Y' will disable failure injections
146	when dealing with private (address space) futexes.
147
148- /sys/kernel/debug/fail_function/inject:
149
150	Format: { 'function-name' | '!function-name' | '' }
151
152	specifies the target function of error injection by name.
153	If the function name leads '!' prefix, given function is
154	removed from injection list. If nothing specified ('')
155	injection list is cleared.
156
157- /sys/kernel/debug/fail_function/injectable:
158
159	(read only) shows error injectable functions and what type of
160	error values can be specified. The error type will be one of
161	below;
162	- NULL:	retval must be 0.
163	- ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
164	- ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
165
166- /sys/kernel/debug/fail_function/<functiuon-name>/retval:
167
168	specifies the "error" return value to inject to the given
169	function for given function. This will be created when
170	user specifies new injection entry.
171
172Boot option
173^^^^^^^^^^^
174
175In order to inject faults while debugfs is not available (early boot time),
176use the boot option::
177
178	failslab=
179	fail_page_alloc=
180	fail_make_request=
181	fail_futex=
182	mmc_core.fail_request=<interval>,<probability>,<space>,<times>
183
184proc entries
185^^^^^^^^^^^^
186
187- /proc/<pid>/fail-nth,
188  /proc/self/task/<tid>/fail-nth:
189
190	Write to this file of integer N makes N-th call in the task fail.
191	Read from this file returns a integer value. A value of '0' indicates
192	that the fault setup with a previous write to this file was injected.
193	A positive integer N indicates that the fault wasn't yet injected.
194	Note that this file enables all types of faults (slab, futex, etc).
195	This setting takes precedence over all other generic debugfs settings
196	like probability, interval, times, etc. But per-capability settings
197	(e.g. fail_futex/ignore-private) take precedence over it.
198
199	This feature is intended for systematic testing of faults in a single
200	system call. See an example below.
201
202How to add new fault injection capability
203-----------------------------------------
204
205- #include <linux/fault-inject.h>
206
207- define the fault attributes
208
209  DECLARE_FAULT_ATTR(name);
210
211  Please see the definition of struct fault_attr in fault-inject.h
212  for details.
213
214- provide a way to configure fault attributes
215
216- boot option
217
218  If you need to enable the fault injection capability from boot time, you can
219  provide boot option to configure it. There is a helper function for it:
220
221	setup_fault_attr(attr, str);
222
223- debugfs entries
224
225  failslab, fail_page_alloc, and fail_make_request use this way.
226  Helper functions:
227
228	fault_create_debugfs_attr(name, parent, attr);
229
230- module parameters
231
232  If the scope of the fault injection capability is limited to a
233  single kernel module, it is better to provide module parameters to
234  configure the fault attributes.
235
236- add a hook to insert failures
237
238  Upon should_fail() returning true, client code should inject a failure:
239
240	should_fail(attr, size);
241
242Application Examples
243--------------------
244
245- Inject slab allocation failures into module init/exit code::
246
247    #!/bin/bash
248
249    FAILTYPE=failslab
250    echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
251    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
252    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
253    echo -1 > /sys/kernel/debug/$FAILTYPE/times
254    echo 0 > /sys/kernel/debug/$FAILTYPE/space
255    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
256    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
257
258    faulty_system()
259    {
260	bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
261    }
262
263    if [ $# -eq 0 ]
264    then
265	echo "Usage: $0 modulename [ modulename ... ]"
266	exit 1
267    fi
268
269    for m in $*
270    do
271	echo inserting $m...
272	faulty_system modprobe $m
273
274	echo removing $m...
275	faulty_system modprobe -r $m
276    done
277
278------------------------------------------------------------------------------
279
280- Inject page allocation failures only for a specific module::
281
282    #!/bin/bash
283
284    FAILTYPE=fail_page_alloc
285    module=$1
286
287    if [ -z $module ]
288    then
289	echo "Usage: $0 <modulename>"
290	exit 1
291    fi
292
293    modprobe $module
294
295    if [ ! -d /sys/module/$module/sections ]
296    then
297	echo Module $module is not loaded
298	exit 1
299    fi
300
301    cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
302    cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
303
304    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
305    echo 10 > /sys/kernel/debug/$FAILTYPE/probability
306    echo 100 > /sys/kernel/debug/$FAILTYPE/interval
307    echo -1 > /sys/kernel/debug/$FAILTYPE/times
308    echo 0 > /sys/kernel/debug/$FAILTYPE/space
309    echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
310    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
311    echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
312    echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
313
314    trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
315
316    echo "Injecting errors into the module $module... (interrupt to stop)"
317    sleep 1000000
318
319------------------------------------------------------------------------------
320
321- Inject open_ctree error while btrfs mount::
322
323    #!/bin/bash
324
325    rm -f testfile.img
326    dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
327    DEVICE=$(losetup --show -f testfile.img)
328    mkfs.btrfs -f $DEVICE
329    mkdir -p tmpmnt
330
331    FAILTYPE=fail_function
332    FAILFUNC=open_ctree
333    echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
334    echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
335    echo N > /sys/kernel/debug/$FAILTYPE/task-filter
336    echo 100 > /sys/kernel/debug/$FAILTYPE/probability
337    echo 0 > /sys/kernel/debug/$FAILTYPE/interval
338    echo -1 > /sys/kernel/debug/$FAILTYPE/times
339    echo 0 > /sys/kernel/debug/$FAILTYPE/space
340    echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
341
342    mount -t btrfs $DEVICE tmpmnt
343    if [ $? -ne 0 ]
344    then
345	echo "SUCCESS!"
346    else
347	echo "FAILED!"
348	umount tmpmnt
349    fi
350
351    echo > /sys/kernel/debug/$FAILTYPE/inject
352
353    rmdir tmpmnt
354    losetup -d $DEVICE
355    rm testfile.img
356
357
358Tool to run command with failslab or fail_page_alloc
359----------------------------------------------------
360In order to make it easier to accomplish the tasks mentioned above, we can use
361tools/testing/fault-injection/failcmd.sh.  Please run a command
362"./tools/testing/fault-injection/failcmd.sh --help" for more information and
363see the following examples.
364
365Examples:
366
367Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
368allocation failure::
369
370	# ./tools/testing/fault-injection/failcmd.sh \
371		-- make -C tools/testing/selftests/ run_tests
372
373Same as above except to specify 100 times failures at most instead of one time
374at most by default::
375
376	# ./tools/testing/fault-injection/failcmd.sh --times=100 \
377		-- make -C tools/testing/selftests/ run_tests
378
379Same as above except to inject page allocation failure instead of slab
380allocation failure::
381
382	# env FAILCMD_TYPE=fail_page_alloc \
383		./tools/testing/fault-injection/failcmd.sh --times=100 \
384		-- make -C tools/testing/selftests/ run_tests
385
386Systematic faults using fail-nth
387---------------------------------
388
389The following code systematically faults 0-th, 1-st, 2-nd and so on
390capabilities in the socketpair() system call::
391
392  #include <sys/types.h>
393  #include <sys/stat.h>
394  #include <sys/socket.h>
395  #include <sys/syscall.h>
396  #include <fcntl.h>
397  #include <unistd.h>
398  #include <string.h>
399  #include <stdlib.h>
400  #include <stdio.h>
401  #include <errno.h>
402
403  int main()
404  {
405	int i, err, res, fail_nth, fds[2];
406	char buf[128];
407
408	system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
409	sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
410	fail_nth = open(buf, O_RDWR);
411	for (i = 1;; i++) {
412		sprintf(buf, "%d", i);
413		write(fail_nth, buf, strlen(buf));
414		res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
415		err = errno;
416		pread(fail_nth, buf, sizeof(buf), 0);
417		if (res == 0) {
418			close(fds[0]);
419			close(fds[1]);
420		}
421		printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
422			res, err);
423		if (atoi(buf))
424			break;
425	}
426	return 0;
427  }
428
429An example output::
430
431	1-th fault Y: res=-1/23
432	2-th fault Y: res=-1/23
433	3-th fault Y: res=-1/12
434	4-th fault Y: res=-1/12
435	5-th fault Y: res=-1/23
436	6-th fault Y: res=-1/23
437	7-th fault Y: res=-1/23
438	8-th fault Y: res=-1/12
439	9-th fault Y: res=-1/12
440	10-th fault Y: res=-1/12
441	11-th fault Y: res=-1/12
442	12-th fault Y: res=-1/12
443	13-th fault Y: res=-1/12
444	14-th fault Y: res=-1/12
445	15-th fault Y: res=-1/12
446	16-th fault N: res=0/12
447