Documentation/fault-injection/fault-injection.rst

5 See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
9 --------------------------------------
11 - failslab
15 - fail_page_alloc
19 - fail_usercopy
23 - fail_futex
25   injects futex deadlock and uaddr fault errors.
27 - fail_sunrpc
31 - fail_make_request
33   injects disk IO errors on devices permitted by setting
34   /sys/block/<device>/make-it-fail or
35   /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
37 - fail_mmc_request
39   injects MMC data errors on devices permitted by setting
42 - fail_function
48 - NVMe fault injection
50   inject NVMe status code and retry flag on devices permitted by setting
52   status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
53   retry flag can be set via the debugfs.
55 - Null test block driver fault injection
61   inject init_hctx() errors by setting config items under
64 Configure fault-injection capabilities behavior
65 -----------------------------------------------
70 fault-inject-debugfs kernel module provides some debugfs entries for runtime
71 configuration of fault-injection capabilities.
73 - /sys/kernel/debug/fail*/probability:
79 	Note that one-failure-per-hundred is a very high error rate
83 - /sys/kernel/debug/fail*/interval:
86 	should_fail() that pass all the other tests.
91 - /sys/kernel/debug/fail*/times:
93 	specifies how many times failures may happen at most. A value of -1
96 - /sys/kernel/debug/fail*/space:
102 - /sys/kernel/debug/fail*/verbose
108 	log line per failure; '2' will print a call trace too -- useful
111 - /sys/kernel/debug/fail*/task-filter:
117 	/proc/<pid>/make-it-fail==1.
119 - /sys/kernel/debug/fail*/require-start,
120   /sys/kernel/debug/fail*/require-end,
121   /sys/kernel/debug/fail*/reject-start,
122   /sys/kernel/debug/fail*/reject-end:
131 - /sys/kernel/debug/fail*/stacktrace-depth:
134 	for a caller within [require-start,require-end) OR
135 	[reject-start,reject-end).
137 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
144 - /sys/kernel/debug/failslab/ignore-gfp-wait:
145 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
152 - /sys/kernel/debug/fail_page_alloc/min-order:
157 - /sys/kernel/debug/fail_futex/ignore-private:
164 - /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
171 - /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
178 - /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
185 - /sys/kernel/debug/fail_function/inject:
187 	Format: { 'function-name' | '!function-name' | '' }
194 - /sys/kernel/debug/fail_function/injectable:
199 	- NULL:	retval must be 0.
200 	- ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
201 	- ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
203 - /sys/kernel/debug/fail_function/<function-name>/retval:
209 	$ printf %#x -12 > retval
227 - /proc/<pid>/fail-nth,
228   /proc/self/task/<tid>/fail-nth:
230 	Write to this file of integer N makes N-th call in the task fail.
234 	Note that this file enables all types of faults (slab, futex, etc).
235 	This setting takes precedence over all other generic debugfs settings
236 	like probability, interval, times, etc. But per-capability settings
237 	(e.g. fail_futex/ignore-private) take precedence over it.
244 --------------------------
252 Since the function-level error injection forcibly changes the code path
257 - The function returns an error code if it fails, and the callers must check
260 - The function does not execute any code which can change any state before
267 (free objects) functions are usually harder to inject errors than allocate
268 functions. If errors of such release functions are not correctly handled
283 There are 4 types of errors defined in include/asm-generic/error-injection.h
290   This function will return an `-errno` error code if it fails. e.g. return
291   -EINVAL if the input is wrong. This will include the functions which will
292   return an address which encodes `-errno` by ERR_PTR() macro.
295   This function will return an `-errno` or `NULL` if it fails. If the caller
300   This function will return `true` (non-zero positive value) if it fails.
308 -----------------------------------------
310 - #include <linux/fault-inject.h>
312 - define the fault attributes
316   Please see the definition of struct fault_attr in fault-inject.h
319 - provide a way to configure fault attributes
321 - boot option
328 - debugfs entries
335 - module parameters
341 - add a hook to insert failures
348 --------------------
350 - Inject slab allocation failures into module init/exit code::
355     echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
358     echo -1 > /sys/kernel/debug/$FAILTYPE/times
361     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
365 	bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
368     if [ $# -eq 0 ]
380 	faulty_system modprobe -r $m
383 ------------------------------------------------------------------------------
385 - Inject page allocation failures only for a specific module::
392     if [ -z $module ]
400     if [ ! -d /sys/module/$module/sections ]
406     cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
407     cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
409     echo N > /sys/kernel/debug/$FAILTYPE/task-filter
412     echo -1 > /sys/kernel/debug/$FAILTYPE/times
415     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
416     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
417     echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
421     echo "Injecting errors into the module $module... (interrupt to stop)"
424 ------------------------------------------------------------------------------
426 - Inject open_ctree error while btrfs mount::
430     rm -f testfile.img
432     DEVICE=$(losetup --show -f testfile.img)
433     mkfs.btrfs -f $DEVICE
434     mkdir -p tmpmnt
439     printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
440     echo N > /sys/kernel/debug/$FAILTYPE/task-filter
443     echo -1 > /sys/kernel/debug/$FAILTYPE/times
447     mount -t btrfs $DEVICE tmpmnt
448     if [ $? -ne 0 ]
459     losetup -d $DEVICE
464 ----------------------------------------------------
466 tools/testing/fault-injection/failcmd.sh.  Please run a command
467 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
472 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
475 	# ./tools/testing/fault-injection/failcmd.sh \
476 		-- make -C tools/testing/selftests/ run_tests
481 	# ./tools/testing/fault-injection/failcmd.sh --times=100 \
482 		-- make -C tools/testing/selftests/ run_tests
488 		./tools/testing/fault-injection/failcmd.sh --times=100 \
489 		-- make -C tools/testing/selftests/ run_tests
491 Systematic faults using fail-nth
492 ---------------------------------
494 The following code systematically faults 0-th, 1-st, 2-nd and so on
513 	system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
514 	sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
526 		printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
536 	1-th fault Y: res=-1/23
537 	2-th fault Y: res=-1/23
538 	3-th fault Y: res=-1/12
539 	4-th fault Y: res=-1/12
540 	5-th fault Y: res=-1/23
541 	6-th fault Y: res=-1/23
542 	7-th fault Y: res=-1/23
543 	8-th fault Y: res=-1/12
544 	9-th fault Y: res=-1/12
545 	10-th fault Y: res=-1/12
546 	11-th fault Y: res=-1/12
547 	12-th fault Y: res=-1/12
548 	13-th fault Y: res=-1/12
549 	14-th fault Y: res=-1/12
550 	15-th fault Y: res=-1/12
551 	16-th fault N: res=0/12