1=========================================== 2Fault injection capabilities infrastructure 3=========================================== 4 5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug. 6 7 8Available fault injection capabilities 9-------------------------------------- 10 11- failslab 12 13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 14 15- fail_page_alloc 16 17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 18 19- fail_usercopy 20 21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...) 22 23- fail_futex 24 25 injects futex deadlock and uaddr fault errors. 26 27- fail_sunrpc 28 29 injects kernel RPC client and server failures. 30 31- fail_make_request 32 33 injects disk IO errors on devices permitted by setting 34 /sys/block/<device>/make-it-fail or 35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct()) 36 37- fail_mmc_request 38 39 injects MMC data errors on devices permitted by setting 40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 41 42- fail_function 43 44 injects error return on specific functions, which are marked by 45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries 46 under /sys/kernel/debug/fail_function. No boot option supported. 47 48- NVMe fault injection 49 50 inject NVMe status code and retry flag on devices permitted by setting 51 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default 52 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and 53 retry flag can be set via the debugfs. 54 55 56Configure fault-injection capabilities behavior 57----------------------------------------------- 58 59debugfs entries 60^^^^^^^^^^^^^^^ 61 62fault-inject-debugfs kernel module provides some debugfs entries for runtime 63configuration of fault-injection capabilities. 64 65- /sys/kernel/debug/fail*/probability: 66 67 likelihood of failure injection, in percent. 68 69 Format: <percent> 70 71 Note that one-failure-per-hundred is a very high error rate 72 for some testcases. Consider setting probability=100 and configure 73 /sys/kernel/debug/fail*/interval for such testcases. 74 75- /sys/kernel/debug/fail*/interval: 76 77 specifies the interval between failures, for calls to 78 should_fail() that pass all the other tests. 79 80 Note that if you enable this, by setting interval>1, you will 81 probably want to set probability=100. 82 83- /sys/kernel/debug/fail*/times: 84 85 specifies how many times failures may happen at most. A value of -1 86 means "no limit". Note, though, that this file only accepts unsigned 87 values. So, if you want to specify -1, you better use 'printf' instead 88 of 'echo', e.g.: $ printf %#x -1 > times 89 90- /sys/kernel/debug/fail*/space: 91 92 specifies an initial resource "budget", decremented by "size" 93 on each call to should_fail(,size). Failure injection is 94 suppressed until "space" reaches zero. 95 96- /sys/kernel/debug/fail*/verbose 97 98 Format: { 0 | 1 | 2 } 99 100 specifies the verbosity of the messages when failure is 101 injected. '0' means no messages; '1' will print only a single 102 log line per failure; '2' will print a call trace too -- useful 103 to debug the problems revealed by fault injection. 104 105- /sys/kernel/debug/fail*/task-filter: 106 107 Format: { 'Y' | 'N' } 108 109 A value of 'N' disables filtering by process (default). 110 Any positive value limits failures to only processes indicated by 111 /proc/<pid>/make-it-fail==1. 112 113- /sys/kernel/debug/fail*/require-start, 114 /sys/kernel/debug/fail*/require-end, 115 /sys/kernel/debug/fail*/reject-start, 116 /sys/kernel/debug/fail*/reject-end: 117 118 specifies the range of virtual addresses tested during 119 stacktrace walking. Failure is injected only if some caller 120 in the walked stacktrace lies within the required range, and 121 none lies within the rejected range. 122 Default required range is [0,ULONG_MAX) (whole of virtual address space). 123 Default rejected range is [0,0). 124 125- /sys/kernel/debug/fail*/stacktrace-depth: 126 127 specifies the maximum stacktrace depth walked during search 128 for a caller within [require-start,require-end) OR 129 [reject-start,reject-end). 130 131- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 132 133 Format: { 'Y' | 'N' } 134 135 default is 'N', setting it to 'Y' won't inject failures into 136 highmem/user allocations. 137 138- /sys/kernel/debug/failslab/ignore-gfp-wait: 139- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 140 141 Format: { 'Y' | 'N' } 142 143 default is 'N', setting it to 'Y' will inject failures 144 only into non-sleep allocations (GFP_ATOMIC allocations). 145 146- /sys/kernel/debug/fail_page_alloc/min-order: 147 148 specifies the minimum page allocation order to be injected 149 failures. 150 151- /sys/kernel/debug/fail_futex/ignore-private: 152 153 Format: { 'Y' | 'N' } 154 155 default is 'N', setting it to 'Y' will disable failure injections 156 when dealing with private (address space) futexes. 157 158- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect: 159 160 Format: { 'Y' | 'N' } 161 162 default is 'N', setting it to 'Y' will disable disconnect 163 injection on the RPC client. 164 165- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect: 166 167 Format: { 'Y' | 'N' } 168 169 default is 'N', setting it to 'Y' will disable disconnect 170 injection on the RPC server. 171 172- /sys/kernel/debug/fail_function/inject: 173 174 Format: { 'function-name' | '!function-name' | '' } 175 176 specifies the target function of error injection by name. 177 If the function name leads '!' prefix, given function is 178 removed from injection list. If nothing specified ('') 179 injection list is cleared. 180 181- /sys/kernel/debug/fail_function/injectable: 182 183 (read only) shows error injectable functions and what type of 184 error values can be specified. The error type will be one of 185 below; 186 - NULL: retval must be 0. 187 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096). 188 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096). 189 190- /sys/kernel/debug/fail_function/<function-name>/retval: 191 192 specifies the "error" return value to inject to the given function. 193 This will be created when the user specifies a new injection entry. 194 Note that this file only accepts unsigned values. So, if you want to 195 use a negative errno, you better use 'printf' instead of 'echo', e.g.: 196 $ printf %#x -12 > retval 197 198Boot option 199^^^^^^^^^^^ 200 201In order to inject faults while debugfs is not available (early boot time), 202use the boot option:: 203 204 failslab= 205 fail_page_alloc= 206 fail_usercopy= 207 fail_make_request= 208 fail_futex= 209 mmc_core.fail_request=<interval>,<probability>,<space>,<times> 210 211proc entries 212^^^^^^^^^^^^ 213 214- /proc/<pid>/fail-nth, 215 /proc/self/task/<tid>/fail-nth: 216 217 Write to this file of integer N makes N-th call in the task fail. 218 Read from this file returns a integer value. A value of '0' indicates 219 that the fault setup with a previous write to this file was injected. 220 A positive integer N indicates that the fault wasn't yet injected. 221 Note that this file enables all types of faults (slab, futex, etc). 222 This setting takes precedence over all other generic debugfs settings 223 like probability, interval, times, etc. But per-capability settings 224 (e.g. fail_futex/ignore-private) take precedence over it. 225 226 This feature is intended for systematic testing of faults in a single 227 system call. See an example below. 228 229How to add new fault injection capability 230----------------------------------------- 231 232- #include <linux/fault-inject.h> 233 234- define the fault attributes 235 236 DECLARE_FAULT_ATTR(name); 237 238 Please see the definition of struct fault_attr in fault-inject.h 239 for details. 240 241- provide a way to configure fault attributes 242 243- boot option 244 245 If you need to enable the fault injection capability from boot time, you can 246 provide boot option to configure it. There is a helper function for it: 247 248 setup_fault_attr(attr, str); 249 250- debugfs entries 251 252 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way. 253 Helper functions: 254 255 fault_create_debugfs_attr(name, parent, attr); 256 257- module parameters 258 259 If the scope of the fault injection capability is limited to a 260 single kernel module, it is better to provide module parameters to 261 configure the fault attributes. 262 263- add a hook to insert failures 264 265 Upon should_fail() returning true, client code should inject a failure: 266 267 should_fail(attr, size); 268 269Application Examples 270-------------------- 271 272- Inject slab allocation failures into module init/exit code:: 273 274 #!/bin/bash 275 276 FAILTYPE=failslab 277 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 278 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 279 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 280 printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 281 echo 0 > /sys/kernel/debug/$FAILTYPE/space 282 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 283 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 284 285 faulty_system() 286 { 287 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 288 } 289 290 if [ $# -eq 0 ] 291 then 292 echo "Usage: $0 modulename [ modulename ... ]" 293 exit 1 294 fi 295 296 for m in $* 297 do 298 echo inserting $m... 299 faulty_system modprobe $m 300 301 echo removing $m... 302 faulty_system modprobe -r $m 303 done 304 305------------------------------------------------------------------------------ 306 307- Inject page allocation failures only for a specific module:: 308 309 #!/bin/bash 310 311 FAILTYPE=fail_page_alloc 312 module=$1 313 314 if [ -z $module ] 315 then 316 echo "Usage: $0 <modulename>" 317 exit 1 318 fi 319 320 modprobe $module 321 322 if [ ! -d /sys/module/$module/sections ] 323 then 324 echo Module $module is not loaded 325 exit 1 326 fi 327 328 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 329 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 330 331 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 332 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 333 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 334 printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 335 echo 0 > /sys/kernel/debug/$FAILTYPE/space 336 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 337 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 338 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 339 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 340 341 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 342 343 echo "Injecting errors into the module $module... (interrupt to stop)" 344 sleep 1000000 345 346------------------------------------------------------------------------------ 347 348- Inject open_ctree error while btrfs mount:: 349 350 #!/bin/bash 351 352 rm -f testfile.img 353 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 354 DEVICE=$(losetup --show -f testfile.img) 355 mkfs.btrfs -f $DEVICE 356 mkdir -p tmpmnt 357 358 FAILTYPE=fail_function 359 FAILFUNC=open_ctree 360 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 361 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 362 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 363 echo 100 > /sys/kernel/debug/$FAILTYPE/probability 364 echo 0 > /sys/kernel/debug/$FAILTYPE/interval 365 printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times 366 echo 0 > /sys/kernel/debug/$FAILTYPE/space 367 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 368 369 mount -t btrfs $DEVICE tmpmnt 370 if [ $? -ne 0 ] 371 then 372 echo "SUCCESS!" 373 else 374 echo "FAILED!" 375 umount tmpmnt 376 fi 377 378 echo > /sys/kernel/debug/$FAILTYPE/inject 379 380 rmdir tmpmnt 381 losetup -d $DEVICE 382 rm testfile.img 383 384 385Tool to run command with failslab or fail_page_alloc 386---------------------------------------------------- 387In order to make it easier to accomplish the tasks mentioned above, we can use 388tools/testing/fault-injection/failcmd.sh. Please run a command 389"./tools/testing/fault-injection/failcmd.sh --help" for more information and 390see the following examples. 391 392Examples: 393 394Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab 395allocation failure:: 396 397 # ./tools/testing/fault-injection/failcmd.sh \ 398 -- make -C tools/testing/selftests/ run_tests 399 400Same as above except to specify 100 times failures at most instead of one time 401at most by default:: 402 403 # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 404 -- make -C tools/testing/selftests/ run_tests 405 406Same as above except to inject page allocation failure instead of slab 407allocation failure:: 408 409 # env FAILCMD_TYPE=fail_page_alloc \ 410 ./tools/testing/fault-injection/failcmd.sh --times=100 \ 411 -- make -C tools/testing/selftests/ run_tests 412 413Systematic faults using fail-nth 414--------------------------------- 415 416The following code systematically faults 0-th, 1-st, 2-nd and so on 417capabilities in the socketpair() system call:: 418 419 #include <sys/types.h> 420 #include <sys/stat.h> 421 #include <sys/socket.h> 422 #include <sys/syscall.h> 423 #include <fcntl.h> 424 #include <unistd.h> 425 #include <string.h> 426 #include <stdlib.h> 427 #include <stdio.h> 428 #include <errno.h> 429 430 int main() 431 { 432 int i, err, res, fail_nth, fds[2]; 433 char buf[128]; 434 435 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait"); 436 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid)); 437 fail_nth = open(buf, O_RDWR); 438 for (i = 1;; i++) { 439 sprintf(buf, "%d", i); 440 write(fail_nth, buf, strlen(buf)); 441 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds); 442 err = errno; 443 pread(fail_nth, buf, sizeof(buf), 0); 444 if (res == 0) { 445 close(fds[0]); 446 close(fds[1]); 447 } 448 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y', 449 res, err); 450 if (atoi(buf)) 451 break; 452 } 453 return 0; 454 } 455 456An example output:: 457 458 1-th fault Y: res=-1/23 459 2-th fault Y: res=-1/23 460 3-th fault Y: res=-1/12 461 4-th fault Y: res=-1/12 462 5-th fault Y: res=-1/23 463 6-th fault Y: res=-1/23 464 7-th fault Y: res=-1/23 465 8-th fault Y: res=-1/12 466 9-th fault Y: res=-1/12 467 10-th fault Y: res=-1/12 468 11-th fault Y: res=-1/12 469 12-th fault Y: res=-1/12 470 13-th fault Y: res=-1/12 471 14-th fault Y: res=-1/12 472 15-th fault Y: res=-1/12 473 16-th fault N: res=0/12 474