linux/Documentation/fault-injection/fault-injection.txt
<<
>>
Prefs
   1Fault injection capabilities infrastructure
   2===========================================
   3
   4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
   5
   6
   7Available fault injection capabilities
   8--------------------------------------
   9
  10o failslab
  11
  12  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
  13
  14o fail_page_alloc
  15
  16  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
  17
  18o fail_make_request
  19
  20  injects disk IO errors on devices permitted by setting
  21  /sys/block/<device>/make-it-fail or
  22  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
  23
  24o fail_mmc_request
  25
  26  injects MMC data errors on devices permitted by setting
  27  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
  28
  29Configure fault-injection capabilities behavior
  30-----------------------------------------------
  31
  32o debugfs entries
  33
  34fault-inject-debugfs kernel module provides some debugfs entries for runtime
  35configuration of fault-injection capabilities.
  36
  37- /sys/kernel/debug/fail*/probability:
  38
  39        likelihood of failure injection, in percent.
  40        Format: <percent>
  41
  42        Note that one-failure-per-hundred is a very high error rate
  43        for some testcases.  Consider setting probability=100 and configure
  44        /sys/kernel/debug/fail*/interval for such testcases.
  45
  46- /sys/kernel/debug/fail*/interval:
  47
  48        specifies the interval between failures, for calls to
  49        should_fail() that pass all the other tests.
  50
  51        Note that if you enable this, by setting interval>1, you will
  52        probably want to set probability=100.
  53
  54- /sys/kernel/debug/fail*/times:
  55
  56        specifies how many times failures may happen at most.
  57        A value of -1 means "no limit".
  58
  59- /sys/kernel/debug/fail*/space:
  60
  61        specifies an initial resource "budget", decremented by "size"
  62        on each call to should_fail(,size).  Failure injection is
  63        suppressed until "space" reaches zero.
  64
  65- /sys/kernel/debug/fail*/verbose
  66
  67        Format: { 0 | 1 | 2 }
  68        specifies the verbosity of the messages when failure is
  69        injected.  '0' means no messages; '1' will print only a single
  70        log line per failure; '2' will print a call trace too -- useful
  71        to debug the problems revealed by fault injection.
  72
  73- /sys/kernel/debug/fail*/task-filter:
  74
  75        Format: { 'Y' | 'N' }
  76        A value of 'N' disables filtering by process (default).
  77        Any positive value limits failures to only processes indicated by
  78        /proc/<pid>/make-it-fail==1.
  79
  80- /sys/kernel/debug/fail*/require-start:
  81- /sys/kernel/debug/fail*/require-end:
  82- /sys/kernel/debug/fail*/reject-start:
  83- /sys/kernel/debug/fail*/reject-end:
  84
  85        specifies the range of virtual addresses tested during
  86        stacktrace walking.  Failure is injected only if some caller
  87        in the walked stacktrace lies within the required range, and
  88        none lies within the rejected range.
  89        Default required range is [0,ULONG_MAX) (whole of virtual address space).
  90        Default rejected range is [0,0).
  91
  92- /sys/kernel/debug/fail*/stacktrace-depth:
  93
  94        specifies the maximum stacktrace depth walked during search
  95        for a caller within [require-start,require-end) OR
  96        [reject-start,reject-end).
  97
  98- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
  99
 100        Format: { 'Y' | 'N' }
 101        default is 'N', setting it to 'Y' won't inject failures into
 102        highmem/user allocations.
 103
 104- /sys/kernel/debug/failslab/ignore-gfp-wait:
 105- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
 106
 107        Format: { 'Y' | 'N' }
 108        default is 'N', setting it to 'Y' will inject failures
 109        only into non-sleep allocations (GFP_ATOMIC allocations).
 110
 111- /sys/kernel/debug/fail_page_alloc/min-order:
 112
 113        specifies the minimum page allocation order to be injected
 114        failures.
 115
 116o Boot option
 117
 118In order to inject faults while debugfs is not available (early boot time),
 119use the boot option:
 120
 121        failslab=
 122        fail_page_alloc=
 123        fail_make_request=
 124        mmc_core.fail_request=<interval>,<probability>,<space>,<times>
 125
 126How to add new fault injection capability
 127-----------------------------------------
 128
 129o #include <linux/fault-inject.h>
 130
 131o define the fault attributes
 132
 133  DECLARE_FAULT_INJECTION(name);
 134
 135  Please see the definition of struct fault_attr in fault-inject.h
 136  for details.
 137
 138o provide a way to configure fault attributes
 139
 140- boot option
 141
 142  If you need to enable the fault injection capability from boot time, you can
 143  provide boot option to configure it. There is a helper function for it:
 144
 145        setup_fault_attr(attr, str);
 146
 147- debugfs entries
 148
 149  failslab, fail_page_alloc, and fail_make_request use this way.
 150  Helper functions:
 151
 152        fault_create_debugfs_attr(name, parent, attr);
 153
 154- module parameters
 155
 156  If the scope of the fault injection capability is limited to a
 157  single kernel module, it is better to provide module parameters to
 158  configure the fault attributes.
 159
 160o add a hook to insert failures
 161
 162  Upon should_fail() returning true, client code should inject a failure.
 163
 164        should_fail(attr, size);
 165
 166Application Examples
 167--------------------
 168
 169o Inject slab allocation failures into module init/exit code
 170
 171#!/bin/bash
 172
 173FAILTYPE=failslab
 174echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
 175echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 176echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 177echo -1 > /sys/kernel/debug/$FAILTYPE/times
 178echo 0 > /sys/kernel/debug/$FAILTYPE/space
 179echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 180echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 181
 182faulty_system()
 183{
 184        bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
 185}
 186
 187if [ $# -eq 0 ]
 188then
 189        echo "Usage: $0 modulename [ modulename ... ]"
 190        exit 1
 191fi
 192
 193for m in $*
 194do
 195        echo inserting $m...
 196        faulty_system modprobe $m
 197
 198        echo removing $m...
 199        faulty_system modprobe -r $m
 200done
 201
 202------------------------------------------------------------------------------
 203
 204o Inject page allocation failures only for a specific module
 205
 206#!/bin/bash
 207
 208FAILTYPE=fail_page_alloc
 209module=$1
 210
 211if [ -z $module ]
 212then
 213        echo "Usage: $0 <modulename>"
 214        exit 1
 215fi
 216
 217modprobe $module
 218
 219if [ ! -d /sys/module/$module/sections ]
 220then
 221        echo Module $module is not loaded
 222        exit 1
 223fi
 224
 225cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
 226cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
 227
 228echo N > /sys/kernel/debug/$FAILTYPE/task-filter
 229echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 230echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 231echo -1 > /sys/kernel/debug/$FAILTYPE/times
 232echo 0 > /sys/kernel/debug/$FAILTYPE/space
 233echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 234echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 235echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
 236echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
 237
 238trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
 239
 240echo "Injecting errors into the module $module... (interrupt to stop)"
 241sleep 1000000
 242
 243Tool to run command with failslab or fail_page_alloc
 244----------------------------------------------------
 245In order to make it easier to accomplish the tasks mentioned above, we can use
 246tools/testing/fault-injection/failcmd.sh.  Please run a command
 247"./tools/testing/fault-injection/failcmd.sh --help" for more information and
 248see the following examples.
 249
 250Examples:
 251
 252Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
 253allocation failure.
 254
 255        # ./tools/testing/fault-injection/failcmd.sh \
 256                -- make -C tools/testing/selftests/ run_tests
 257
 258Same as above except to specify 100 times failures at most instead of one time
 259at most by default.
 260
 261        # ./tools/testing/fault-injection/failcmd.sh --times=100 \
 262                -- make -C tools/testing/selftests/ run_tests
 263
 264Same as above except to inject page allocation failure instead of slab
 265allocation failure.
 266
 267        # env FAILCMD_TYPE=fail_page_alloc \
 268                ./tools/testing/fault-injection/failcmd.sh --times=100 \
 269                -- make -C tools/testing/selftests/ run_tests
 270
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.