linux/Documentation/fault-injection/fault-injection.txt
<<
>>
Prefs
   1Fault injection capabilities infrastructure
   2===========================================
   3
   4See also drivers/md/faulty.c and "every_nth" module option for scsi_debug.
   5
   6
   7Available fault injection capabilities
   8--------------------------------------
   9
  10o failslab
  11
  12  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
  13
  14o fail_page_alloc
  15
  16  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
  17
  18o fail_make_request
  19
  20  injects disk IO errors on devices permitted by setting
  21  /sys/block/<device>/make-it-fail or
  22  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
  23
  24Configure fault-injection capabilities behavior
  25-----------------------------------------------
  26
  27o debugfs entries
  28
  29fault-inject-debugfs kernel module provides some debugfs entries for runtime
  30configuration of fault-injection capabilities.
  31
  32- /debug/fail*/probability:
  33
  34        likelihood of failure injection, in percent.
  35        Format: <percent>
  36
  37        Note that one-failure-per-hundred is a very high error rate
  38        for some testcases.  Consider setting probability=100 and configure
  39        /debug/fail*/interval for such testcases.
  40
  41- /debug/fail*/interval:
  42
  43        specifies the interval between failures, for calls to
  44        should_fail() that pass all the other tests.
  45
  46        Note that if you enable this, by setting interval>1, you will
  47        probably want to set probability=100.
  48
  49- /debug/fail*/times:
  50
  51        specifies how many times failures may happen at most.
  52        A value of -1 means "no limit".
  53
  54- /debug/fail*/space:
  55
  56        specifies an initial resource "budget", decremented by "size"
  57        on each call to should_fail(,size).  Failure injection is
  58        suppressed until "space" reaches zero.
  59
  60- /debug/fail*/verbose
  61
  62        Format: { 0 | 1 | 2 }
  63        specifies the verbosity of the messages when failure is
  64        injected.  '0' means no messages; '1' will print only a single
  65        log line per failure; '2' will print a call trace too -- useful
  66        to debug the problems revealed by fault injection.
  67
  68- /debug/fail*/task-filter:
  69
  70        Format: { 'Y' | 'N' }
  71        A value of 'N' disables filtering by process (default).
  72        Any positive value limits failures to only processes indicated by
  73        /proc/<pid>/make-it-fail==1.
  74
  75- /debug/fail*/require-start:
  76- /debug/fail*/require-end:
  77- /debug/fail*/reject-start:
  78- /debug/fail*/reject-end:
  79
  80        specifies the range of virtual addresses tested during
  81        stacktrace walking.  Failure is injected only if some caller
  82        in the walked stacktrace lies within the required range, and
  83        none lies within the rejected range.
  84        Default required range is [0,ULONG_MAX) (whole of virtual address space).
  85        Default rejected range is [0,0).
  86
  87- /debug/fail*/stacktrace-depth:
  88
  89        specifies the maximum stacktrace depth walked during search
  90        for a caller within [require-start,require-end) OR
  91        [reject-start,reject-end).
  92
  93- /debug/fail_page_alloc/ignore-gfp-highmem:
  94
  95        Format: { 'Y' | 'N' }
  96        default is 'N', setting it to 'Y' won't inject failures into
  97        highmem/user allocations.
  98
  99- /debug/failslab/ignore-gfp-wait:
 100- /debug/fail_page_alloc/ignore-gfp-wait:
 101
 102        Format: { 'Y' | 'N' }
 103        default is 'N', setting it to 'Y' will inject failures
 104        only into non-sleep allocations (GFP_ATOMIC allocations).
 105
 106- /debug/fail_page_alloc/min-order:
 107
 108        specifies the minimum page allocation order to be injected
 109        failures.
 110
 111o Boot option
 112
 113In order to inject faults while debugfs is not available (early boot time),
 114use the boot option:
 115
 116        failslab=
 117        fail_page_alloc=
 118        fail_make_request=<interval>,<probability>,<space>,<times>
 119
 120How to add new fault injection capability
 121-----------------------------------------
 122
 123o #include <linux/fault-inject.h>
 124
 125o define the fault attributes
 126
 127  DECLARE_FAULT_INJECTION(name);
 128
 129  Please see the definition of struct fault_attr in fault-inject.h
 130  for details.
 131
 132o provide a way to configure fault attributes
 133
 134- boot option
 135
 136  If you need to enable the fault injection capability from boot time, you can
 137  provide boot option to configure it. There is a helper function for it:
 138
 139        setup_fault_attr(attr, str);
 140
 141- debugfs entries
 142
 143  failslab, fail_page_alloc, and fail_make_request use this way.
 144  Helper functions:
 145
 146        init_fault_attr_entries(entries, attr, name);
 147        void cleanup_fault_attr_entries(entries);
 148
 149- module parameters
 150
 151  If the scope of the fault injection capability is limited to a
 152  single kernel module, it is better to provide module parameters to
 153  configure the fault attributes.
 154
 155o add a hook to insert failures
 156
 157  Upon should_fail() returning true, client code should inject a failure.
 158
 159        should_fail(attr, size);
 160
 161Application Examples
 162--------------------
 163
 164o Inject slab allocation failures into module init/exit code
 165
 166#!/bin/bash
 167
 168FAILTYPE=failslab
 169echo Y > /debug/$FAILTYPE/task-filter
 170echo 10 > /debug/$FAILTYPE/probability
 171echo 100 > /debug/$FAILTYPE/interval
 172echo -1 > /debug/$FAILTYPE/times
 173echo 0 > /debug/$FAILTYPE/space
 174echo 2 > /debug/$FAILTYPE/verbose
 175echo 1 > /debug/$FAILTYPE/ignore-gfp-wait
 176
 177faulty_system()
 178{
 179        bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
 180}
 181
 182if [ $# -eq 0 ]
 183then
 184        echo "Usage: $0 modulename [ modulename ... ]"
 185        exit 1
 186fi
 187
 188for m in $*
 189do
 190        echo inserting $m...
 191        faulty_system modprobe $m
 192
 193        echo removing $m...
 194        faulty_system modprobe -r $m
 195done
 196
 197------------------------------------------------------------------------------
 198
 199o Inject page allocation failures only for a specific module
 200
 201#!/bin/bash
 202
 203FAILTYPE=fail_page_alloc
 204module=$1
 205
 206if [ -z $module ]
 207then
 208        echo "Usage: $0 <modulename>"
 209        exit 1
 210fi
 211
 212modprobe $module
 213
 214if [ ! -d /sys/module/$module/sections ]
 215then
 216        echo Module $module is not loaded
 217        exit 1
 218fi
 219
 220cat /sys/module/$module/sections/.text > /debug/$FAILTYPE/require-start
 221cat /sys/module/$module/sections/.data > /debug/$FAILTYPE/require-end
 222
 223echo N > /debug/$FAILTYPE/task-filter
 224echo 10 > /debug/$FAILTYPE/probability
 225echo 100 > /debug/$FAILTYPE/interval
 226echo -1 > /debug/$FAILTYPE/times
 227echo 0 > /debug/$FAILTYPE/space
 228echo 2 > /debug/$FAILTYPE/verbose
 229echo 1 > /debug/$FAILTYPE/ignore-gfp-wait
 230echo 1 > /debug/$FAILTYPE/ignore-gfp-highmem
 231echo 10 > /debug/$FAILTYPE/stacktrace-depth
 232
 233trap "echo 0 > /debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
 234
 235echo "Injecting errors into the module $module... (interrupt to stop)"
 236sleep 1000000
 237
 238
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.