2Firmware-Assisted Dump
   5July 2011
   7The goal of firmware-assisted dump is to enable the dump of
   8a crashed system, and to do so from a fully-reset system, and
   9to minimize the total elapsed time until the system is back
  10in production use.
  12- Firmware-Assisted Dump (FADump) infrastructure is intended to replace
  13  the existing phyp assisted dump.
  14- Fadump uses the same firmware interfaces and memory reservation model
  15  as phyp assisted dump.
  16- Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
  17  in the ELF format in the same way as kdump. This helps us reuse the
  18  kdump infrastructure for dump capture and filtering.
  19- Unlike phyp dump, userspace tool does not need to refer any sysfs
  20  interface while reading /proc/vmcore.
  21- Unlike phyp dump, FADump allows user to release all the memory reserved
  22  for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
  23- Once enabled through kernel boot parameter, FADump can be
  24  started/stopped through /sys/kernel/fadump_registered interface (see
  25  sysfs files section below) and can be easily integrated with kdump
  26  service start/stop init scripts.
  28Comparing with kdump or other strategies, firmware-assisted
  29dump offers several strong, practical advantages:
  31-  Unlike kdump, the system has been reset, and loaded
  32   with a fresh copy of the kernel.  In particular,
  33   PCI and I/O devices have been reinitialized and are
  34   in a clean, consistent state.
  35-  Once the dump is copied out, the memory that held the dump
  36   is immediately available to the running kernel. And therefore,
  37   unlike kdump, FADump doesn't need a 2nd reboot to get back
  38   the system to the production configuration.
  40The above can only be accomplished by coordination with,
  41and assistance from the Power firmware. The procedure is
  42as follows:
  44-  The first kernel registers the sections of memory with the
  45   Power firmware for dump preservation during OS initialization.
  46   These registered sections of memory are reserved by the first
  47   kernel during early boot.
  49-  When system crashes, the Power firmware will copy the registered
  50   low memory regions (boot memory) from source to destination area.
  51   It will also save hardware PTE's.
  53   NOTE:
  54         The term 'boot memory' means size of the low memory chunk
  55         that is required for a kernel to boot successfully when
  56         booted with restricted memory. By default, the boot memory
  57         size will be the larger of 5% of system RAM or 256MB.
  58         Alternatively, user can also specify boot memory size
  59         through boot parameter 'crashkernel=' which will override
  60         the default calculated size. Use this option if default
  61         boot memory size is not sufficient for second kernel to
  62         boot successfully. For syntax of crashkernel= parameter,
  63         refer to Documentation/admin-guide/kdump/kdump.rst. If any
  64         offset is provided in crashkernel= parameter, it will be
  65         ignored as FADump uses a predefined offset to reserve memory
  66         for boot memory dump preservation in case of a crash.
  68-  After the low memory (boot memory) area has been saved, the
  69   firmware will reset PCI and other hardware state.  It will
  70   *not* clear the RAM. It will then launch the bootloader, as
  71   normal.
  73-  The freshly booted kernel will notice that there is a new node
  74   (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
  75   on OPAL platform) in the device tree, indicating that
  76   there is crash data available from a previous boot. During
  77   the early boot OS will reserve rest of the memory above
  78   boot memory size effectively booting with restricted memory
  79   size. This will make sure that this kernel (also, referred
  80   to as second kernel or capture kernel) will not touch any
  81   of the dump memory area.
  83-  User-space tools will read /proc/vmcore to obtain the contents
  84   of memory, which holds the previous crashed kernel dump in ELF
  85   format. The userspace tools may copy this info to disk, or
  86   network, nas, san, iscsi, etc. as desired.
  88-  Once the userspace tool is done saving dump, it will echo
  89   '1' to /sys/kernel/fadump_release_mem to release the reserved
  90   memory back to general use, except the memory required for
  91   next firmware-assisted dump registration.
  93   e.g.::
  95     # echo 1 > /sys/kernel/fadump_release_mem
  97Please note that the firmware-assisted dump feature
  98is only available on POWER6 and above systems on pSeries
  99(PowerVM) platform and POWER9 and above systems with OP940
 100or later firmware versions on PowerNV (OPAL) platform.
 101Note that, OPAL firmware exports ibm,opal/dump node when
 102FADump is supported on PowerNV platform.
 104On OPAL based machines, system first boots into an intermittent
 105kernel (referred to as petitboot kernel) before booting into the
 106capture kernel. This kernel would have minimal kernel and/or
 107userspace support to process crash data. Such kernel needs to
 108preserve previously crash'ed kernel's memory for the subsequent
 109capture kernel boot to process this crash data. Kernel config
 110option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
 111to ensure that crash data is preserved to process later.
 113-- On OPAL based machines (PowerNV), if the kernel is build with
 114   CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
 115   exported as /sys/firmware/opal/mpipl/core file. This procfs file is
 116   helpful in debugging OPAL crashes with GDB. The kernel memory
 117   used for exporting this procfs file can be released by echo'ing
 118   '1' to /sys/firmware/opal/mpipl/release_core node.
 120   e.g.
 121     # echo 1 > /sys/firmware/opal/mpipl/release_core
 123Implementation details:
 126During boot, a check is made to see if firmware supports
 127this feature on that particular machine. If it does, then
 128we check to see if an active dump is waiting for us. If yes
 129then everything but boot memory size of RAM is reserved during
 130early boot (See Fig. 2). This area is released once we finish
 131collecting the dump from user land scripts (e.g. kdump scripts)
 132that are run. If there is dump data, then the
 133/sys/kernel/fadump_release_mem file is created, and the reserved
 134memory is held.
 136If there is no waiting dump data, then only the memory required to
 137hold CPU state, HPTE region, boot memory dump, FADump header and
 138elfcore header, is usually reserved at an offset greater than boot
 139memory size (see Fig. 1). This area is *not* released: this region
 140will be kept permanently reserved, so that it can act as a receptacle
 141for a copy of the boot memory content in addition to CPU state and
 142HPTE region, in the case a crash does occur.
 144Since this reserved memory area is used only after the system crash,
 145there is no point in blocking this significant chunk of memory from
 146production kernel. Hence, the implementation uses the Linux kernel's
 147Contiguous Memory Allocator (CMA) for memory reservation if CMA is
 148configured for kernel. With CMA reservation this memory will be
 149available for applications to use it, while kernel is prevented from
 150using it. With this FADump will still be able to capture all of the
 151kernel memory and most of the user space memory except the user pages
 152that were present in CMA region::
 154  o Memory Reservation during first kernel
 156  Low memory                                                 Top of memory
 157  0    boot memory size   |<--- Reserved dump area --->|       |
 158  |           |           |    Permanent Reservation   |       |
 159  V           V           |                            |       V
 160  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
 161  |           |           |///|////|  DUMP | HDR | ELF |////|  |
 162  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
 163        |                   ^    ^     ^      ^           ^
 164        |                   |    |     |      |           |
 165        \                  CPU  HPTE   /      |           |
 166         ------------------------------       |           |
 167      Boot memory content gets transferred    |           |
 168      to reserved area by firmware at the     |           |
 169      time of crash.                          |           |
 170                                          FADump Header   |
 171                                           (meta area)    |
 172                                                          |
 173                                                          |
 174                      Metadata: This area holds a metadata structure whose
 175                      address is registered with f/w and retrieved in the
 176                      second kernel after crash, on platforms that support
 177                      tags (OPAL). Having such structure with info needed
 178                      to process the crashdump eases dump capture process.
 180                   Fig. 1
 183  o Memory Reservation during second kernel after crash
 185  Low memory                                              Top of memory
 186  0      boot memory size                                      |
 187  |           |<------------ Crash preserved area ------------>|
 188  V           V           |<--- Reserved dump area --->|       |
 189  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
 190  |           |           |///|////|  DUMP | HDR | ELF |////|  |
 191  +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
 192        |                                           |
 193        V                                           V
 194   Used by second                             /proc/vmcore
 195   kernel to boot
 197        +---+
 198        |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
 199        +---+    figures are not always present. For example, OPAL platform
 200                 does not have CPU & HPTE regions while Metadata region is
 201                 not supported on pSeries currently.
 203                   Fig. 2
 206Currently the dump will be copied from /proc/vmcore to a new file upon
 207user intervention. The dump data available through /proc/vmcore will be
 208in ELF format. Hence the existing kdump infrastructure (kdump scripts)
 209to save the dump works fine with minor modifications. KDump scripts on
 210major Distro releases have already been modified to work seamlessly (no
 211user intervention in saving the dump) when FADump is used, instead of
 212KDump, as dump mechanism.
 214The tools to examine the dump will be same as the ones
 215used for kdump.
 217How to enable firmware-assisted dump (FADump):
 2201. Set config option CONFIG_FA_DUMP=y and build kernel.
 2212. Boot into linux kernel with 'fadump=on' kernel cmdline option.
 222   By default, FADump reserved memory will be initialized as CMA area.
 223   Alternatively, user can boot linux kernel with 'fadump=nocma' to
 224   prevent FADump to use CMA.
 2253. Optionally, user can also set 'crashkernel=' kernel cmdline
 226   to specify size of the memory to reserve for boot memory dump
 227   preservation.
 230     1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
 231        use 'crashkernel=' to specify size of the memory to reserve
 232        for boot memory dump preservation.
 233     2. If firmware-assisted dump fails to reserve memory then it
 234        will fallback to existing kdump mechanism if 'crashkernel='
 235        option is set at kernel cmdline.
 236     3. if user wants to capture all of user space memory and ok with
 237        reserved memory not available to production system, then
 238        'fadump=nocma' kernel parameter can be used to fallback to
 239        old behaviour.
 241Sysfs/debugfs files:
 244Firmware-assisted dump feature uses sysfs file system to hold
 245the control files and debugfs file to display memory reserved region.
 247Here is the list of files under kernel sysfs:
 249 /sys/kernel/fadump_enabled
 250    This is used to display the FADump status.
 252    - 0 = FADump is disabled
 253    - 1 = FADump is enabled
 255    This interface can be used by kdump init scripts to identify if
 256    FADump is enabled in the kernel and act accordingly.
 258 /sys/kernel/fadump_registered
 259    This is used to display the FADump registration status as well
 260    as to control (start/stop) the FADump registration.
 262    - 0 = FADump is not registered.
 263    - 1 = FADump is registered and ready to handle system crash.
 265    To register FADump echo 1 > /sys/kernel/fadump_registered and
 266    echo 0 > /sys/kernel/fadump_registered for un-register and stop the
 267    FADump. Once the FADump is un-registered, the system crash will not
 268    be handled and vmcore will not be captured. This interface can be
 269    easily integrated with kdump service start/stop.
 271 /sys/kernel/fadump/mem_reserved
 273   This is used to display the memory reserved by FADump for saving the
 274   crash dump.
 276 /sys/kernel/fadump_release_mem
 277    This file is available only when FADump is active during
 278    second kernel. This is used to release the reserved memory
 279    region that are held for saving crash dump. To release the
 280    reserved memory echo 1 to it::
 282        echo 1  > /sys/kernel/fadump_release_mem
 284    After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
 285    file will change to reflect the new memory reservations.
 287    The existing userspace tools (kdump infrastructure) can be easily
 288    enhanced to use this interface to release the memory reserved for
 289    dump and continue without 2nd reboot.
 291Note: /sys/kernel/fadump_release_opalcore sysfs has moved to
 292      /sys/firmware/opal/mpipl/release_core
 294 /sys/firmware/opal/mpipl/release_core
 296    This file is available only on OPAL based machines when FADump is
 297    active during capture kernel. This is used to release the memory
 298    used by the kernel to export /sys/firmware/opal/mpipl/core file. To
 299    release this memory, echo '1' to it:
 301    echo 1  > /sys/firmware/opal/mpipl/release_core
 303Note: The following FADump sysfs files are deprecated.
 306| Deprecated                       | Alternative                    |
 308| /sys/kernel/fadump_enabled       | /sys/kernel/fadump/enabled     |
 310| /sys/kernel/fadump_registered    | /sys/kernel/fadump/registered  |
 312| /sys/kernel/fadump_release_mem   | /sys/kernel/fadump/release_mem |
 315Here is the list of files under powerpc debugfs:
 316(Assuming debugfs is mounted on /sys/kernel/debug directory.)
 318 /sys/kernel/debug/powerpc/fadump_region
 319    This file shows the reserved memory regions if FADump is
 320    enabled otherwise this file is empty. The output format
 321    is::
 323      <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
 325    and for kernel DUMP region is:
 327    DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes
 329    e.g.
 330    Contents when FADump is registered during first kernel::
 332      # cat /sys/kernel/debug/powerpc/fadump_region
 333      CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
 334      HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
 335      DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
 337    Contents when FADump is active during second kernel::
 339      # cat /sys/kernel/debug/powerpc/fadump_region
 340      CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
 341      HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
 342      DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
 343          : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
 347      Please refer to Documentation/filesystems/debugfs.rst on
 348      how to mount the debugfs filesystem.
 353 - Need to come up with the better approach to find out more
 354   accurate boot memory size that is required for a kernel to
 355   boot successfully when booted with restricted memory.
 356 - The FADump implementation introduces a FADump crash info structure
 357   in the scratch area before the ELF core header. The idea of introducing
 358   this structure is to pass some important crash info data to the second
 359   kernel which will help second kernel to populate ELF core header with
 360   correct data before it gets exported through /proc/vmcore. The current
 361   design implementation does not address a possibility of introducing
 362   additional fields (in future) to this structure without affecting
 363   compatibility. Need to come up with the better approach to address this.
 365   The possible approaches are:
 367        1. Introduce version field for version tracking, bump up the version
 368        whenever a new field is added to the structure in future. The version
 369        field can be used to find out what fields are valid for the current
 370        version of the structure.
 371        2. Reserve the area of predefined size (say PAGE_SIZE) for this
 372        structure and have unused area as reserved (initialized to zero)
 373        for future field additions.
 375   The advantage of approach 1 over 2 is we don't need to reserve extra space.
 377Author: Mahesh Salgaonkar <>
 379This document is based on the original documentation written for phyp
 381assisted dump by Linas Vepstas and Manish Ahuja.