1Frontswap provides a "transcendent memory" interface for swap pages.
   2In some environments, dramatic performance savings may be obtained because
   3swapped pages are saved in RAM (or a RAM-like device) instead of a swap disk.
   5(Note, frontswap -- and cleancache (merged at 3.0) -- are the "frontends"
   6and the only necessary changes to the core kernel for transcendent memory;
   7all other supporting code -- the "backends" -- is implemented as drivers.
   8See the article "Transcendent memory in a nutshell" for a detailed
   9overview of frontswap and related kernel parts:
  10 )
  12Frontswap is so named because it can be thought of as the opposite of
  13a "backing" store for a swap device.  The storage is assumed to be
  14a synchronous concurrency-safe page-oriented "pseudo-RAM device" conforming
  15to the requirements of transcendent memory (such as Xen's "tmem", or
  16in-kernel compressed memory, aka "zcache", or future RAM-like devices);
  17this pseudo-RAM device is not directly accessible or addressable by the
  18kernel and is of unknown and possibly time-varying size.  The driver
  19links itself to frontswap by calling frontswap_register_ops to set the
  20frontswap_ops funcs appropriately and the functions it provides must
  21conform to certain policies as follows:
  23An "init" prepares the device to receive frontswap pages associated
  24with the specified swap device number (aka "type").  A "store" will
  25copy the page to transcendent memory and associate it with the type and
  26offset associated with the page. A "load" will copy the page, if found,
  27from transcendent memory into kernel memory, but will NOT remove the page
  28from transcendent memory.  An "invalidate_page" will remove the page
  29from transcendent memory and an "invalidate_area" will remove ALL pages
  30associated with the swap type (e.g., like swapoff) and notify the "device"
  31to refuse further stores with that swap type.
  33Once a page is successfully stored, a matching load on the page will normally
  34succeed.  So when the kernel finds itself in a situation where it needs
  35to swap out a page, it first attempts to use frontswap.  If the store returns
  36success, the data has been successfully saved to transcendent memory and
  37a disk write and, if the data is later read back, a disk read are avoided.
  38If a store returns failure, transcendent memory has rejected the data, and the
  39page can be written to swap as usual.
  41If a backend chooses, frontswap can be configured as a "writethrough
  42cache" by calling frontswap_writethrough().  In this mode, the reduction
  43in swap device writes is lost (and also a non-trivial performance advantage)
  44in order to allow the backend to arbitrarily "reclaim" space used to
  45store frontswap pages to more completely manage its memory usage.
  47Note that if a page is stored and the page already exists in transcendent memory
  48(a "duplicate" store), either the store succeeds and the data is overwritten,
  49or the store fails AND the page is invalidated.  This ensures stale data may
  50never be obtained from frontswap.
  52If properly configured, monitoring of frontswap is done via debugfs in
  53the /sys/kernel/debug/frontswap directory.  The effectiveness of
  54frontswap can be measured (across all swap devices) with:
  56failed_stores   - how many store attempts have failed
  57loads           - how many loads were attempted (all should succeed)
  58succ_stores     - how many store attempts have succeeded
  59invalidates     - how many invalidates were attempted
  61A backend implementation may provide additional metrics.
  651) Where's the value?
  67When a workload starts swapping, performance falls through the floor.
  68Frontswap significantly increases performance in many such workloads by
  69providing a clean, dynamic interface to read and write swap pages to
  70"transcendent memory" that is otherwise not directly addressable to the kernel.
  71This interface is ideal when data is transformed to a different form
  72and size (such as with compression) or secretly moved (as might be
  73useful for write-balancing for some RAM-like devices).  Swap pages (and
  74evicted page-cache pages) are a great use for this kind of slower-than-RAM-
  75but-much-faster-than-disk "pseudo-RAM device" and the frontswap (and
  76cleancache) interface to transcendent memory provides a nice way to read
  77and write -- and indirectly "name" -- the pages.
  79Frontswap -- and cleancache -- with a fairly small impact on the kernel,
  80provides a huge amount of flexibility for more dynamic, flexible RAM
  81utilization in various system configurations:
  83In the single kernel case, aka "zcache", pages are compressed and
  84stored in local memory, thus increasing the total anonymous pages
  85that can be safely kept in RAM.  Zcache essentially trades off CPU
  86cycles used in compression/decompression for better memory utilization.
  87Benchmarks have shown little or no impact when memory pressure is
  88low while providing a significant performance improvement (25%+)
  89on some workloads under high memory pressure.
  91"RAMster" builds on zcache by adding "peer-to-peer" transcendent memory
  92support for clustered systems.  Frontswap pages are locally compressed
  93as in zcache, but then "remotified" to another system's RAM.  This
  94allows RAM to be dynamically load-balanced back-and-forth as needed,
  95i.e. when system A is overcommitted, it can swap to system B, and
  96vice versa.  RAMster can also be configured as a memory server so
  97many servers in a cluster can swap, dynamically as needed, to a single
  98server configured with a large amount of RAM... without pre-configuring
  99how much of the RAM is available for each of the clients!
 101In the virtual case, the whole point of virtualization is to statistically
 102multiplex physical resources across the varying demands of multiple
 103virtual machines.  This is really hard to do with RAM and efforts to do
 104it well with no kernel changes have essentially failed (except in some
 105well-publicized special-case workloads).
 106Specifically, the Xen Transcendent Memory backend allows otherwise
 107"fallow" hypervisor-owned RAM to not only be "time-shared" between multiple
 108virtual machines, but the pages can be compressed and deduplicated to
 109optimize RAM utilization.  And when guest OS's are induced to surrender
 110underutilized RAM (e.g. with "selfballooning"), sudden unexpected
 111memory pressure may result in swapping; frontswap allows those pages
 112to be swapped to and from hypervisor RAM (if overall host system memory
 113conditions allow), thus mitigating the potentially awful performance impact
 114of unplanned swapping.
 116A KVM implementation is underway and has been RFC'ed to lkml.  And,
 117using frontswap, investigation is also underway on the use of NVM as
 118a memory extension technology.
 1202) Sure there may be performance advantages in some situations, but
 121   what's the space/time overhead of frontswap?
 123If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles into
 124nothingness and the only overhead is a few extra bytes per swapon'ed
 125swap device.  If CONFIG_FRONTSWAP is enabled but no frontswap "backend"
 126registers, there is one extra global variable compared to zero for
 127every swap page read or written.  If CONFIG_FRONTSWAP is enabled
 128AND a frontswap backend registers AND the backend fails every "store"
 129request (i.e. provides no memory despite claiming it might),
 130CPU overhead is still negligible -- and since every frontswap fail
 131precedes a swap page write-to-disk, the system is highly likely
 132to be I/O bound and using a small fraction of a percent of a CPU
 133will be irrelevant anyway.
 135As for space, if CONFIG_FRONTSWAP is enabled AND a frontswap backend
 136registers, one bit is allocated for every swap page for every swap
 137device that is swapon'd.  This is added to the EIGHT bits (which
 138was sixteen until about 2.6.34) that the kernel already allocates
 139for every swap page for every swap device that is swapon'd.  (Hugh
 140Dickins has observed that frontswap could probably steal one of
 141the existing eight bits, but let's worry about that minor optimization
 142later.)  For very large swap disks (which are rare) on a standard
 1434K pagesize, this is 1MB per 32GB swap.
 145When swap pages are stored in transcendent memory instead of written
 146out to disk, there is a side effect that this may create more memory
 147pressure that can potentially outweigh the other advantages.  A
 148backend, such as zcache, must implement policies to carefully (but
 149dynamically) manage memory limits to ensure this doesn't happen.
 1513) OK, how about a quick overview of what this frontswap patch does
 152   in terms that a kernel hacker can grok?
 154Let's assume that a frontswap "backend" has registered during
 155kernel initialization; this registration indicates that this
 156frontswap backend has access to some "memory" that is not directly
 157accessible by the kernel.  Exactly how much memory it provides is
 158entirely dynamic and random.
 160Whenever a swap-device is swapon'd frontswap_init() is called,
 161passing the swap device number (aka "type") as a parameter.
 162This notifies frontswap to expect attempts to "store" swap pages
 163associated with that number.
 165Whenever the swap subsystem is readying a page to write to a swap
 166device (c.f swap_writepage()), frontswap_store is called.  Frontswap
 167consults with the frontswap backend and if the backend says it does NOT
 168have room, frontswap_store returns -1 and the kernel swaps the page
 169to the swap device as normal.  Note that the response from the frontswap
 170backend is unpredictable to the kernel; it may choose to never accept a
 171page, it could accept every ninth page, or it might accept every
 172page.  But if the backend does accept a page, the data from the page
 173has already been copied and associated with the type and offset,
 174and the backend guarantees the persistence of the data.  In this case,
 175frontswap sets a bit in the "frontswap_map" for the swap device
 176corresponding to the page offset on the swap device to which it would
 177otherwise have written the data.
 179When the swap subsystem needs to swap-in a page (swap_readpage()),
 180it first calls frontswap_load() which checks the frontswap_map to
 181see if the page was earlier accepted by the frontswap backend.  If
 182it was, the page of data is filled from the frontswap backend and
 183the swap-in is complete.  If not, the normal swap-in code is
 184executed to obtain the page of data from the real swap device.
 186So every time the frontswap backend accepts a page, a swap device read
 187and (potentially) a swap device write are replaced by a "frontswap backend
 188store" and (possibly) a "frontswap backend loads", which are presumably much
 1914) Can't frontswap be configured as a "special" swap device that is
 192   just higher priority than any real swap device (e.g. like zswap,
 193   or maybe swap-over-nbd/NFS)?
 195No.  First, the existing swap subsystem doesn't allow for any kind of
 196swap hierarchy.  Perhaps it could be rewritten to accommodate a hierarchy,
 197but this would require fairly drastic changes.  Even if it were
 198rewritten, the existing swap subsystem uses the block I/O layer which
 199assumes a swap device is fixed size and any page in it is linearly
 200addressable.  Frontswap barely touches the existing swap subsystem,
 201and works around the constraints of the block I/O subsystem to provide
 202a great deal of flexibility and dynamicity.
 204For example, the acceptance of any swap page by the frontswap backend is
 205entirely unpredictable. This is critical to the definition of frontswap
 206backends because it grants completely dynamic discretion to the
 207backend.  In zcache, one cannot know a priori how compressible a page is.
 208"Poorly" compressible pages can be rejected, and "poorly" can itself be
 209defined dynamically depending on current memory constraints.
 211Further, frontswap is entirely synchronous whereas a real swap
 212device is, by definition, asynchronous and uses block I/O.  The
 213block I/O layer is not only unnecessary, but may perform "optimizations"
 214that are inappropriate for a RAM-oriented device including delaying
 215the write of some pages for a significant amount of time.  Synchrony is
 216required to ensure the dynamicity of the backend and to avoid thorny race
 217conditions that would unnecessarily and greatly complicate frontswap
 218and/or the block I/O subsystem.  That said, only the initial "store"
 219and "load" operations need be synchronous.  A separate asynchronous thread
 220is free to manipulate the pages stored by frontswap.  For example,
 221the "remotification" thread in RAMster uses standard asynchronous
 222kernel sockets to move compressed frontswap pages to a remote machine.
 223Similarly, a KVM guest-side implementation could do in-guest compression
 224and use "batched" hypercalls.
 226In a virtualized environment, the dynamicity allows the hypervisor
 227(or host OS) to do "intelligent overcommit".  For example, it can
 228choose to accept pages only until host-swapping might be imminent,
 229then force guests to do their own swapping.
 231There is a downside to the transcendent memory specifications for
 232frontswap:  Since any "store" might fail, there must always be a real
 233slot on a real swap device to swap the page.  Thus frontswap must be
 234implemented as a "shadow" to every swapon'd device with the potential
 235capability of holding every page that the swap device might have held
 236and the possibility that it might hold no pages at all.  This means
 237that frontswap cannot contain more pages than the total of swapon'd
 238swap devices.  For example, if NO swap device is configured on some
 239installation, frontswap is useless.  Swapless portable devices
 240can still use frontswap but a backend for such devices must configure
 241some kind of "ghost" swap device and ensure that it is never used.
 2435) Why this weird definition about "duplicate stores"?  If a page
 244   has been previously successfully stored, can't it always be
 245   successfully overwritten?
 247Nearly always it can, but no, sometimes it cannot.  Consider an example
 248where data is compressed and the original 4K page has been compressed
 249to 1K.  Now an attempt is made to overwrite the page with data that
 250is non-compressible and so would take the entire 4K.  But the backend
 251has no more space.  In this case, the store must be rejected.  Whenever
 252frontswap rejects a store that would overwrite, it also must invalidate
 253the old data and ensure that it is no longer accessible.  Since the
 254swap subsystem then writes the new data to the read swap device,
 255this is the correct course of action to ensure coherency.
 2576) What is frontswap_shrink for?
 259When the (non-frontswap) swap subsystem swaps out a page to a real
 260swap device, that page is only taking up low-value pre-allocated disk
 261space.  But if frontswap has placed a page in transcendent memory, that
 262page may be taking up valuable real estate.  The frontswap_shrink
 263routine allows code outside of the swap subsystem to force pages out
 264of the memory managed by frontswap and back into kernel-addressable memory.
 265For example, in RAMster, a "suction driver" thread will attempt
 266to "repatriate" pages sent to a remote machine back to the local machine;
 267this is driven using the frontswap_shrink mechanism when memory pressure
 2707) Why does the frontswap patch create the new include file swapfile.h?
 272The frontswap code depends on some swap-subsystem-internal data
 273structures that have, over the years, moved back and forth between
 274static and global.  This seemed a reasonable compromise:  Define
 275them as global but declare them in a new include file that isn't
 276included by the large number of source files that include swap.h.
 278Dan Magenheimer, last updated April 9, 2012