2What is Linux Memory Policy?
   4In the Linux kernel, "memory policy" determines from which node the kernel will
   5allocate memory in a NUMA system or in an emulated NUMA system.  Linux has
   6supported platforms with Non-Uniform Memory Access architectures since 2.4.?.
   7The current memory policy support was added to Linux 2.6 around May 2004.  This
   8document attempts to describe the concepts and APIs of the 2.6 memory policy
  11Memory policies should not be confused with cpusets
  13which is an administrative mechanism for restricting the nodes from which
  14memory may be allocated by a set of processes. Memory policies are a
  15programming interface that a NUMA-aware application can take advantage of.  When
  16both cpusets and policies are applied to a task, the restrictions of the cpuset
  17takes priority.  See "MEMORY POLICIES AND CPUSETS" below for more details.
  21Scope of Memory Policies
  23The Linux kernel supports _scopes_ of memory policy, described here from
  24most general to most specific:
  26    System Default Policy:  this policy is "hard coded" into the kernel.  It
  27    is the policy that governs all page allocations that aren't controlled
  28    by one of the more specific policy scopes discussed below.  When the
  29    system is "up and running", the system default policy will use "local
  30    allocation" described below.  However, during boot up, the system
  31    default policy will be set to interleave allocations across all nodes
  32    with "sufficient" memory, so as not to overload the initial boot node
  33    with boot-time allocations.
  35    Task/Process Policy:  this is an optional, per-task policy.  When defined
  36    for a specific task, this policy controls all page allocations made by or
  37    on behalf of the task that aren't controlled by a more specific scope.
  38    If a task does not define a task policy, then all page allocations that
  39    would have been controlled by the task policy "fall back" to the System
  40    Default Policy.
  42        The task policy applies to the entire address space of a task. Thus,
  43        it is inheritable, and indeed is inherited, across both fork()
  44        [clone() w/o the CLONE_VM flag] and exec*().  This allows a parent task
  45        to establish the task policy for a child task exec()'d from an
  46        executable image that has no awareness of memory policy.  See the
  47        MEMORY POLICY APIS section, below, for an overview of the system call
  48        that a task may use to set/change its task/process policy.
  50        In a multi-threaded task, task policies apply only to the thread
  51        [Linux kernel task] that installs the policy and any threads
  52        subsequently created by that thread.  Any sibling threads existing
  53        at the time a new task policy is installed retain their current
  54        policy.
  56        A task policy applies only to pages allocated after the policy is
  57        installed.  Any pages already faulted in by the task when the task
  58        changes its task policy remain where they were allocated based on
  59        the policy at the time they were allocated.
  61    VMA Policy:  A "VMA" or "Virtual Memory Area" refers to a range of a task's
  62    virtual address space.  A task may define a specific policy for a range
  63    of its virtual address space.   See the MEMORY POLICIES APIS section,
  64    below, for an overview of the mbind() system call used to set a VMA
  65    policy.
  67    A VMA policy will govern the allocation of pages that back this region of
  68    the address space.  Any regions of the task's address space that don't
  69    have an explicit VMA policy will fall back to the task policy, which may
  70    itself fall back to the System Default Policy.
  72    VMA policies have a few complicating details:
  74        VMA policy applies ONLY to anonymous pages.  These include pages
  75        allocated for anonymous segments, such as the task stack and heap, and
  76        any regions of the address space mmap()ed with the MAP_ANONYMOUS flag.
  77        If a VMA policy is applied to a file mapping, it will be ignored if
  78        the mapping used the MAP_SHARED flag.  If the file mapping used the
  79        MAP_PRIVATE flag, the VMA policy will only be applied when an
  80        anonymous page is allocated on an attempt to write to the mapping--
  81        i.e., at Copy-On-Write.
  83        VMA policies are shared between all tasks that share a virtual address
  84        space--a.k.a. threads--independent of when the policy is installed; and
  85        they are inherited across fork().  However, because VMA policies refer
  86        to a specific region of a task's address space, and because the address
  87        space is discarded and recreated on exec*(), VMA policies are NOT
  88        inheritable across exec().  Thus, only NUMA-aware applications may
  89        use VMA policies.
  91        A task may install a new VMA policy on a sub-range of a previously
  92        mmap()ed region.  When this happens, Linux splits the existing virtual
  93        memory area into 2 or 3 VMAs, each with it's own policy.
  95        By default, VMA policy applies only to pages allocated after the policy
  96        is installed.  Any pages already faulted into the VMA range remain
  97        where they were allocated based on the policy at the time they were
  98        allocated.  However, since 2.6.16, Linux supports page migration via
  99        the mbind() system call, so that page contents can be moved to match
 100        a newly installed policy.
 102    Shared Policy:  Conceptually, shared policies apply to "memory objects"
 103    mapped shared into one or more tasks' distinct address spaces.  An
 104    application installs a shared policies the same way as VMA policies--using
 105    the mbind() system call specifying a range of virtual addresses that map
 106    the shared object.  However, unlike VMA policies, which can be considered
 107    to be an attribute of a range of a task's address space, shared policies
 108    apply directly to the shared object.  Thus, all tasks that attach to the
 109    object share the policy, and all pages allocated for the shared object,
 110    by any task, will obey the shared policy.
 112        As of 2.6.22, only shared memory segments, created by shmget() or
 113        mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy.  When shared
 114        policy support was added to Linux, the associated data structures were
 115        added to hugetlbfs shmem segments.  At the time, hugetlbfs did not
 116        support allocation at fault time--a.k.a lazy allocation--so hugetlbfs
 117        shmem segments were never "hooked up" to the shared policy support.
 118        Although hugetlbfs segments now support lazy allocation, their support
 119        for shared policy has not been completed.
 121        As mentioned above [re: VMA policies], allocations of page cache
 122        pages for regular files mmap()ed with MAP_SHARED ignore any VMA
 123        policy installed on the virtual address range backed by the shared
 124        file mapping.  Rather, shared page cache pages, including pages backing
 125        private mappings that have not yet been written by the task, follow
 126        task policy, if any, else System Default Policy.
 128        The shared policy infrastructure supports different policies on subset
 129        ranges of the shared object.  However, Linux still splits the VMA of
 130        the task that installs the policy for each range of distinct policy.
 131        Thus, different tasks that attach to a shared memory segment can have
 132        different VMA configurations mapping that one shared object.  This
 133        can be seen by examining the /proc/<pid>/numa_maps of tasks sharing
 134        a shared memory region, when one task has installed shared policy on
 135        one or more ranges of the region.
 137Components of Memory Policies
 139    A Linux memory policy consists of a "mode", optional mode flags, and an
 140    optional set of nodes.  The mode determines the behavior of the policy,
 141    the optional mode flags determine the behavior of the mode, and the
 142    optional set of nodes can be viewed as the arguments to the policy
 143    behavior.
 145   Internally, memory policies are implemented by a reference counted
 146   structure, struct mempolicy.  Details of this structure will be discussed
 147   in context, below, as required to explain the behavior.
 149   Linux memory policy supports the following 4 behavioral modes:
 151        Default Mode--MPOL_DEFAULT:  This mode is only used in the memory
 152        policy APIs.  Internally, MPOL_DEFAULT is converted to the NULL
 153        memory policy in all policy scopes.  Any existing non-default policy
 154        will simply be removed when MPOL_DEFAULT is specified.  As a result,
 155        MPOL_DEFAULT means "fall back to the next most specific policy scope."
 157            For example, a NULL or default task policy will fall back to the
 158            system default policy.  A NULL or default vma policy will fall
 159            back to the task policy.
 161            When specified in one of the memory policy APIs, the Default mode
 162            does not use the optional set of nodes.
 164            It is an error for the set of nodes specified for this policy to
 165            be non-empty.
 167        MPOL_BIND:  This mode specifies that memory must come from the
 168        set of nodes specified by the policy.  Memory will be allocated from
 169        the node in the set with sufficient free memory that is closest to
 170        the node where the allocation takes place.
 172        MPOL_PREFERRED:  This mode specifies that the allocation should be
 173        attempted from the single node specified in the policy.  If that
 174        allocation fails, the kernel will search other nodes, in order of
 175        increasing distance from the preferred node based on information
 176        provided by the platform firmware.
 177        containing the cpu where the allocation takes place.
 179            Internally, the Preferred policy uses a single node--the
 180            preferred_node member of struct mempolicy.  When the internal
 181            mode flag MPOL_F_LOCAL is set, the preferred_node is ignored and
 182            the policy is interpreted as local allocation.  "Local" allocation
 183            policy can be viewed as a Preferred policy that starts at the node
 184            containing the cpu where the allocation takes place.
 186            It is possible for the user to specify that local allocation is
 187            always preferred by passing an empty nodemask with this mode.
 188            If an empty nodemask is passed, the policy cannot use the
 189            MPOL_F_STATIC_NODES or MPOL_F_RELATIVE_NODES flags described
 190            below.
 192        MPOL_INTERLEAVED:  This mode specifies that page allocations be
 193        interleaved, on a page granularity, across the nodes specified in
 194        the policy.  This mode also behaves slightly differently, based on
 195        the context where it is used:
 197            For allocation of anonymous pages and shared memory pages,
 184txt#L49"" iulULT meau         name="L146kvm/"line" ne.a="liney_policy.txt#L49atiotion/l327ine" 2mm/numa_meory_policy.txt#L101" id=2L101"20lass="line" name=    set of nodesIferren name="L8">  s="line" /a>   , as a icy.t="line" ne.a="liney_policy.txt#L49atiotion/l327ine" 2mm/numa_meory_policy.txt#L102" id=2L102"20lass="line" name="L18 >
  vg dista147L19had mappi">
  vg duma  e policy is interprTth sufges and shared memory pages,
 197NULL oemory  set  the ke zonelisto the shared policy support.
 197   e mapping.  Rathercy.txt#L146" id="L146" class="l="line" name="L25">  25
 c"line" ne.a="liney_policy.txt#L49atiotion/l327ine" 2mm/numa_meory_policy.txt#L108" id=2L108"20 the 19emory_poli  th entir194  rat i" classh  Teot;fall bag distair194  6 a page granularity, uma_memory_
     ill sehangeer, name="L85he policy, r/a>  EAVEnnce from tan   struescrib"L158">ed belowa href="Documentation/vm/numa_memref="Documentation2mm/numa_meory_policy.txt#L113" id=2L113"21lass="line" name=159t;up and running", orksm the cify that local allocation is
 149   Linux memory polame="L141"> 141st general to most specific:
 189    /a>     "> 18MPOL_PREFERRED:  Th/a>         If  um"line" name="L127"> 127
  11Memfaul namei37">  37  ibute /numa_mM flas "line" ne.a="liney_policy.txt#L49atiotion/l327ine" 2vm/numa_meemory_policy.txt#L20" id=L120"22lass="line" n142ame="L5es only tome="L149"> 149morymappial, perpty nodemask with this mode.
  5rred_node ationbt wask's a_mM"line" name="L191"> 191
      (B"L1ercy.txt#L14) ationul nameAULT meanws="linents, created by shmget() or
 194ayDEFAULT tnc142 127
      -fall bag distas"ap the addr"line" name="L127"> 127
 149cify that local allocation is
 232 233
 234 235ayDEFAULT tu where ht> 149ci 236 237 149ciul nameADefaul alloc, orksm th or &quonhref="Documentation/vm/numa_memory_policy.txt#L92" id2="L138" c2ass="line" name="L138"> 238      -fall occurscyy_
 149ciul nameAom tef="Documentation/vm/numa_memory_policy.txt#L92" id2==L129"22 ass="line" name="L139"> 239 240 241 242  5rred_RELATIVE_NODES flags desca>   ,ehave use thsidres wfhref="Documentation/2vm/numa_meemory_policy.txt#L24" i="L143" c2ass="line" name="L143"> 243 244 245 246         If  um"line" name="L127"> 127
 247 248
 249 250 251 252  5rred_node ationbt was (deedwnode at_STATIC_NODES or MP)k's a_mef="Documentation/2vm/numa_memmory_policy.txt#L22" i="L143" c2ass="line" name="L153"> 253 191
 254 255ayDEFAULT teated by shmget() or
 256 149ci/a>
 257 258 259 260 261ne.a="ls 262 149ci/a>
 263 149cef="Documentation/2vm/numa_memmory_policy.txt#L22" i==L124"22lass="line" name="L164"> 264 265 266 127
 267 268 269 149cisk is pass pagessuppdhr" namearede ad
 270n5t, the  href="Documentation/vm/numa_memory_policy.txt#L194" i2="L171" c2ass="line" name="L171"> 271 149cisk is pashe policy
 272 273 274 275
 277 149ci 278 149ci 279 280 281 282 283 284 285ayDEFAULT tninewitp 286 287 288 149cify that a map
 289 290 291 292  5rred_RELATI_NODES or MPOdesca>   ,ehave use thsidres wfhref="Documentation/2vm/numa_meemory_policy.txt#L24" i="L193" c2ass="line" name="L193"> 293 294 295 296 297 384deng ion/axt   rce counted
 y.  _pu
 3f nod3sIferrsaveare will bo the task picy.  Detmkml spe clawgln;gion.
e counted
 3vg di3ta147L39had 30 197<3a>NUL3 oemorhe iny inslocation is
 325 397 he iny po, orkfile m polame="L141">re will bucturoto onemousodesef="Documentation/2vm/numa_memmory_policy.txt#L22" 3=7L105"203lss="line" name="L197"> 3"L1463vm/numure, structousodesm.
e counteid to Li,e argume149"> 149ci.
e countee discussroma_mef="Documentation/2vm/numa_memmory_policy.txt#L22" 3=8L105"203lhe 19emory_policy.txt#L3r194<3aimpleoleted.
     i3l seh3ngeer,on;gion.
e counted
cntatbetwmplety tef="Documentation/2vm/numa_memmory_policy.txt#L22" 3=2L100"203ass="line" name=b/vm/nem3num i3="ame=nineNUMAabeRED fl or &qUsag" or &qut is " the memoryeneral toref="Documentation/vm/numa_me2ory_policy.txt#L117" i3=2L113"213ass="line" name=159 325 349 327 189<3a>   3/a>   2)ning thaa146" idlicy is introsine the behavi is intlso be3 11Memfaul hrnul n ame="35es o3ly tome="pathhooked flNotess clofL_F_RELA ThiDefaul allocusag" or &quextendss the nodes dw   tef="Documentation/vm/numa_memory_policy.txt#L92" id3d=L121"223ass="line" name="L191"> 391
< naef="Documentation/2vm/numa_memmory_policy.txt#L22" 3d="112"223ass="line" name=Wie="Lts3ng a 39     3(B"L13rcy.txWee vieavoid pla empty nxtrrence countedmentatnglead uma_3194 327   rneame s#Lriemora pagcrysaveaify thsedupbe 149ciolicy will f#Lrilicy willia str
 332 149cimm> 149cimmap_s thfL_Fanuladmentatnglequery flagelsr
_icy.  Det()a href="Documentation/vm/numa_memory_policy.txt#L182" i3="L133" c3ass="line" name="L133"> 333 334 335aya hrenine" nethanulaemore" ne. allocathileasksodesm149"s nethanula cef="Documentation/2vm/numa_memmory_policy.txt#L22" 3="L136" c3ass="line" name="L136"> 336 337 3383) Plocations be
 340 341 149reslicy.titSfmplllocations be
< ef="Documentation/vm/numa_memory_policy.txt#L193" i3=="112"223ass="line" name="L142"> 342 343 344 345 346re will baddspty nxtrrence countef="Documentation/2m/numa_me2ory_policy.txt#L114" i3="L137" c3ass="line" name="L147"> 347 348re will  flag.  ed to ee" namewussromrguibenxtrref="Documentation/2m/numa_me2ory_policy.txt#L114" i3=2L119"213ass="line" name="L149"> 349 149resfinis a f allocus1"  or &qulicy canno., Weeome fsromrgutef="Documentation/2m/numa_me2ory_policy.txt#L114" i3="L150" c3ass="line" name="L150"> 350 351 352 353 354 355ayB1
 356re will bode.resp stxtk,1spolicy on
< policy.locations be
 358 3"lined         osnently, baeNUMAabeRED flTuibenxtrrnyy_
hnula viewedavoidfges anerefeef="Documentation/vm/numa_memory_policy.txt#L92" id3="L160" c3ass="line" name="L160"> 360 361 1txtk   ef="Documentation/vm/numa_memory_policy.txt#L92" id3="L162" c3ass="line" name="L162"> 362  ibuteappromrdataafhrmck tations may
<.ef="Documentation/vm/numa_memory_policy.txt#L92" id3="L163" c3ass="line" name="L163"> 363 364 365 366 1413eaify thcck safhrmeau roha     pages,
alloc flage>
 367 149ci158">ed sp Anpohref="Documentation/2vm/numa_meemory_policy.txt#L24" 3="L168" c3ass="line" name="L168"> 368 149ci158">ed sp An.ef="Documentation/vm/numa_memory_policy.txt#L92" id3=119"213asass="line" name="L169"> 369 370
 371eithe see. lcklocas closes  ibpara href="Documentation/2v/numa_me2emory_policy.txt#L25" 3="L172" c3ass="line" name="L172"> 372 373 374 375 376 377 378 379 380 381 149ciicy.tsha     meA> 149ci alloc meA/lt;pess  pages,
alloc or &quloylso ef="Documentation/vm/numa_memory_policy.txt#L91" id3="L172" c3ass="line" name="L182"> 382 383 149nameA> 149 fl  149nameA> 149 po, o"line" biutnueA hrin the dsmeau       ef="Documentation/vm/numa_memory_policy.txt#L91" id3==L124"223ass="line" name="L184"> 384 149laxbeRE> 149  ds flOost specific:

 385ayDEFAUer) sy" nedes > 149lso > 149a 386 387 388dende   l ef="Documentation/2vm/numa_memmory_policy.txt#L22"d2="L189"3c2ass="line" name="L189"3 289<3836classef="Documentation/2vm/numa_memmory_policy.txt#L22"id="L190"3c2ass="line" name="L190"3 290<3a2


            Querior4,denpend     ef="Documentation/vm/numa_memory_policy.txt#L91" i i="L189"3c2s shmem"6" class="line3c2pb/3m/nem/numa_memkuma_lio > 1c:
 149adende   l ef="Documentation/2vm/numa_memmory_policy.txt#L224i3=2L100"403ass="line" name="L192"4 3vg 4034 194<3a>N4L3 oemIhled s ne./) SpoliPl allocafha Rrneamef T  meA> 149Ai158">eS
   e mappinghelotaf) syeavoi*olicyennnodsiga helotlen,k l,ioylsohref="Documentation/vm/numa_memory_policy.txt#L83" 4 3=7L105"403lss="line" name="L197"4 3"L146kvm/numa_memory_
eeeeeeeLLeredtbodsiga helotat*namennnodsiga helotatmaxbeohref="Documentation/vm/numa_memory_policy.txt#L83" 4 3=8L105"403lhe 19emory_policy.txt4L3r194  6 a f) syd(thled ssspolicy cannthe" ne.a="li(oylso t*namenatmaxbes): Vaaef="Documentation/vm/numa_memory_policy.txt#L164"4 3=1L100"403ass="line" name=a>    4i3l s4hangeer, name=a allocafha;gion.a &6" idlicytsha     meA> 149ci158">ed sp aaef="Documentation/vm/numa_memory_policy.txt#L164"4 3=2L100"403ass="line" name=b/vm/n4m3num413            the" ne.a="liney_po> 1olicyo > 149nd_po> 1leno > 149at;5 ED:auteque 1c:
 149adende   P).ef="Documentation/vm/numa_memory_policy.txt#L92" 4i3=6L105"403ass="line" name="L149"4 349<41>
 184<3a> 41263 114/3>MemAlwnodghses  strictly ibpara  do hrILemoi, a ation/vm/ame4"35es4o3ly taFAUems,a> .

   4 3(B"42353194<+icy
leteileun-;s libg&q

 149ciskia s 149cy
   href="Documentation/vm/numa_memory_policy.txt#L182"4i3="L141"4c3ass="line" name="L151"4 351<4a3    esfy thancafii
,-3s fllovify tso > 149lcegmentef="Documentation/2vm/numa_memmory_policy.txt#L224d3="L143"4c3ass="line" name="L153"4 353<4a35lincy pagesety

Tl acharigispLXR softwLXR AUemunity5lxo@gtlem.no5 kindsedhosby sy ttef="Docuhttp://www.polpill-gtlA/">RolpillhrILA/l AS5