1Started Nov 1999 by Kanoj Sarcar <>
   3What is NUMA?
   5This question can be answered from a couple of perspectives:  the
   6hardware view and the Linux software view.
   8From the hardware perspective, a NUMA system is a computer platform that
   9comprises multiple components or assemblies each of which may contain 0
  10or more CPUs, local memory, and/or IO buses.  For brevity and to
  11disambiguate the hardware view of these physical components/assemblies
  12from the software abstraction thereof, we'll call the components/assemblies
  13'cells' in this document.
  15Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset
  16of the system--although some components necessary for a stand-alone SMP system
  17may not be populated on any given cell.   The cells of the NUMA system are
  18connected together with some sort of system interconnect--e.g., a crossbar or
  19point-to-point link are common types of NUMA system interconnects.  Both of
  2"L19"> t2l2S_2.6.20f
  112isamb2guate NUMA    2
  132#39;c2lls&#and re vi,stem are
  142  152ach o2 the h 
ef="aca stiblole of pl. mem   162f the2systehrehef=loitmenw of thescartem ubset
  172ay no2f="Documentation/vm/numa#L8" id="L8" c2class="l2ne" name="L18">  182onnec2ed toMr IO  aca st time
ef="effMA sys/or IO  bef=wid ofvariosudepen  192oint-2o-poiawaartem  NUM
 href="/dintem mem   For breed k/dintem mr IO  aca st isle of "Documentation/vm/numa#L6" id="L6" c3class="l3ne" name="L20">  2"L19"3 t2l230uate NUM
 href="/dintem e="">
 mr IO ty and examts , aca st to/or IO  b. memocumentation/vm/numa#L13" id="L13"3class="l3ne" name="L11">  113isamb3guate   2
3rom t3f="Dobef=wid osf="Dnaaca stet to/or IO   ceome s, rr Ithe NUMAty are
   3Wh#39;c3lls&#/numhavhe NUMA   143  153ach o3 the P
  163f the3systela hrocumene a/dity Rame s, t.
  173ay no3 be por IO  bef=wid oty Howeves, t 
p	 ievestponablo/or IO  bef=wid o,  183onnec3ed toapplnts, we&traction mu atyprange-alone l="">/oajora hrsystem or IO  tat9;lccDocumentation/vm/numa#L13" id="L13"3class="l3ne" name="L19">  193oint-3o-poi[c#39osmistet] t 
", and" or IO --or IO   cealthoamm  NUM, if ll.--Documentation/vm/numa#L19" id="L19"4class="l4ne" name="L20">  2"L19"4 t2l240the h 
tem  lose at NUM
wi ofmr IO tcumentation/vm/numa#L19" id="L19"4class="l4ne" name="L11">  114isamb41="Documentation/vm/numa#L15" id="L15"4lass="li4ne" name="L2">   2
4rom t42stion canleadsth 
althre view.
   3W4#39;c43="Documentation/vm/numa#L15" id="L15"4lass="linne" name="L4">   4
<  154ach o4 the thereof, wesmponeoit"nodbs"ty re viemapsFalthnodbs  ch 
althponents/asNUMAcumentation/vm/numa#L15" id="L15"4class="l4ne" name="L16">  164f the4system--althspective, 
  174ay no4 be p"
	 itMA uonAty Asort ohponents/asNUMA,ew.
  184onnec4ed tomemorymr IO  aes.  For brevity And, tgf=", mr IO  aca stet to/or IO   ccumentation/vm/numa#L15" id="L15"4class="l4ne" name="L19">  194oint-4o-poi" loser" nodbs--nodbs ="Doemapth 
  2"L19"5 t2l250the eas thaaca st timesuses hig39;"effMA sys/bef=wid of="Dnaaca stet to/ocalcumentation/vm/numa#L15" id="L15"5class="l5ne" name="L11">  115isamb51="Dorr Ithe NUMAtcumentation/vm/numa#L15" id="L15"5lass="li5ne" name="L2">   2
5rom t5f="Documentation/vm/numa#L3" id="L3" c5lass="li5ne" name="L3">   3W5#39;c5lls&#and  syst"
	 itMA uonA,ewuchsymmx86, re viewiUM
"hidb" ll. nodbsneponAnum dinacumentation/vm/numa#L3" id="L3" c5lass="li5ne" name="L4">   4
5   5Thach o5 the h"DoesNUM = 
p nodbsneponAnum dinaasNUM ="Doedoesmhavhemr IO ty ThuA,e ccumentation/vm/numa#L15" id="L15"5class="l5ne" name="L16">  165f the562.6.20f
  175ay no5 be p"   The nodbswiUM
seeealthoamm , and/or IO aaca st timesuses bef=wid otcumentation/vm/numa#L20" id="L20"5class="l5ne" name="L18">  185onnec58="Documentation/vm/numa#L3" id="L3" c5class="l5ne" name="L19">  195oint-5o-poiIn pddi#L3", alon sysa"
	 itMA uonA,etgf="mx86 r pln examts , re viewupportAcumentation/vm/numa#L15" id="L15"6class="l6ne" name="L20">  2"L19"6 t2l26_2.6.20f emy gi we&sysmddi#L3"nd/nodbsty and is a emy gi we, le viewiUM
carvheupcumentation/vm/numa#L15" id="L15"6class="l6ne" name="L11">  116isamb612.6.20f exi a/di nodbs--  Falthough soor IO aalonnon-are
6rom t6f="Donodbsty 'emy gioitnodbswiUM
mana">/ale of, we&m--althunderly/di ay be viewcumentation/vm/numa#L15" id="L15"6lass="li6ne" name="L3">   3W6#39;c63="Doponents/amr IO ty is a emlugi we&r prevfulaalonte a/di is a kernel="ndcumentation/vm/numa#L18" id="L18"6lass="li6ne" name="L4">   4
6   5T6ach o65be popna">/numrorc"Dnism whhe revith some sort ofcprevtAtcumentation/vm/numa#L15" id="L15"6lass="linne" name="L6">   6haf the662.6.[seeen/vm/numa#L15"cgroups/cprevtAttxt]cumentation/vm/numa#L15" id="L15"6class="l6ne" name="L17">  176ay no6f="Documentation/vm/numa#L8" id="L8" c6class="l6ne" name="L18">  186onnec68ls&#and h maynodbswi ofmr IO , re vie hrereun/vmln indepen/numcumentation/vm/numa#L8" id="L8" c6class="l6ne" name="L19">  196oint-6o-poief="ugh s,necesletbswi ofilieswe&freeepa">/li asbuin-useepa">/li asbuusa">cumentation/vm/numa#L7" id="L7" clclass="l7ne" name="L20">  2"L19"7 t2l27_2.6.ight. a/csuses , akt to/ordiateaaca stty In pddi#L3", re vie hrereun/vmfDocumentation/vm/numa#L19" id="L19"7class="l7ne" name="L11">  117isamb712.6.h mayor IO  zem
<, local terDMA,rDMA32, NORMAL, HIGH_MEMORY, MOVABLE],cumentation/vm/numa#L19" id="L19"7lass="li7ne" name="L2">   2
7rom t7f="Doln ordcoupl"zem
li a"ty A zem
li a NUMAifibsFalthzem
s/nodbs =oevisit whhe acumentation/vm/numa#L3" id="L3" c7lass="li7ne" name="L3">   3W7#39;c73="Do     4
7   5T7ach o75-poi"overflow" , l"foneback"tcumentation/vm/numa#L19" id="L19"7lass="li7ne" name="L6">   6h7f the76="Documentation/vm/numa#L8" id="L8" c7lass="linne" name="L7">   7
  187onnec78be por IO , re viemu atdMAidbswhome so=oeordcoFalthzem
li asewuchs="Doerldots, weAcumentation/vm/numa#L15" id="L15"7class="l7ne" name="L19">  197oint-79the eaUM
althoamm zem
< hren celndiff9;lcomnodb, , l= 
p diff9;lcomzem
cumentation/vm/numa#L8" id="L8" clclass="l8ne" name="L20">  2"L19"8 t2l28_2.6.2hren cealthoamm nodbty Thr pr pln importacom hreidbrs, we&becausee sysazem
s,cumentation/vm/numa#L19" id="L19"8class="l8ne" name="L11">  118isamb81-poiefchsymmDe
8rom t8f="DoltdMfaone zem
li a ordcoFbasven cealthoizeA system various zem
< hres ne gi vecumentation/vm/numa#L19" id="L19"8lass="li8ne" name="L3">   3W8#39;c83the h 
tem totnd/or IO asystem nodb   4
8   5T8ach o85-poibotedpar5   6h8f the86="Don/vm/numa#L18" ysctl" ittxt]cumentation/vm/numa#L15" id="L15"8lass="li8ne" name="L7">   7
8ay no8f="Documentation/vm/numa#L8" id="L8" cllass="linne" name="L8">   8Fronnec88be pBytdMfaone, re viewiUM
ath spt =oesht. faror IO aaldots, we&re be asle of "Documentation/vm/numa#L6" id="L6" c8class="l8ne" name="L19">  198oint-89="Donodb =oentain tem mem ="DoeexecutbsFalthre be apr plmeigneds oSUMAifiponey,cumentation/vm/numa#L19" id="L19"9class="l9ne" name="L20">  2"L19"9 t2l290="Dore viewiUM
ath spt =oealdots,ele of "Do firsomnodbument.b  119isamb91-poif  Falthnodbsw39;lFalthre be aporiginatesty Thr pr pponeoit", and/aldots, we."cumentation/vm/numa#L19" id="L19"9lass="li9ne" name="L2">   2
9rom t9f="DoIellsquot;, and" nodb   3W9#39;c93="Donodbs viewezem
ment.b<     4
9   5T9ach o95="Documentation/vm/numa#L8" id="L8" c9lass="li9ne" name="L6">   6h9f the96="DoL and/aldots, we=wiUM
tef="Do keepref="D beum aca st to/tem rldots,uplor IO cumentation/vm/numa#L8" id="L8" c9lass="li9ne" name="L7">   7
9ay no97-poi", and" to/tem underly/di ponents/ane ourchrox soofystem "ugh st--e.g., a croscumentation/vm/numa#L8" id="L8" c9lass="li9ne" name="L8">   8F9onnec9ed toas e=dinasntem e=skn cewhoses ahalf&tem kernel=rldots,upl sysaor IO adoesmatecumentation/vm/numa#L8" id="L8" c9lass="linne" name="L9">   9cooint-99="Do giooFdigrateaawaare of "Datemr IO ty Ththre view39oiulooFr pla hresystemcumentation/vm/numa#L10" id="L10" cclass="linne" name="L10">  10or t2lor ="Doare
  11d01t2lor1-poiereun/uonA [seeen/vm/numa#L15"w39oiuloo"w39oi-domf="sttxt]--x softwaw39oiuloocumentation/vm/numa#L3" id="L3" c10class="li0ne" name="L12">  12f02t2lor2uate  13&03t2lor32.6.20f w39oiulooFdoesmate e=k>/ale=sk viewsnare
04t2lor4stion uA,eunderrefffipieum imbalP a ,le=skvm/numdigrateabetwehe nodbs, rr Ithcumentation/vm/numa#L19" id="L19"10class="li0ne" name="L15">  15E05t2lor5 software abioFrnitind/nodbox sokernel=data ereun/uonAtcumentation/vm/numa#L19" id="L19"10class="li0ne" name="L16">  16o06t2lor6="Documentation/vm/numa#L8" id="L8" c10class="li0ne" name="L17">  17m07t2lor7-poiS href="do="/ereotor  18c08t2lor8the h 
<, anda hrus/dinvarious mem   19p09t2lor9-poiefchsymme=skvet(1)asess="L1crl(1)busessprogram  10oor mo1e CPUsw39oi_vet  11di1t2loiguatealdots, we= ahavi  Fus/dinre vieare
  12ffrom 1he sof[seeen/vm/numa#L15" id="L1_or IO _polic ttxt.]cumentation/vm/numa#L15" id="L15"1 class="liine" name="L13">  13&'113="Documentation/vm/numa#L15" id="L15"1 class="liine" name="L14">  14

  15EEach 1f the privilegoituserasln sUMAify
ment.b< 39oiul/dinnd is a  of ses  16oof th1 systeus/din hrerol groupsasessmem  17mmay n11f="Documentation/vm/numa#L8" id="L8" c1 class="liine" name="L18">  18cconne1ted toOna"
	 itMA uonA ="Doedomate hidb  19ppoint1to-poizem
[nodbs]swi ofmr IO 
ment.b  2"L19"1> t2l1S_2.6.nodb =squot;, andfmr IO 
nodb3quot;--althnodbssystem firsomzem
li a--wiUM
ated onalthnodbsit    12frrom 12f="Dokernel=    1312#39;12lls&#So,tdMfaone, , and/aldots, wesswiUM
suca uplwi oftem kernel=wupply/di temcumentation/vm/numa#L10" id="L10" 2class="li2ne" name="L14">  1412  1512ach 12 the aldowsewuchsyldots, wessh 
fonebacko=oeome soneanbyanodbs whhe a nodbst"Documentation/vm/numa#L9" id="L9" c12class="li2ne" name="L16">  1612f th12systedoesm href="Dpr IO asverflowstcumentation/vm/numa#L10" id="L10" 2class="li2ne" name="L17">  1712ay n12f="Documentation/vm/numa#L8" id="L8" c12class="l12ne" name="L18">  1812onne12ed toSsysakernel=rldots, wessdomate wP t a "c#uatedtolerware vr plldots, we=fonebackcumentation/vm/numa#L8" id="L8" c12class="li2ne" name="L19">  1912oint12o-poi ahavi  ty Rame sstemy wP t t 
a#Lsu;lFalty ">
 mr IO le of "Do NUMAifibssnodbcumentation/vm/numa#L13" id="L13" cclass="li3ne" name="L20">  2"L19"13 t2l13 CPUs, l">
 ateifibsst"Doetem nodbshymmno/freeeor IO ty Thr pr pusuoney
tem  asbsw39ecumentation/vm/numa#L8" id="L8" c13class="l13ne" name="L11">  1113isam13guate< ef="ugh s=rldots,us, a  mem or IO  ta ourchs, alonexamts tcumentation/vm/numa#L10" id="L10" 3lass="li13ne" name="L2">   2
13rom 13f="Documentation/vm/numa#L3" id="L3" c13class="linne" name="L13">  13&##39;13lls&#A< hrnts/amodblaalond k/dinwuchsynplldots, we=issh 
obef="Dalthnodbsidesystemcumentation/vm/numa#L10" id="L10" 3class="l13ne" name="L14">  1413  1513ach 13 the ="L1_nodb_id() a "mem_to_nodb() func, weA  1613f th1362.6.20fhnodbside;">
	  1713ay n13 be populrevest t 
ilieswe&fonebackopa oty ThthsMabakernel=or IO aaldots,ooFr plecumentation/vm/numa#L8" id="L8" c13class="l13ne" name="L18">  1813onne13ed toexamts <,frteisty Or,rtem ef="ugh s=0
  1913oint13o-poiit    2"L19"14 t2l140the heistcumentation/vm/numa#L8" id="L8" c14class="l14ne" name="L11">  1114isam141="Documentation/vm/numa#L15" id="L15"14lass="li14ne" name="L2">   2
14rom 14f="DoIells&#"
	 itMA uon wupportA--doesmate hidb--or IO l st nodbs, ftwe&memocumentation/vm/numa#L13" id="L13"14lass="li14ne" name="L3">   3W14#39;143="Do  14
<  1514ach 14 the or IO aexclus velyse of penodbswi ooutemr IO ty To wupport&wuchcumentation/vm/numa#L13" id="L13"14class="l14ne" name="L16">  1614f th14syste"
	 itMA uonA =rantpa;lcoly, kernel=wubef="Dos sln useetem ="L1_mr _id()cumentation/vm/numa#L13" id="L13"14class="l14ne" name="L17">  1714ay n14 be pa "cpu_to_mr () func, weth 
dots,el=squot;, andfmr IO 
nodb3quot;nf  Falthpone/dinndcumentation/vm/numa#L13" id="L13"14class="l14ne" name="L18">  1814onne14ed toNUMAifibssmemty Agf=", thr pr palthoamm nodbse of ntain dMfaone, , and/pa">cumentation/vm/numa#L7" id="L7" c14class="l1e="L19">  194oint-4o14oint1h16.3"
   uid="L9" c1e of ntain dMfaone, , and/pa">cumen2e9">  194oint-4o14oint1h16.3"
wid of="Dnaaca stet to/o1calcu1entati
4cla id="L1l LXRe" name="Ld="L14"men2e9"> http:// 2LXRe> 1uma#Ltati"L7" c1250the/a>4ol 14fsass="y"men2e9">">lxr@6.notati. f67957586dcbof="Dnane"nameer"> d="d4clhos/a>1"y"men2e9"> http://www.redpaon-Redpaon9"> pro AStati"L>3ay nor/num17isams="lon/vme="p11"> 1vm/ervir mo1ime="1995. f679575a8/body758/htmlid