2Memory Hotplug
   5Created:                                        Jul 28 2007
   6Add description of notifier of memory hotplug   Oct 11 2007
   8This document is about memory hotplug including how-to-use and current status.
   9Because Memory Hotplug is still under development, contents of this text will
  10be changed often.
  121. Introduction
  13  1.1 purpose of memory hotplug
  14  1.2. Phases of memory hotplug
  15  1.3. Unit of Memory online/offline operation
  162. Kernel Configuration
  173. sysfs files for memory hotplug
  184. Physical memory hot-add phase
  19  4.1 Hardware(Firmware) Support
  20  4.2 Notify memory hot-add event by hand
  215. Logical Memory hot-add phase
  22  5.1. State of memory
  23  5.2. How to online memory
  246. Logical memory remove
  25  6.1 Memory offline and ZONE_MOVABLE
  26  6.2. How to offline memory
  277. Physical memory remove
  288. Memory hotplug event notifier
  299. Future Work List
  31Note(1): x86_64's has special implementation for memory hotplug.
  32         This text does not describe it.
  33Note(2): This text assumes that sysfs is mounted at /sys.
  371. Introduction
  401.1 purpose of memory hotplug
  42Memory Hotplug allows users to increase/decrease the amount of memory.
  43Generally, there are two purposes.
  45(A) For changing the amount of memory.
  46    This is to allow a feature like capacity on demand.
  47(B) For installing/removing DIMMs or NUMA-nodes physically.
  48    This is to exchange DIMMs/NUMA-nodes, reduce power consumption, etc.
  50(A) is required by highly virtualized environments and (B) is required by
  51hardware which supports memory power management.
  53Linux memory hotplug is designed for both purpose.
  561.2. Phases of memory hotplug
  58There are 2 phases in Memory Hotplug.
  59  1) Physical Memory Hotplug phase
  60  2) Logical Memory Hotplug phase.
  62The First phase is to communicate hardware/firmware and make/erase
  63environment for hotplugged memory. Basically, this phase is necessary
  64for the purpose (B), but this is good phase for communication between
  65highly virtualized environments too.
  67When memory is hotplugged, the kernel recognizes new memory, makes new memory
  68management tables, and makes sysfs files for new memory's operation.
  70If firmware supports notification of connection of new memory to OS,
  71this phase is triggered automatically. ACPI can notify this event. If not,
  72"probe" operation by system administration is used instead.
  73(see Section 4.).
  75Logical Memory Hotplug phase is to change memory state into
  76available/unavailable for users. Amount of memory from user's view is
  77changed by this phase. The kernel makes all memory in it as free pages
  78when a memory range is available.
  80In this document, this phase is described as online/offline.
  82Logical Memory Hotplug phase is triggered by write of sysfs file by system
  83administrator. For the hot-add case, it must be executed after Physical Hotplug
  84phase by hand.
  85(However, if you writes udev's hotplug scripts for memory hotplug, these
  86 phases can be execute in seamless way.)
  891.3. Unit of Memory online/offline operation
  91Memory hotplug uses SPARSEMEM memory model. SPARSEMEM divides the whole memory
  92into chunks of the same size. The chunk is called a "section". The size of
  93a section is architecture dependent. For example, power uses 16MiB, ia64 uses
  941GiB. The unit of online/offline operation is "one section". (see Section 3.)
  96To determine the size of sections, please read this file:
 100This file shows the size of sections in byte.
 1032. Kernel Configuration
 105To use memory hotplug feature, kernel must be compiled with following
 106config options.
 108- For all memory hotplug
 109    Memory model -> Sparse Memory  (CONFIG_SPARSEMEM)
 110    Allow for memory hot-add       (CONFIG_MEMORY_HOTPLUG)
 112- To enable memory removal, the followings are also necessary
 113    Allow for memory hot remove    (CONFIG_MEMORY_HOTREMOVE)
 114    Page Migration                 (CONFIG_MIGRATION)
 116- For ACPI memory hotplug, the followings are also necessary
 117    Memory hotplug (under ACPI Support menu) (CONFIG_ACPI_HOTPLUG_MEMORY)
 118    This option can be kernel module.
 120- As a related configuration, if your box has a feature of NUMA-node hotplug
 121  via ACPI, then this option is necessary too.
 122    ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu)
 124    This option can be kernel module too.
 1274 sysfs files for memory hotplug
 129All sections have their device information in sysfs.  Each section is part of
 130a memory block under /sys/devices/system/memory as
 133(XXX is the section id.)
 135Now, XXX is defined as (start_address_of_section / section_size) of the first
 136section contained in the memory block.  The files 'phys_index' and
 137'end_phys_index' under each directory report the beginning and end section id's
 138for the memory block covered by the sysfs directory.  It is expected that all
 139memory sections in this range are present and no memory holes exist in the
 140range. Currently there is no way to determine if there is a memory hole, but
 141the existence of one should not affect the hotplug capabilities of the memory
 144For example, assume 1GiB section size. A device for a memory starting at
 1450x100000000 is /sys/device/system/memory/memory4
 146(0x100000000 / 1Gib = 4)
 147This device covers address range [0x100000000 ... 0x140000000)
 149Under each section, you can see 4 or 5 files, the end_phys_index file being
 150a recent addition and not present on older kernels.
 158'phys_index'      : read-only and contains section id of the first section
 159                    in the memory block, same as XXX.
 160'end_phys_index'  : read-only and contains section id of the last section
 161                    in the memory block.
 162'state'           : read-write
 163                    at read:  contains online/offline state of memory.
 164                    at write: user can specify "online_kernel",
 165                    "online_movable", "online", "offline" command
 166                    which will be performed on al sections in the block.
 167'phys_device'     : read-only: designed to show the name of physical memory
 168                    device.  This is not well implemented now.
 169'removable'       : read-only: contains an integer value indicating
 170                    whether the memory block is removable or not
 171                    removable.  A value of 1 indicates that the memory
 172                    block is removable and a value of 0 indicates that
 173                    it is not removable. A memory block is removable only if
 174                    every section in the block is removable.
  78when a1 memo17umentation/memory-hotplug.txt#L149" id="L149" clas="line" n1ame="L79">  79
  80In thi1s doc1ment, s opsymbolic  80ks 
  82Logica1l Mem1ry Hote 1GiB secttation/memory-hotplug.txt#L377" id="L17ory
  83admini1strat18system/memory/memoryXXX/eef="ref="0t#L377"9ry  (CO../../emovable
  84phase 1by ha18umentation/memory-hotplug.txt#L135" id="L135" clas="line" n1ame="L85">  85(Howev1er, i1 you wA back 80k on alhref= sec       ation/memory-hotplug.txt#L135" id="L135" clas=="line" name="L86">  86 phase1s can18system/memory/memoryXXX/removable
  891.3. U1nit o1 Memor hot-add phase
  91Memory1 hotp19umentation/memory-hotplug.txt#L82" id="L82" class1="line" n1ame="L92">  92into c1hunks1of theare) Support
  93a sect1ion i1 archief="Documentation/memory-hotplug.txt#L129" id="L129" clas=="line" name="L94">  941GiB. 1The u1it of Onspecial/"Docuplatons ,ase
  96To det1ermin1 the sIn ge two ,
  98/sys/d1evice1/systeLg isipts fo
  19  4.1 Hait o1cument hotpl.implemon al se9542PI can notify tation/memory-hotplug.txt#L165" id="L165" cla2s="line" 2ame="L100"> 100This 2ile s20umentation/memory-hotplug.txt#L31" id="L31" class2s="line" 2ame="L101"> 101
 102-----2-----2------You may informole by si"Docutureselfocumte of memory.
 1032. Ke2nel C2nfigurPle:
 104-----2-----20umentation/memory-hotplug.txt#L135" id="L135" cla2s="line" 2ame="L105"> 105To us2 memo2y hotpts notification of co 106confi2 opti2ns.
 108- For2all m2mory hI's vi in sysfug, und,ase
 109    M2mory 20umentation/memory-hotplug.txt#L120" id="L120" cla2s="line" 2ame="L110"> 110    A2low f21umentation/memory-hotplug.txt#L31" id="L31" class2s="line" 2ame="L111"> 111
 112- To 2nable21----------
 113    A2low f2r memoIn some
 114    P2ge Mi2rationndd event by handgeredref="Df py.  ref="Der usesu po 115
 116- For2ACPI 21umentation/memory-hotplug.txt#L67" id="L67" class2s="line" 2ame="L117"> 117    M2mory 2otplugd as NER).
 118    T2is op2ion caemory bd envirodent. For exaef="shotle:
  20  4.2 N rela22umentation/memory-hotplug.txt#L31" id="L31" class2="line" naame="L21">  215. LogiACPI,2then tPon b catirfasysfug
  22  5.1. Sable22system/memory/memoryXXX/start_phion bation/memory-hotplug.txt#L173" id="L173" cla2=="line" 2ame="L23">  23  5.2. ONFIG22umentation/memory-hotplug.txt#L144" id="L144" cla2="line" naame="L24">  246. Logiis op2ion caYfiles, tw.
  25  6.1 Mef="D2cumentation/memory-hotplug.txt#L126" id="L126" cla2="line" naame="L26">  26  6.2. -----2------ipt7; echo ection / section_  277. Physs fil22umentation/memory-hotplug.txt#L88" id="L88" class2="line" naame="L28">  288. Memo-----22hases inn, [ection / section_  299. Futuction22ns in this r.
 130a mem2ry bl2ck und href="D memory hotplu). Yfiipts .
 132/sys/2evice23umentation/memory-hotplug.txt#L53" id="L53" class2s="line" 2ame="L133"> 133(XXX 2s the23umentation/memory-hotplug.txt#L144" id="L144" cla2s="line" 2ame="L134"> 134
 135Now, 2XX is2define="Document--
 136secti2n con2ained hot-add phase
 138for t2e mem23umentation/memory-hotplug.txt#L149" id="L149" cla2s="line" 2ame="L139"> 139memor2 sect2ons inry
 140range2 Curr24 141the e2isten2e of oToathe (te of memory.
) 142block2
 144For e2ample2cumentation/memory-hotplug.txt#L45" id="L45" class2s="line" 2ame="L145"> 1450x10020000024umentation/memory-hotplug.txt#L126" id="L126" cla2s="line" 2ame="L146"> 146(0x102000002 / 1GiIef="Documentf
 147This 2evice2coversIef="Documentf
 149Under2each 2cumentation/memory-hotplug.txt#L50" id="L50" class2s="line" 2ame="L150"> 150a rec2nt ad2ition  memory
 152/sys/2evice2/systeEvenory holged, the kerne
 153/sys/2evice2/systeFf memcume 154/sys/2evice2cumentation/memory-hotplug.txt#L55" id="L55" class2s="line" 2ame="L155"> 155/sys/2evice2/systeFcumte ofcum
<9;s informole by s  "offline&quf=o 156/sys/2evice25umentation/memory-hotplug.txt#L67" id="L67" class2s="line" 2ame="L157"> 157
 158'2hys_i25umentation/memory-hotplug.txt#L149" id="L149" cla2s="line" 2ame="L159"> 159     2     2       kernee ofcumton alentte into
a href=" typeon
 161     2     2cumentation/memory-hotplug.txt#L62" id="L62" class2s="line" 2ame="L162"> 162'2tate&239;      s inn, [y-hotpuot;onlil  (COm/memory/memoryXXX/start_phocumen"Documentation/memory-hotplug.txt#L156" id="L156" cla2s="line" 2ame="L163"> 163     2     2      (ument  href="Dlimit:esent cumentf
 164     2     26umentation/memory-hotplug.txt#L55" id="L55" class2s="line" 2ame="L165"> 165     2     2      Andory holged, the
  26  6.2. Hvice2cumentation/memory-hotplug.txt#L67" id="L67" class2s="line" 2ame="L167"> 167'2hys_d267-22hases inn, [y-hotphref="Dl  (COm/memory/memoryXXX/start_phocumen"Documentation/memory-hotplug.txt#L156" id="L156" cla2s="line" 2ame="L168"> 168     2     2      (ument  href="Dlimit:esent cumentf
 169'2emova2cumentation/memory-hotplug.txt#L70" id="L70" class2s="line" 2ame="L170"> 170     2     2      Alug
 171     2     2      s. Amount cumentfon al sethe amouqtation/memory-hotplug.txt#L33" id="L33" class2s="line" 2ame="L172"> 172     2     27umentation/memory-hotplug.txt#L53" id="L53" class2s="line" 2ame="L173"> 173     2     2       no way t,e 174     2     2       kernmay  setse. Thepref hreftation/memory-hotplug.txt#L124" id="L124" cla2s="line" 2ame="L175"> 175
 176NOTE:2  78when a2 memo27-------------------
  80In thi2s doc28  81
  82Logica2l Mem2ry Hot6.1line operref="Doentoef="Documentation/memory-hotplug.txt#L82" id="L82" class2=="line" 2ame="L83">  83admini2strat28archief="Documentation/memory-hotplug.txt#L129" id="L129" cla2=="line" 2ame="L84">  84phase 2by ha28umentine operref="cumternmorllowinlia, if tse.a this rt;offl. Beca feature, krref="Dation/memory-hotplug.txt#L129" id="L129" cla2=="line" 2ame="L85">  85(Howev2er, i2 you wUMA-f pn it  86 phase2s can28systea href="Documnclury
  891.3. U2nit o28umentation/memory-hotplug.txt#L70" id="L70" class2""line" n2ame="L90">  90------2-----2
  91Memory2 hotp29ument(2) m     ocs frumentes
  92into c2hunks29umentation/memory-hotplug.txt#L53" id="L53" class2="line" n2ame="L93">  93a sect2ion i29 memoIn a hr href="D memory hotplu, Lg isipts foture, krref="Dlses
  941GiB. 2The u2it of s frrumentes
  96To det2ermin2 the s<    cach"shoFcumtref="cumtaref="Doceb.
m       (nizes new memUMA-f pguaranteDation/memory-hotplug.txt#L129" id="L129" cla2=ref="Doc2ame="L97">  97
  98/sys/d2evice29umentation/memory-hotplug.txt#L149" id="L149" cla2="line" n2mme="L19">  19  4.1 2ait o2cumentd as a bomorel modu, theakcumtaref="Doceory poeonsistg
 100This 3ile s30umentpcumententiByine_kerncumtine&qunew mecorl=y systemum; commot;onlicorl=y systebomorel mod 5 files,ation/memory-hotplug.txt#L77" id="L77" class3s1"line" 3ame="L91">  91Memory3ef="D3cumentc     oef="Document...a z542Pory poernjth fref=", thet;onlilumenttation/memory-hotplug.txt#L175" id="L175" cla3s="line" 3ame="L102"> 102-----3-----3------(Sea href=ry-hotplug.txtnew me-paramis as5" chation/memory-hotplug.txt#L88" id="L88" class3s="line" 3ame="L103"> 1032. Ke3nel C30umentation/memory-hotplug.txt#L144" id="L144" cla3s"line" n3ame="L104"> 104-----3-----30umentA size.y.  It  is UMA-y systTOTALy systeaom user's virefbomortimescribedbomorel modation/memory-hotplug.txt#L144" id="L144" cla3s5line" n3ame="L95">  95
  96To det3 opti30umentation/memory-hotplug.txt#L67" id="L67" class3s"line" n3ame="L107"> 107
 108- For3all m30       S byte.
39;s viemor, thet;onlilument (emor, themory.
)  19  4.1 3mory 3odel -&gS byte.
39;s vi, thet;onlilument (, themory.
) 110    A3low f31umentation/memory-hotplug.txt#L31" id="L31" class3s="line" 3ame="L111"> 111
 112- To 3nable312      S byte.
39;s viemor, thet;onlilument (emor, themory.
) 113    A3low f3r memoryS byte.
39;s vi, thet;onlilument (, themory.
) 114    P3ge Mi31umentation/memory-hotplug.txt#L55" id="L55" class3s="line" 3ame="L115"> 115
 116- For3ACPI 31umentNot
) 117    M3mory 3otplugf pef="Document.implementTBDtation/memory-hotplug.txt#L175" id="L175" cla3s="line" 3ame="L118"> 118    T3is op31umentation/memory-hotplug.txt#L149" id="L149" cla3s"line" n3ame="L119"> 119
  20  4.2 3 rela32ument6memory
  215. Log3ACPI,32  22  5.1.3Sable32-----You es, rref="Doeref="Doceb.
emcume chunk is t is catirfasys hrefwa href= itation/memory-hotplug.txt#L161" id="L161" cla3=="line" 3ame="L23">  23  5.2.3ONFIG32ument this rt;offssatation/memory-hotplug.txt#L175" id="L175" cla3="line" n3ame="L24">  246. Log3is op32umentation/memory-hotplug.txt#L55" id="L55" class3="line" n3ame="L25">  25  6.1 3ef="D3cumentases inn, [yref="Do  (COm/memory/memoryXXX/start_phocumen"Documentation/memory-hotplug.txt#L156" id="L156" cla3="line" n3ame="L26">  26  6.2.3-----32umentation/memory-hotplug.txt#L67" id="L67" class3=="line" 3ame="L27">  277. Phy3s fil32oversIefemory.
  288. Mem3-----32ory hI&#it f Amt,f
  299. Fut3ction32ns inEvenory eref="Doceit.
 130a mem3ry bl330ry hI&#it it.
cipts oremory beipts un : read-only:memory
<9;sipts .
 132/sys/3evice33umentAref="Docery repef="Documenth="Dtonsiddirectf= seonliltf= seemory.
d easiy tation/memory-hotplug.txt#L165" id="L165" cla3s="line" 3ame="L133"> 133(XXX 3s the33umentButmry rep
busyla hre,le. may return -EBUSY. Evenory ered, the
 135Now, 3XX is3defineemory.
 136secti3n con3ained (e 1GiB section
<    ref="f 137'3nd_ph33--
 138for t3e mem33umentation/memory-hotplug.txt#L149" id="L149" cla3s="line" 3ame="L139"> 139memor3 sect3ons inConsiddiug.tx ation/memory-hotplug.txt#L135" id="L135" cla3s="line" 3ame="L140"> 140range3 Curr34 141the e3isten3e of o envrepentor pguaranteDmry"L13gcumt39;s viry repanyp
itu    (. Butmen neentation/memory-hotplug.txt#L77" id="L77" class3s="line" 3ame="L142"> 142block3
 144For e3ample3cumentit.
aom user&#retopehref=120e
 1450x10030000034umentation/memory-hotplug.txt#L126" id="L126" cla3s="line" 3ame="L146"> 146(0x1030000034-------------------
 147This 3evice3covers7hot-add phase
 149Under3each 3cumentNeenDmorll memory hotplu yet....ation/memory-hotplug.txt#L88" id="L88" class3s="line" 3ame="L150"> 150a rec3nt ad3ition  --add eion of nowinle of nr&#reMORY_worksDocuOSor pnotifica.ation/memory-hotplug.txt#L88" id="L88" class3s="line" 3ame="L151"> 151
 152/sys/3evice35umentation/memory-hotplug.txt#L53" id="L53" class3s="line" 3ame="L153"> 153/sys/3evice35archief="Document---------
 154/sys/3evice3cument8.nder ACPI Supporref="Dnnectiotation/memory-hotplug.txt#L77" id="L77" class3s="line" 3ame="L155"> 155/sys/3evice35efine="Document--
 156/sys/3evice35umentder ACPI SupporUMA-ref="Dnnectiot.impeterarD 6 type00S,
157 158'3hys_i35ument 159 3 3 Ge two if beforll 160'3nd_ph36 prepicatiobryXXX/"Df pumenlnDocumentys_in< ion 161 3 3 f pion 162'3tate&36umentation/memory-hotplug.txt#L53" id="L53" class3s="line" 3ame="L163"> 163 3 3 164 3 3 Ge two if ry 165 3 36umentation/memory-hotplug.txt#L126" id="L126" cla3s"line" n3mme="L26"> 26 6.2.3Hvice3cument 167'3hys_d367 Ge two if w is oer ACPIasDauccestfuoirobroughtrt;offl. s_intionbackxmayation/memory-hotplug.txt#L82" id="L82" class3s="line" 3ame="L168"> 168 3 3 ion 169'3emova3cumentation/memory-hotplug.txt#L70" id="L70" class3s="line" 3ame="L170"> 170 3 3 171 3 3 Ge two if tf= sgs 172 3 3 longg 173 3 3 fugptn ali, fe. s_intionbackxes, behref= r pn"Doced, theknowsef paation/memory-hotplug.txt#L82" id="L82" class3s="line" 3ame="L174"> 174 3 3 iobryXXX/i="om#al mhref="Dodred, the 175 176NOTE:3 78when a3 memo37 ="Doin the m hrefwt atXX/p if tf=emory. .ation/memory-hotplug.txt#L88" id="L88" class3s="line" 3ame="L79"> 79 80In thi3s doc38 81 82Logica3l Mem38umentation/memory-hotplug.txt#L53" id="L53" class3=="line" 3ame="L83"> 83admini3strat38archiAntionbackxrout="Dles, sereguld rrectoation/memory-hotplug.txt#L53" id="L53" class3=="line" 3ame="L84"> 84phase 3by ha38 d="L53"_#L53" _nnectiot(tionback_func,rprioritytation/memory-hotplug.txt#L134" id="L134" cla3=="line" 3ame="L85"> 85(Howev3er, i38umentation/memory-hotplug.txt#L176" id="L176" cla3=="line" 3ame="L86"> 86 phase3s can38systeT"Doin odicarghotplf="Dtionbackxfuncthe m(acthe ) 87 88 891.3. U3nit o38umentstructDocumen_nnectD {ation/memory-hotplug.txt#L149" id="L149" cla3""line" n3ame="L90"> 90------3-----39 whethun name olongDa href=fntation/memory-hotplug.txt#L171" id="L171" cla3="line" n3ame="L91"> 91Memory3 hotp391 whethun name olongDnr_umenttation/memory-hotplug.txt#L171" id="L171" cla3="line" n3ame="L92"> 92into c3hunks39 blockcatla hrus_tse. T_nid_nnrmaltation/memory-hotplug.txt#L171" id="L171" cla3=="line" 3ame="L93"> 93a sect3ion i393 blockcatla hrus_tse. T_nid_ envtation/memory-hotplug.txt#L171" id="L171" cla3=="line" 3ame="L94"> 941GiB. 3The u394 blockcatla hrus_tse. T_nidtation/memory-hotplug.txt#L171" id="L171" cla3=="line" 3ame="L95"> 95 96To det3ermin39umentation/memory-hotplug.txt#L67" id="L67" class3=ref="Doc3ame="L97"> 97 98/sys/d3evice39umentnr_ument fug#y 19 4.1 3ait o3cument hrus_tse. T_nid_nnrmal fugpe"Dnnd mhf w is NDNORMAL 100This 4ile s40umentfug(on al s)gpe"/clear,ory holement-1ex fin 91Memory4ef="D4cumenta hrus_tse. T_nid_ env fugpe"Dnnd mhf w is NDHIGH 92into c4-----4------fug(on al s)gpe"/clear,ory holement-1ex fin 93a sect4nel C40umenta hrus_tse. T_nid fugpe"Dnnd mhf w is ND 941GiB. 4-----40umentpe"/clear. ItDocaa h mea (#L88" less)Dnnd mgetsl 95 96To det4 opti40/ 1GiIefa hrus_tse. Td_nid*o (C= 0,Dtionbackxshouldec /discar istructureug, th fiation/memory-hotplug.txt#L88" id="L88" class4s7"line" 4ame="L97"> 97 108- For4all m40umentation/memory-hotplug.txt#L149" id="L149" cla4s9"line" 4ame="L19"> 19 4.1 4mory 4odel -ef="Document"Dation/memory-hotplug.txt#L88" id="L88" class4s="line" 4ame="L110"> 110 A4low f41ument9. F href Workation/memory-hotplug.txt#L88" id="L88" class4s="line" 4ame="L111"> 111 112- To 4nable412 - ion 113 A4low f413 blsysctllcumea hemorrolwdex&tation/memory-hotplug.txt#L85" id="L85" class4s"line" n4ame="L114"> 114 P4ge Mi414 - hysicumaocumenhin the mndich-add pha in sysrel, i 115 116- For4ACPI 41ument - tesles exmak si"Doettir039;s virref="cumtation/memory-hotplug.txt#L85" id="L85" class4s7"line" 4ame="L117"> 117 M4mory 417ment - hcument HugeTLB < m (es exrref="cumtation/memory-hotplug.txt#L85" id="L85" class4s="line" 4ame="L118"> 118 T4is op418ment - #L8map#reMORcumtat039;s virref="&tation/memory-hotplug.txt#L85" id="L85" class4s9"line" 4ame="L119"> 119 20 4.2 4 rela42umentation/memory-hotplug.txt#L31" id="L31" class4=="line" 4ame="L21"> 215. Log4ACPI,42
footir"> T"DooriginalrLXRD oftficattory. tion/memo"/projpabs/lxr">LXRDcommunitya hrscribedexperiotplul versDoceb. tion/memomailto:lxr@lg">lxr@lg is.noa hr.
subfootir"> lxr.lg kindlnhhesteteb. tion/memohttp://www.redpn a-lg">Redpn aDLg pro ASa hrscproviddi0S, Lg isDtonsultcumta exrpdiug.txugpery/mem sinsys1995.