linux/Documentation/cgroups/memcg_test.txt
<<
>>
Prefs
   1Memory Resource Controller(Memcg)  Implementation Memo.
   2Last Updated: 2009/1/20
   3Base Kernel Version: based on 2.6.29-rc2.
   4
   5Because VM is getting complex (one of reasons is memcg...), memcg's behavior
   6is complex. This is a document for memcg's internal behavior.
   7Please note that implementation details can be changed.
   8
   9(*) Topics on API should be in Documentation/cgroups/memory.txt)
  10
  110. How to record usage ?
  12   2 objects are used.
  13
  14   page_cgroup ....an object per page.
  15        Allocated at boot or memory hotplug. Freed at memory hot removal.
  16
  17   swap_cgroup ... an entry per swp_entry.
  18        Allocated at swapon(). Freed at swapoff().
  19
  20   The page_cgroup has USED bit and double count against a page_cgroup never
  21   occurs. swap_cgroup is used only when a charged page is swapped-out.
  22
  231. Charge
  24
  25   a page/swp_entry may be charged (usage += PAGE_SIZE) at
  26
  27        mem_cgroup_newpage_charge()
  28          Called at new page fault and Copy-On-Write.
  29
  30        mem_cgroup_try_charge_swapin()
  31          Called at do_swap_page() (page fault on swap entry) and swapoff.
  32          Followed by charge-commit-cancel protocol. (With swap accounting)
  33          At commit, a charge recorded in swap_cgroup is removed.
  34
  35        mem_cgroup_cache_charge()
  36          Called at add_to_page_cache()
  37
  38        mem_cgroup_cache_charge_swapin()
  39          Called at shmem's swapin.
  40
  41        mem_cgroup_prepare_migration()
  42          Called before migration. "extra" charge is done and followed by
  43          charge-commit-cancel protocol.
  44          At commit, charge against oldpage or newpage will be committed.
  45
  462. Uncharge
  47  a page/swp_entry may be uncharged (usage -= PAGE_SIZE) by
  48
  49        mem_cgroup_uncharge_page()
  50          Called when an anonymous page is fully unmapped. I.e., mapcount goes
  51          to 0. If the page is SwapCache, uncharge is delayed until
  52          mem_cgroup_uncharge_swapcache().
  53
  54        mem_cgroup_uncharge_cache_page()
  55          Called when a page-cache is deleted from radix-tree. If the page is
  56          SwapCache, uncharge is delayed until mem_cgroup_uncharge_swapcache().
  57
  58        mem_cgroup_uncharge_swapcache()
  59          Called when SwapCache is removed from radix-tree. The charge itself
  60          is moved to swap_cgroup. (If mem+swap controller is disabled, no
  61          charge to swap occurs.)
  62
  63        mem_cgroup_uncharge_swap()
  64          Called when swp_entry's refcnt goes down to 0. A charge against swap
  65          disappears.
  66
  67        mem_cgroup_end_migration(old, new)
  68        At success of migration old is uncharged (if necessary), a charge
  69        to new page is committed. At failure, charge to old page is committed.
  70
  713. charge-commit-cancel
  72        In some case, we can't know this "charge" is valid or not at charging
  73        (because of races).
  74        To handle such case, there are charge-commit-cancel functions.
  75                mem_cgroup_try_charge_XXX
  76                mem_cgroup_commit_charge_XXX
  77                mem_cgroup_cancel_charge_XXX
  78        these are used in swap-in and migration.
  79
  80        At try_charge(), there are no flags to say "this page is charged".
  81        at this point, usage += PAGE_SIZE.
  82
  83        At commit(), the function checks the page should be charged or not
  84        and set flags or avoid charging.(usage -= PAGE_SIZE)
  85
  86        At cancel(), simply usage -= PAGE_SIZE.
  87
  88Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y.
  89
  904. Anonymous
  91        Anonymous page is newly allocated at
  92                  - page fault into MAP_ANONYMOUS mapping.
  93                  - Copy-On-Write.
  94        It is charged right after it's allocated before doing any page table
  95        related operations. Of course, it's uncharged when another page is used
  96        for the fault address.
  97
  98        At freeing anonymous page (by exit() or munmap()), zap_pte() is called
  99        and pages for ptes are freed one by one.(see mm/memory.c). Uncharges
 100        are done at page_remove_rmap() when page_mapcount() goes down to 0.
 101
 102        Another page freeing is by page-reclaim (vmscan.c) and anonymous
 103        pages are swapped out. In this case, the page is marked as
 104        PageSwapCache(). uncharge() routine doesn't uncharge the page marked
 105        as SwapCache(). It's delayed until __delete_from_swap_cache().
 106
 107        4.1 Swap-in.
 108        At swap-in, the page is taken from swap-cache. There are 2 cases.
 109
 110        (a) If the SwapCache is newly allocated and read, it has no charges.
 111        (b) If the SwapCache has been mapped by processes, it has been
 112            charged already.
 113
 114        This swap-in is one of the most complicated work. In do_swap_page(),
 115        following events occur when pte is unchanged.
 116
 117        (1) the page (SwapCache) is looked up.
 118        (2) lock_page()
 119        (3) try_charge_swapin()
 120        (4) reuse_swap_page() (may call delete_swap_cache())
 121        (5) commit_charge_swapin()
 122        (6) swap_free().
 123
 124        Considering following situation for example.
 125
 126        (A) The page has not been charged before (2) and reuse_swap_page()
 127            doesn't call delete_from_swap_cache().
 128        (B) The page has not been charged before (2) and reuse_swap_page()
 129            calls delete_from_swap_cache().
 130        (C) The page has been charged before (2) and reuse_swap_page() doesn't
 131            call delete_from_swap_cache().
 132        (D) The page has been charged before (2) and reuse_swap_page() calls
 133            delete_from_swap_cache().
 134
 135            memory.usage/memsw.usage changes to this page/swp_entry will be
 136         Case          (A)      (B)       (C)     (D)
 137         Event
 138       Before (2)     0/ 1     0/ 1      1/ 1    1/ 1
 139          ===========================================
 140          (3)        +1/+1    +1/+1     +1/+1   +1/+1
 141          (4)          -       0/ 0       -     -1/ 0
 142          (5)         0/-1     0/ 0     -1/-1    0/ 0
 143          (6)          -       0/-1       -      0/-1
 144          ===========================================
 145       Result         1/ 1     1/ 1      1/ 1    1/ 1
 146
 147       In any cases, charges to this page should be 1/ 1.
 148
 149        4.2 Swap-out.
 150        At swap-out, typical state transition is below.
 151
 152        (a) add to swap cache. (marked as SwapCache)
 153            swp_entry's refcnt += 1.
 154        (b) fully unmapped.
 155            swp_entry's refcnt += # of ptes.
 156        (c) write back to swap.
 157        (d) delete from swap cache. (remove from SwapCache)
 158            swp_entry's refcnt -= 1.
 159
 160
 161        At (b), the page is marked as SwapCache and not uncharged.
 162        At (d), the page is removed from SwapCache and a charge in page_cgroup
 163        is moved to swap_cgroup.
 164
 165        Finally, at task exit,
 166        (e) zap_pte() is called and swp_entry's refcnt -=1 -> 0.
 167        Here, a charge in swap_cgroup disappears.
 168
 1695. Page Cache
 170        Page Cache is charged at
 171        - add_to_page_cache_locked().
 172
 173        uncharged at
 174        - __remove_from_page_cache().
 175
 176        The logic is very clear. (About migration, see below)
 177        Note: __remove_from_page_cache() is called by remove_from_page_cache()
 178        and __remove_mapping().
 179
 1806. Shmem(tmpfs) Page Cache
 181        Memcg's charge/uncharge have special handlers of shmem. The best way
 182        to understand shmem's page state transition is to read mm/shmem.c.
 183        But brief explanation of the behavior of memcg around shmem will be
 184        helpful to understand the logic.
 185
 186        Shmem's page (just leaf page, not direct/indirect block) can be on
 187                - radix-tree of shmem's inode.
 188                - SwapCache.
 189                - Both on radix-tree and SwapCache. This happens at swap-in
 190                  and swap-out,
 191
 192        It's charged when...
 193        - A new page is added to shmem's radix-tree.
 194        - A swp page is read. (move a charge from swap_cgroup to page_cgroup)
 195        It's uncharged when
 196        - A page is removed from radix-tree and not SwapCache.
 197        - When SwapCache is removed, a charge is moved to swap_cgroup.
 198        - When swp_entry's refcnt goes down to 0, a charge in swap_cgroup
 199          disappears.
 200
 2017. Page Migration
 202        One of the most complicated functions is page-migration-handler.
 203        Memcg has 2 routines. Assume that we are migrating a page's contents
 204        from OLDPAGE to NEWPAGE.
 205
 206        Usual migration logic is..
 207        (a) remove the page from LRU.
 208        (b) allocate NEWPAGE (migration target)
 209        (c) lock by lock_page().
 210        (d) unmap all mappings.
 211        (e-1) If necessary, replace entry in radix-tree.
 212        (e-2) move contents of a page.
 213        (f) map all mappings again.
 214        (g) pushback the page to LRU.
 215        (-) OLDPAGE will be freed.
 216
 217        Before (g), memcg should complete all necessary charge/uncharge to
 218        NEWPAGE/OLDPAGE.
 219
 220        The point is....
 221        - If OLDPAGE is anonymous, all charges will be dropped at (d) because
 222          try_to_unmap() drops all mapcount and the page will not be
 223          SwapCache.
 224
 225        - If OLDPAGE is SwapCache, charges will be kept at (g) because
 226          __delete_from_swap_cache() isn't called at (e-1)
 227
 228        - If OLDPAGE is page-cache, charges will be kept at (g) because
 229          remove_from_swap_cache() isn't called at (e-1)
 230
 231        memcg provides following hooks.
 232
 233        - mem_cgroup_prepare_migration(OLDPAGE)
 234          Called after (b) to account a charge (usage += PAGE_SIZE) against
 235          memcg which OLDPAGE belongs to.
 236
 237        - mem_cgroup_end_migration(OLDPAGE, NEWPAGE)
 238          Called after (f) before (g).
 239          If OLDPAGE is used, commit OLDPAGE again. If OLDPAGE is already
 240          charged, a charge by prepare_migration() is automatically canceled.
 241          If NEWPAGE is used, commit NEWPAGE and uncharge OLDPAGE.
 242
 243          But zap_pte() (by exit or munmap) can be called while migration,
 244          we have to check if OLDPAGE/NEWPAGE is a valid page after commit().
 245
 2468. LRU
 247        Each memcg has its own private LRU. Now, it's handling is under global
 248        VM's control (means that it's handled under global zone->lru_lock).
 249        Almost all routines around memcg's LRU is called by global LRU's
 250        list management functions under zone->lru_lock().
 251
 252        A special function is mem_cgroup_isolate_pages(). This scans
 253        memcg's private LRU and call __isolate_lru_page() to extract a page
 254        from LRU.
 255        (By __isolate_lru_page(), the page is removed from both of global and
 256         private LRU.)
 257
 258
 2599. Typical Tests.
 260
 261 Tests for racy cases.
 262
 263 9.1 Small limit to memcg.
 264        When you do test to do racy case, it's good test to set memcg's limit
 265        to be very small rather than GB. Many races found in the test under
 266        xKB or xxMB limits.
 267        (Memory behavior under GB and Memory behavior under MB shows very
 268         different situation.)
 269
 270 9.2 Shmem
 271        Historically, memcg's shmem handling was poor and we saw some amount
 272        of troubles here. This is because shmem is page-cache but can be
 273        SwapCache. Test with shmem/tmpfs is always good test.
 274
 275 9.3 Migration
 276        For NUMA, migration is an another special case. To do easy test, cpuset
 277        is useful. Following is a sample script to do migration.
 278
 279        mount -t cgroup -o cpuset none /opt/cpuset
 280
 281        mkdir /opt/cpuset/01
 282        echo 1 > /opt/cpuset/01/cpuset.cpus
 283        echo 0 > /opt/cpuset/01/cpuset.mems
 284        echo 1 > /opt/cpuset/01/cpuset.memory_migrate
 285        mkdir /opt/cpuset/02
 286        echo 1 > /opt/cpuset/02/cpuset.cpus
 287        echo 1 > /opt/cpuset/02/cpuset.mems
 288        echo 1 > /opt/cpuset/02/cpuset.memory_migrate
 289
 290        In above set, when you moves a task from 01 to 02, page migration to
 291        node 0 to node 1 will occur. Following is a script to migrate all
 292        under cpuset.
 293        --
 294        move_task()
 295        {
 296        for pid in $1
 297        do
 298                /bin/echo $pid >$2/tasks 2>/dev/null
 299                echo -n $pid
 300                echo -n " "
 301        done
 302        echo END
 303        }
 304
 305        G1_TASK=`cat ${G1}/tasks`
 306        G2_TASK=`cat ${G2}/tasks`
 307        move_task "${G1_TASK}" ${G2} &
 308        --
 309 9.4 Memory hotplug.
 310        memory hotplug test is one of good test.
 311        to offline memory, do following.
 312        # echo offline > /sys/devices/system/memory/memoryXXX/state
 313        (XXX is the place of memory)
 314        This is an easy way to test page migration, too.
 315
 316 9.5 mkdir/rmdir
 317        When using hierarchy, mkdir/rmdir test should be done.
 318        Use tests like the following.
 319
 320        echo 1 >/opt/cgroup/01/memory/use_hierarchy
 321        mkdir /opt/cgroup/01/child_a
 322        mkdir /opt/cgroup/01/child_b
 323
 324        set limit to 01.
 325        add limit to 01/child_b
 326        run jobs under child_a and child_b
 327
 328        create/delete following groups at random while jobs are running.
 329        /opt/cgroup/01/child_a/child_aa
 330        /opt/cgroup/01/child_b/child_bb
 331        /opt/cgroup/01/child_c
 332
 333        running new jobs in new group is also good.
 334
 335 9.6 Mount with other subsystems.
 336        Mounting with other subsystems is a good test because there is a
 337        race and lock dependency with other cgroup subsystems.
 338
 339        example)
 340        # mount -t cgroup none /cgroup -t cpuset,memory,cpu,devices
 341
 342        and do task move, mkdir, rmdir etc...under this.
 343
 344 9.7 swapoff.
 345        Besides management of swap is one of complicated parts of memcg,
 346        call path of swap-in at swapoff is not same as usual swap-in path..
 347        It's worth to be tested explicitly.
 348
 349        For example, test like following is good.
 350        (Shell-A)
 351        # mount -t cgroup none /cgroup -t memory
 352        # mkdir /cgroup/test
 353        # echo 40M > /cgroup/test/memory.limit_in_bytes
 354        # echo 0 > /cgroup/test/tasks
 355        Run malloc(100M) program under this. You'll see 60M of swaps.
 356        (Shell-B)
 357        # move all tasks in /cgroup/test to /cgroup
 358        # /sbin/swapoff -a
 359        # rmdir /cgroup/test
 360        # kill malloc task.
 361
 362        Of course, tmpfs v.s. swapoff test should be tested, too.
 363
 364 9.8 OOM-Killer
 365        Out-of-memory caused by memcg's limit will kill tasks under
 366        the memcg. When hierarchy is used, a task under hierarchy
 367        will be killed by the kernel.
 368        In this case, panic_on_oom shouldn't be invoked and tasks
 369        in other groups shouldn't be killed.
 370
 371        It's not difficult to cause OOM under memcg as following.
 372        Case A) when you can swapoff
 373        #swapoff -a
 374        #echo 50M > /memory.limit_in_bytes
 375        run 51M of malloc
 376
 377        Case B) when you use mem+swap limitation.
 378        #echo 50M > memory.limit_in_bytes
 379        #echo 50M > memory.memsw.limit_in_bytes
 380        run 51M of malloc
 381
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.