1Freezing of tasks
   2        (C) 2007 Rafael J. Wysocki <>, GPL
   4I. What is the freezing of tasks?
   6The freezing of tasks is a mechanism by which user space processes and some
   7kernel threads are controlled during hibernation or system-wide suspend (on some
  10II. How does it work?
  12There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
  13and PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that have
  14PF_NOFREEZE unset (all user space processes and some kernel threads) are
  15regarded as 'freezable' and treated in a special way before the system enters a
  16suspend state as well as before a hibernation image is created (in what follows
  17we only consider hibernation, but the description also applies to suspend).
  19Namely, as the first step of the hibernation procedure the function
  20freeze_processes() (defined in kernel/power/process.c) is called.  A system-wide
  21variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
  22whether the system is to undergo a freezing operation. And freeze_processes()
  23sets this variable.  After this, it executes try_to_freeze_tasks() that sends a
  24fake signal to all user space processes, and wakes up all the kernel threads.
  25All freezable tasks must react to that by calling try_to_freeze(), which
  26results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
  27the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
  28it loop until PF_FROZEN is cleared for it. Then, we say that the task is
  29'frozen' and therefore the set of functions handling this mechanism is referred
  30to as 'the freezer' (these functions are defined in kernel/power/process.c,
  31kernel/freezer.c & include/linux/freezer.h). User space processes are generally
  32frozen before kernel threads.
  34__refrigerator() must not be called directly.  Instead, use the
  35try_to_freeze() function (defined in include/linux/freezer.h), that checks
  36if the task is to be frozen and makes the task enter __refrigerator().
  38For user space processes try_to_freeze() is called automatically from the
  39signal-handling code, but the freezable kernel threads need to call it
  40explicitly in suitable places or use the wait_event_freezable() or
  41wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
  42that combine interruptible sleep with checking if the task is to be frozen and
  43calling try_to_freeze().  The main loop of a freezable kernel thread may look
  44like the following one:
  46        set_freezable();
  47        do {
  48                hub_events();
  49                wait_event_freezable(khubd_wait,
  50                                !list_empty(&hub_event_list) ||
  51                                kthread_should_stop());
  52        } while (!kthread_should_stop() || !list_empty(&hub_event_list));
  54(from drivers/usb/core/hub.c::hub_thread()).
  56If a freezable kernel thread fails to call try_to_freeze() after the freezer has
  57initiated a freezing operation, the freezing of tasks will fail and the entire
  58hibernation operation will be cancelled.  For this reason, freezable kernel
  59threads must call try_to_freeze() somewhere or use one of the
  60wait_event_freezable() and wait_event_freezable_timeout() macros.
  62After the system memory state has been restored from a hibernation image and
  63devices have been reinitialized, the function thaw_processes() is called in
  64order to clear the PF_FROZEN flag for each frozen task.  Then, the tasks that
  65have been frozen leave __refrigerator() and continue running.
  68Rationale behind the functions dealing with freezing and thawing of tasks:
  72  - freezes only userspace tasks
  75  - freezes all tasks (including kernel threads) because we can't freeze
  76    kernel threads without freezing userspace tasks
  79  - thaws only kernel threads; this is particularly useful if we need to do
  80    anything special in between thawing of kernel threads and thawing of
  81    userspace tasks, or if we want to postpone the thawing of userspace tasks
  84  - thaws all tasks (including kernel threads) because we can't thaw userspace
  85    tasks without thawing kernel threads
  88III. Which kernel threads are freezable?
  90Kernel threads are not freezable by default.  However, a kernel thread may clear
  91PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
  92directly is not allowed).  From this point it is regarded as freezable
  93and must call try_to_freeze() in a suitable place.
  95IV. Why do we do that?
  97Generally speaking, there is a couple of reasons to use the freezing of tasks:
  991. The principal reason is to prevent filesystems from being damaged after
 100hibernation.  At the moment we have no simple means of checkpointing
 101filesystems, so if there are any modifications made to filesystem data and/or
 102metadata on disks, we cannot bring them back to the state from before the
 103modifications.  At the same time each hibernation image contains some
 104filesystem-related information that must be consistent with the state of the
 105on-disk data and metadata after the system memory state has been restored from
 106the image (otherwise the filesystems will be damaged in a nasty way, usually
 107making them almost impossible to repair).  We therefore freeze tasks that might
 108cause the on-disk filesystems' data and metadata to be modified after the
 109hibernation image has been created and before the system is finally powered off.
 110The majority of these are user space processes, but if any of the kernel threads
 111may cause something like this to happen, they have to be freezable.
 1132. Next, to create the hibernation image we need to free a sufficient amount of
 114memory (approximately 50% of available RAM) and we need to do that before
 115devices are deactivated, because we generally need them for swapping out.  Then,
 116after the memory for the image has been freed, we don't want tasks to allocate
 117additional memory and we prevent them from doing that by freezing them earlier.
 118[Of course, this also means that device drivers should not allocate substantial
 119amounts of memory from their .suspend() callbacks before hibernation, but this
 120is a separate issue.]
 1223. The third reason is to prevent user space processes and some kernel threads
 123from interfering with the suspending and resuming of devices.  A user space
 124process running on a second CPU while we are suspending devices may, for
 125example, be troublesome and without the freezing of tasks we would need some
 126safeguards against race conditions that might occur in such a case.
 128Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
 129of the discussions on LKML (
 131"RJW:> Why we freeze tasks at all or why we freeze kernel threads?
 133Linus: In many ways, 'at all'.
 135I _do_ realize the IO request queue issues, and that we cannot actually do
 136s2ram with some devices in the middle of a DMA.  So we want to be able to
 137avoid *that*, there's no question about that.  And I suspect that stopping
 138user threads and then waiting for a sync is practically one of the easier
 139ways to do so.
 141So in practice, the 'at all' may become a 'why freeze kernel threads?' and
 142freezing user threads I don't find really objectionable."
 144Still, there are kernel threads that may want to be freezable.  For example, if
 145a kernel thread that belongs to a device driver accesses the device directly, it
 146in principle needs to know when the device is suspended, so that it doesn't try
 147to access it at that time.  However, if the kernel thread is freezable, it will
 148be frozen before the driver's .suspend() callback is executed and it will be
 149thawed after the driver's .resume() callback has run, so it won't be accessing
 150the device while it's suspended.
 1524. Another reason for freezing tasks is to prevent user space processes from
 153realizing that hibernation (or suspend) operation takes place.  Ideally, user
 154space processes should not notice that such a system-wide operation has occurred
 155and should continue running without any problems after the restore (or resume
 156from suspend).  Unfortunately, in the most general case this is quite difficult
 157to achieve without the freezing of tasks.  Consider, for example, a process
 158that depends on all CPUs being online while it's running.  Since we need to
 159disable nonboot CPUs during the hibernation, if this process is not frozen, it
 160may notice that the number of CPUs has changed and may start to work incorrectly
 161because of that.
 163V. Are there any problems related to the freezing of tasks?
 165Yes, there are.
 167First of all, the freezing of kernel threads may be tricky if they depend one
 168on another.  For example, if kernel thread A waits for a completion (in the
 169TASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread B
 170and B is frozen in the meantime, then A will be blocked until B is thawed, which
 171may be undesirable.  That's why kernel threads are not freezable by default.
 173Second, there are the following two problems related to the freezing of user
 174space processes:
 1751. Putting processes into an uninterruptible sleep distorts the load average.
 1762. Now that we have FUSE, plus the framework for doing device drivers in
 177userspace, it gets even more complicated because some userspace processes are
 178now doing the sorts of things that kernel threads do
 181The problem 1. seems to be fixable, although it hasn't been fixed so far.  The
 182other one is more serious, but it seems that we can work around it by using
 183hibernation (and suspend) notifiers (in that case, though, we won't be able to
 184avoid the realization by the user space processes that the hibernation is taking
 187There are also problems that the freezing of tasks tends to expose, although
 188they are not directly related to it.  For example, if request_firmware() is
 189called from a device driver's .resume() routine, it will timeout and eventually
 190fail, because the user land process that should respond to the request is frozen
 191at this point.  So, seemingly, the failure is due to the freezing of tasks.
 192Suppose, however, that the firmware file is located on a filesystem accessible
 193only through another device that hasn't been resumed yet.  In that case,
 194request_firmware() will fail regardless of whether or not the freezing of tasks
 195is used.  Consequently, the problem is not really related to the freezing of
 196tasks, since it generally exists anyway.
 198A driver must have all firmwares it may need in RAM before suspend() is called.
 199If keeping them is not practical, for example due to their size, they must be
 200requested early enough using the suspend notifier API described in notifiers.txt.
 202VI. Are there any precautions to be taken to prevent freezing failures?
 204Yes, there are.
 206First of all, grabbing the 'pm_mutex' lock to mutually exclude a piece of code
 207from system-wide sleep such as suspend/hibernation is not encouraged.
 208If possible, that piece of code must instead hook onto the suspend/hibernation
 209notifiers to achieve mutual exclusion. Look at the CPU-Hotplug code
 210(kernel/cpu.c) for an example.
 212However, if that is not feasible, and grabbing 'pm_mutex' is deemed necessary,
 213it is strongly discouraged to directly call mutex_[un]lock(&pm_mutex) since
 214that could lead to freezing failures, because if the suspend/hibernate code
 215successfully acquired the 'pm_mutex' lock, and hence that other entity failed
 216to acquire the lock, then that task would get blocked in TASK_UNINTERRUPTIBLE
 217state. As a consequence, the freezer would not be able to freeze that task,
 218leading to freezing failure.
 220However, the [un]lock_system_sleep() APIs are safe to use in this scenario,
 221since they ask the freezer to skip freezing this task, since it is anyway
 222"frozen enough" as it is blocked on 'pm_mutex', which will be released
 223only after the entire suspend/hibernation sequence is complete.
 224So, to summarize, use [un]lock_system_sleep() instead of directly using
 225mutex_[un]lock(&pm_mutex). That would prevent freezing failures.
 227V. Miscellaneous
 228/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
 229all user space processes or all freezable kernel threads, in unit of millisecond.
 230The default value is 20000, with range of unsigned integer.