3Linux kernel development in the early 1990's was a pretty loose affair,
   4with relatively small numbers of users and developers involved.  With a
   5user base in the millions and with some 2,000 developers involved over the
   6course of one year, the kernel has since had to evolve a number of
   7processes to keep development happening smoothly.  A solid understanding of
   8how the process works is required in order to be an effective part of it.
  13The kernel developers use a loosely time-based release process, with a new
  14major kernel release happening every two or three months.  The recent
  15release history looks like this:
  17        2.6.26  July 13, 2008
  18        2.6.25  April 16, 2008
  19        2.6.24  January 24, 2008
  20        2.6.23  October 9, 2007
  21        2.6.22  July 8, 2007
  22        2.6.21  April 25, 2007
  23        2.6.20  February 4, 2007
  25Every 2.6.x release is a major kernel release with new features, internal
  26API changes, and more.  A typical 2.6 release can contain over 10,000
  27changesets with changes to several hundred thousand lines of code.  2.6 is
  28thus the leading edge of Linux kernel development; the kernel uses a
  29rolling development model which is continually integrating major changes.
  31A relatively straightforward discipline is followed with regard to the
  32merging of patches for each release.  At the beginning of each development
  33cycle, the "merge window" is said to be open.  At that time, code which is
  34deemed to be sufficiently stable (and which is accepted by the development
  35community) is merged into the mainline kernel.  The bulk of changes for a
  36new development cycle (and all of the major changes) will be merged during
  37this time, at a rate approaching 1,000 changes ("patches," or "changesets")
  38per day.
  40(As an aside, it is worth noting that the changes integrated during the
  41merge window do not come out of thin air; they have been collected, tested,
  42and staged ahead of time.  How that process works will be described in
  43detail later on).
  45The merge window lasts for two weeks.  At the end of this time, Linus
  46Torvalds will declare that the window is closed and release the first of
  47the "rc" kernels.  For the kernel which is destined to be 2.6.26, for
  48example, the release which happens at the end of the merge window will be
  49called 2.6.26-rc1.  The -rc1 release is the signal that the time to merge
  50new features has passed, and that the time to stabilize the next kernel has
  53Over the next six to ten weeks, only patches which fix problems should be
  54submitted to the mainline.  On occasion a more significant change will be
  55allowed, but such occasions are rare; developers who try to merge new
  56features outside of the merge window tend to get an unfriendly reception.
  57As a general rule, if you miss the merge window for a given feature, the
  58best thing to do is to wait for the next development cycle.  (An occasional
  59exception is made for drivers for previously-unsupported hardware; if they
  60touch no in-tree code, they cannot cause regressions and should be safe to
  61add at any time).
  63As fixes make their way into the mainline, the patch rate will slow over
  64time.  Linus releases new -rc kernels about once a week; a normal series
  65will get up to somewhere between -rc6 and -rc9 before the kernel is
  66considered to be sufficiently stable and the final 2.6.x release is made.
  67At that point the whole process starts over again.
  69As an example, here is how the 2.6.25 development cycle went (all dates in
  72        January 24      2.6.24 stable release
  73        February 10     2.6.25-rc1, merge window closes
  74        February 15     2.6.25-rc2
  75        February 24     2.6.25-rc3
  76        March 4         2.6.25-rc4
  77        March 9         2.6.25-rc5
  78        March 16        2.6.25-rc6
  79        March 25        2.6.25-rc7
  80        April 1         2.6.25-rc8
  81        April 11        2.6.25-rc9
  82        April 16        2.6.25 stable release
  84How do the developers decide when to close the development cycle and create
  85the stable release?  The most significant metric used is the list of
  86regressions from previous releases.  No bugs are welcome, but those which
  87break systems which worked in the past are considered to be especially
  88serious.  For this reason, patches which cause regressions are looked upon
  89unfavorably and are quite likely to be reverted during the stabilization
  92The developers' goal is to fix all known regressions before the stable
  93release is made.  In the real world, this kind of perfection is hard to
  94achieve; there are just too many variables in a project of this size.
  95There comes a point where delaying the final release just makes the problem
  96worse; the pile of changes waiting for the next merge window will grow
  97larger, creating even more regressions the next time around.  So most 2.6.x
  98kernels go out with a handful of known regressions though, hopefully, none
  99of them are serious.
 101Once a stable release is made, its ongoing maintenance is passed off to the
 102"stable team," currently comprised of Greg Kroah-Hartman and Chris Wright.
 103The stable team will release occasional updates to the stable release using
 104the 2.6.x.y numbering scheme.  To be considered for an update release, a
 105patch must (1) fix a significant bug, and (2) already be merged into the
 106mainline for the next development kernel.  Continuing our 2.6.25 example,
 107the history (as of this writing) is:
 109        May 1 
 110        May 6  
 111        May 9  
 112        May 15
 113        June 7
 114        June 9
 115        June 16
 116        June 21
 117        June 24
 119Stable updates for a given kernel are made for approximately six months;
 120after that, the maintenance of stable releases is solely the responsibility
 121of the distributors which have shipped that particular kernel.
 126Patches do not go directly from the developer's keyboard into the mainline
 127kernel.  There is, instead, a somewhat involved (if somewhat informal)
 128process designed to ensure that each patch is reviewed for quality and that
 129each patch implements a change which is desirable to have in the mainline.
 130This process can happen quickly for minor fixes, or, in the case of large
 131and controversial changes, go on for years.  Much developer frustration
 132comes from a lack of understanding of this process or from attempts to
 133circumvent it.  
 135In the hopes of reducing that frustration, this document will describe how
 136a patch gets into the kernel.  What follows below is an introduction which
 137describes the process in a somewhat idealized way.  A much more detailed
 138treatment will come in later sections.
 140The stages that a patch goes through are, generally:
 142 - Design.  This is where the real requirements for the patch - and the way
 143   those requirements will be met - are laid out.  Design work is often
 144   done without involving the community, but it is better to do this work
 145   in the open if at all possible; it can save a lot of time redesigning
 146   things later.
 148 - Early review.  Patches are posted to the relevant mailing list, and
 149   developers on that list reply with any comments they may have.  This
 150   process should turn up any major problems with a patch if all goes
 151   well.
 153 - Wider review.  When the patch is getting close to ready for mainline
 154   inclusion, it should be accepted by a relevant subsystem maintainer -
 155   though this acceptance is not a guarantee that the patch will make it
 156   all the way to the mainline.  The patch will show up in the maintainer's
 157   subsystem tree and into the -next trees (described below).  When the
 158   process works, this step leads to more extensive review of the patch and
 159   the discovery of any problems resulting from the integration of this
 160   patch with work being done by others.
 162-  Please note that most maintainers also have day jobs, so merging
 163   your patch may not be their highest priority.  If your patch is
 164   getting feedback about changes that are needed, you should either
 165   make those changes or justify why they should not be made.  If your
 166   patch has no review complaints but is not being merged by its
 167   appropriate subsystem or driver maintainer, you should be persistent
 168   in updating the patch to the current kernel so that it applies cleanly
 169   and keep sending it for review and merging.
 171 - Merging into the mainline.  Eventually, a successful patch will be
 172   merged into the mainline repository managed by Linus Torvalds.  More
 173   comments and/or problems may surface at this time; it is important that
 174   the developer be responsive to these and fix any issues which arise.
 176 - Stable release.  The number of users potentially affected by the patch
 177   is now large, so, once again, new problems may arise.
 179 - Long-term maintenance.  While it is certainly possible for a developer
 180   to forget about code after merging it, that sort of behavior tends to
 181   leave a poor impression in the development community.  Merging code
 182   eliminates some of the maintenance burden, in that others will fix
 183   problems caused by API changes.  But the original developer should
 184   continue to take responsibility for the code if it is to remain useful
 185   in the longer term.
 187One of the largest mistakes made by kernel developers (or their employers)
 188is to try to cut the process down to a single "merging into the mainline"
 189step.  This approach invariably leads to frustration for everybody
 195There is exactly one person who can merge patches into the mainline kernel
 196repository: Linus Torvalds.  But, of the over 12,000 patches which went
 197into the 2.6.25 kernel, only 250 (around 2%) were directly chosen by Linus
 198himself.  The kernel project has long since grown to a size where no single
 199developer could possibly inspect and select every patch unassisted.  The
 200way the kernel developers have addressed this growth is through the use of
 201a lieutenant system built around a chain of trust.
 203The kernel code base is logically broken down into a set of subsystems:
 204networking, specific architecture support, memory management, video
 205devices, etc.  Most subsystems have a designated maintainer, a developer
 206who has overall responsibility for the code within that subsystem.  These
 207subsystem maintainers are the gatekeepers (in a loose way) for the portion
 208of the kernel they manage; they are the ones who will (usually) accept a
 209patch for inclusion into the mainline kernel.
 211Subsystem maintainers each manage their own version of the kernel source
 212tree, usually (but certainly not always) using the git source management
 213tool.  Tools like git (and related tools like quilt or mercurial) allow
 214maintainers to track a list of patches, including authorship information
 215and other metadata.  At any given time, the maintainer can identify which
 216patches in his or her repository are not found in the mainline.
 218When the merge window opens, top-level maintainers will ask Linus to "pull"
 219the patches they have selected for merging from their repositories.  If
 220Linus agrees, the stream of patches will flow up into his repository,
 221becoming part of the mainline kernel.  The amount of attention that Linus
 222pays to specific patches received in a pull operation varies.  It is clear
 223that, sometimes, he looks quite closely.  But, as a general rule, Linus
 224trusts the subsystem maintainers to not send bad patches upstream.
 226Subsystem maintainers, in turn, can pull patches from other maintainers.
 227For example, the networking tree is built from patches which accumulated
 228first in trees dedicated to network device drivers, wireless networking,
 229etc.  This chain of repositories can be arbitrarily long, though it rarely
 230exceeds two or three links.  Since each maintainer in the chain trusts
 231those managing lower-level trees, this process is known as the "chain of
 234Clearly, in a system like this, getting patches into the kernel depends on
 235finding the right maintainer.  Sending patches directly to Linus is not
 236normally the right way to go.
 2392.4: NEXT TREES
 241The chain of subsystem trees guides the flow of patches into the kernel,
 242but it also raises an interesting question: what if somebody wants to look
 243at all of the patches which are being prepared for the next merge window?
 244Developers will be interested in what other changes are pending to see
 245whether there are any conflicts to worry about; a patch which changes a
 246core kernel function prototype, for example, will conflict with any other
 247patches which use the older form of that function.  Reviewers and testers
 248want access to the changes in their integrated form before all of those
 249changes land in the mainline kernel.  One could pull changes from all of
 250the interesting subsystem trees, but that would be a big and error-prone
 253The answer comes in the form of -next trees, where subsystem trees are
 254collected for testing and review.  The older of these trees, maintained by
 255Andrew Morton, is called "-mm" (for memory management, which is how it got
 256started).  The -mm tree integrates patches from a long list of subsystem
 257trees; it also has some patches aimed at helping with debugging.  
 259Beyond that, -mm contains a significant collection of patches which have
 260been selected by Andrew directly.  These patches may have been posted on a
 261mailing list, or they may apply to a part of the kernel for which there is
 262no designated subsystem tree.  As a result, -mm operates as a sort of
 263subsystem tree of last resort; if there is no other obvious path for a
 264patch into the mainline, it is likely to end up in -mm.  Miscellaneous
 265patches which accumulate in -mm will eventually either be forwarded on to
 266an appropriate subsystem tree or be sent directly to Linus.  In a typical
 267development cycle, approximately 10% of the patches going into the mainline
 268get there via -mm.
 270The current -mm patch is available in the "mmotm" (-mm of the moment)
 271directory at:
 275Use of the MMOTM tree is likely to be a frustrating experience, though;
 276there is a definite chance that it will not even compile.
 278The other -next tree, started more recently, is linux-next, maintained by
 279Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
 280the mainline is expected to look like after the next merge window closes.
 281Linux-next trees are announced on the linux-kernel and linux-next mailing
 282lists when they are assembled; they can be downloaded from:
 286Some information about linux-next has been gathered at:
 290How the linux-next tree will fit into the development process is still
 291changing.  As of this writing, the first full development cycle involving
 292linux-next (2.6.26) is coming to an end; thus far, it has proved to be a
 293valuable resource for finding and fixing integration problems before the
 294beginning of the merge window.  See for
 295more information on how linux-next has worked to set up the 2.6.27 merge
 298Some developers have begun to suggest that linux-next should be used as the
 299target for future development as well.  The linux-next tree does tend to be
 300far ahead of the mainline and is more representative of the tree into which
 301any new work will be merged.  The downside to this idea is that the
 302volatility of linux-next tends to make it a difficult development target.
 303See for more information on this topic, and
 304stay tuned; much is still in flux where linux-next is involved.
 3062.4.1: STAGING TREES
 308The kernel source tree now contains the drivers/staging/ directory, where
 309many sub-directories for drivers or filesystems that are on their way to
 310being added to the kernel tree live.  They remain in drivers/staging while
 311they still need more work; once complete, they can be moved into the
 312kernel proper.  This is a way to keep track of drivers that aren't
 313up to Linux kernel coding or quality standards, but people may want to use
 314them and track development.
 316Greg Kroah-Hartman currently (as of 2.6.36) maintains the staging tree.
 317Drivers that still need work are sent to him, with each driver having
 318its own subdirectory in drivers/staging/.  Along with the driver source
 319files, a TODO file should be present in the directory as well.  The TODO
 320file lists the pending work that the driver needs for acceptance into
 321the kernel proper, as well as a list of people that should be Cc'd for any
 322patches to the driver.  Staging drivers that don't currently build should
 323have their config entries depend upon CONFIG_BROKEN.  Once they can
 324be successfully built without outside patches, CONFIG_BROKEN can be removed.
 3262.5: TOOLS
 328As can be seen from the above text, the kernel development process depends
 329heavily on the ability to herd collections of patches in various
 330directions.  The whole thing would not work anywhere near as well as it
 331does without suitably powerful tools.  Tutorials on how to use these tools
 332are well beyond the scope of this document, but there is space for a few
 335By far the dominant source code management system used by the kernel
 336community is git.  Git is one of a number of distributed version control
 337systems being developed in the free software community.  It is well tuned
 338for kernel development, in that it performs quite well when dealing with
 339large repositories and large numbers of patches.  It also has a reputation
 340for being difficult to learn and use, though it has gotten better over
 341time.  Some sort of familiarity with git is almost a requirement for kernel
 342developers; even if they do not use it for their own work, they'll need git
 343to keep up with what other developers (and the mainline) are doing.
 345Git is now packaged by almost all Linux distributions.  There is a home
 346page at:
 350That page has pointers to documentation and tutorials.  One should be
 351aware, in particular, of the Kernel Hacker's Guide to git, which has
 352information specific to kernel development:
 356Among the kernel developers who do not use git, the most popular choice is
 357almost certainly Mercurial:
 361Mercurial shares many features with git, but it provides an interface which
 362many find easier to use.
 364The other tool worth knowing about is Quilt:
 368Quilt is a patch management system, rather than a source code management
 369system.  It does not track history over time; it is, instead, oriented
 370toward tracking a specific set of changes against an evolving code base.
 371Some major subsystem maintainers use quilt to manage patches intended to go
 372upstream.  For the management of certain kinds of trees (-mm, for example),
 373quilt is the best tool for the job.
 378A great deal of Linux kernel development work is done by way of mailing
 379lists.  It is hard to be a fully-functioning member of the community
 380without joining at least one list somewhere.  But Linux mailing lists also
 381represent a potential hazard to developers, who risk getting buried under a
 382load of electronic mail, running afoul of the conventions used on the Linux
 383lists, or both.
 385Most kernel mailing lists are run on; the master list can
 386be found at:
 390There are lists hosted elsewhere, though; a number of them are at
 393The core mailing list for kernel development is, of course, linux-kernel.
 394This list is an intimidating place to be; volume can reach 500 messages per
 395day, the amount of noise is high, the conversation can be severely
 396technical, and participants are not always concerned with showing a high
 397degree of politeness.  But there is no other place where the kernel
 398development community comes together as a whole; developers who avoid this
 399list will miss important information.
 401There are a few hints which can help with linux-kernel survival:
 403- Have the list delivered to a separate folder, rather than your main
 404  mailbox.  One must be able to ignore the stream for sustained periods of
 405  time.
 407- Do not try to follow every conversation - nobody else does.  It is
 408  important to filter on both the topic of interest (though note that
 409  long-running conversations can drift away from the original subject
 410  without changing the email subject line) and the people who are
 411  participating.  
 413- Do not feed the trolls.  If somebody is trying to stir up an angry
 414  response, ignore them.
 416- When responding to linux-kernel email (or that on other lists) preserve
 417  the Cc: header for all involved.  In the absence of a strong reason (such
 418  as an explicit request), you should never remove recipients.  Always make
 419  sure that the person you are responding to is in the Cc: list.  This
 420  convention also makes it unnecessary to explicitly ask to be copied on
 421  replies to your postings.
 423- Search the list archives (and the net as a whole) before asking
 424  questions.  Some developers can get impatient with people who clearly
 425  have not done their homework.
 427- Avoid top-posting (the practice of putting your answer above the quoted
 428  text you are responding to).  It makes your response harder to read and
 429  makes a poor impression.
 431- Ask on the correct mailing list.  Linux-kernel may be the general meeting
 432  point, but it is not the best place to find developers from all
 433  subsystems.
 435The last point - finding the correct mailing list - is a common place for
 436beginning developers to go wrong.  Somebody who asks a networking-related
 437question on linux-kernel will almost certainly receive a polite suggestion
 438to ask on the netdev list instead, as that is the list frequented by most
 439networking developers.  Other lists exist for the SCSI, video4linux, IDE,
 440filesystem, etc. subsystems.  The best place to look for mailing lists is
 441in the MAINTAINERS file packaged with the kernel source.
 446Questions about how to get started with the kernel development process are
 447common - from both individuals and companies.  Equally common are missteps
 448which make the beginning of the relationship harder than it has to be.
 450Companies often look to hire well-known developers to get a development
 451group started.  This can, in fact, be an effective technique.  But it also
 452tends to be expensive and does not do much to grow the pool of experienced
 453kernel developers.  It is possible to bring in-house developers up to speed
 454on Linux kernel development, given the investment of a bit of time.  Taking
 455this time can endow an employer with a group of developers who understand
 456the kernel and the company both, and who can help to train others as well.
 457Over the medium term, this is often the more profitable approach.
 459Individual developers are often, understandably, at a loss for a place to
 460start.  Beginning with a large project can be intimidating; one often wants
 461to test the waters with something smaller first.  This is the point where
 462some developers jump into the creation of patches fixing spelling errors or
 463minor coding style issues.  Unfortunately, such patches create a level of
 464noise which is distracting for the development community as a whole, so,
 465increasingly, they are looked down upon.  New developers wishing to
 466introduce themselves to the community will not get the sort of reception
 467they wish for by these means.
 469Andrew Morton gives this advice for aspiring kernel developers
 471        The #1 project for all kernel beginners should surely be "make sure
 472        that the kernel runs perfectly at all times on all machines which
 473        you can lay your hands on".  Usually the way to do this is to work
 474        with others on getting things fixed up (this can require
 475        persistence!) but that's fine - it's a part of kernel development.
 479In the absence of obvious problems to fix, developers are advised to look
 480at the current lists of regressions and open bugs in general.  There is
 481never any shortage of issues in need of fixing; by addressing these issues,
 482developers will gain experience with the process while, at the same time,
 483building respect with the rest of the development community.
 484 kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.