linux/Documentation/development-process/2.Process
<<
>>
Prefs
   12: HOW THE DEVELOPMENT PROCESS WORKS
   2
   3Linux kernel development in the early 1990's was a pretty loose affair,
   4with relatively small numbers of users and developers involved.  With a
   5user base in the millions and with some 2,000 developers involved over the
   6course of one year, the kernel has since had to evolve a number of
   7processes to keep development happening smoothly.  A solid understanding of
   8how the process works is required in order to be an effective part of it.
   9
  10
  112.1: THE BIG PICTURE
  12
  13The kernel developers use a loosely time-based release process, with a new
  14major kernel release happening every two or three months.  The recent
  15release history looks like this:
  16
  17        2.6.38  March 14, 2011
  18        2.6.37  January 4, 2011
  19        2.6.36  October 20, 2010
  20        2.6.35  August 1, 2010
  21        2.6.34  May 15, 2010
  22        2.6.33  February 24, 2010
  23
  24Every 2.6.x release is a major kernel release with new features, internal
  25API changes, and more.  A typical 2.6 release can contain nearly 10,000
  26changesets with changes to several hundred thousand lines of code.  2.6 is
  27thus the leading edge of Linux kernel development; the kernel uses a
  28rolling development model which is continually integrating major changes.
  29
  30A relatively straightforward discipline is followed with regard to the
  31merging of patches for each release.  At the beginning of each development
  32cycle, the "merge window" is said to be open.  At that time, code which is
  33deemed to be sufficiently stable (and which is accepted by the development
  34community) is merged into the mainline kernel.  The bulk of changes for a
  35new development cycle (and all of the major changes) will be merged during
  36this time, at a rate approaching 1,000 changes ("patches," or "changesets")
  37per day.
  38
  39(As an aside, it is worth noting that the changes integrated during the
  40merge window do not come out of thin air; they have been collected, tested,
  41and staged ahead of time.  How that process works will be described in
  42detail later on).
  43
  44The merge window lasts for approximately two weeks.  At the end of this
  45time, Linus Torvalds will declare that the window is closed and release the
  46first of the "rc" kernels.  For the kernel which is destined to be 2.6.40,
  47for example, the release which happens at the end of the merge window will
  48be called 2.6.40-rc1.  The -rc1 release is the signal that the time to
  49merge new features has passed, and that the time to stabilize the next
  50kernel has begun.
  51
  52Over the next six to ten weeks, only patches which fix problems should be
  53submitted to the mainline.  On occasion a more significant change will be
  54allowed, but such occasions are rare; developers who try to merge new
  55features outside of the merge window tend to get an unfriendly reception.
  56As a general rule, if you miss the merge window for a given feature, the
  57best thing to do is to wait for the next development cycle.  (An occasional
  58exception is made for drivers for previously-unsupported hardware; if they
  59touch no in-tree code, they cannot cause regressions and should be safe to
  60add at any time).
  61
  62As fixes make their way into the mainline, the patch rate will slow over
  63time.  Linus releases new -rc kernels about once a week; a normal series
  64will get up to somewhere between -rc6 and -rc9 before the kernel is
  65considered to be sufficiently stable and the final 2.6.x release is made.
  66At that point the whole process starts over again.
  67
  68As an example, here is how the 2.6.38 development cycle went (all dates in
  692011):
  70
  71        January 4       2.6.37 stable release
  72        January 18      2.6.38-rc1, merge window closes
  73        January 21      2.6.38-rc2
  74        February 1      2.6.38-rc3
  75        February 7      2.6.38-rc4
  76        February 15     2.6.38-rc5
  77        February 21     2.6.38-rc6
  78        March 1         2.6.38-rc7
  79        March 7         2.6.38-rc8
  80        March 14        2.6.38 stable release
  81
  82How do the developers decide when to close the development cycle and create
  83the stable release?  The most significant metric used is the list of
  84regressions from previous releases.  No bugs are welcome, but those which
  85break systems which worked in the past are considered to be especially
  86serious.  For this reason, patches which cause regressions are looked upon
  87unfavorably and are quite likely to be reverted during the stabilization
  88period.
  89
  90The developers' goal is to fix all known regressions before the stable
  91release is made.  In the real world, this kind of perfection is hard to
  92achieve; there are just too many variables in a project of this size.
  93There comes a point where delaying the final release just makes the problem
  94worse; the pile of changes waiting for the next merge window will grow
  95larger, creating even more regressions the next time around.  So most 2.6.x
  96kernels go out with a handful of known regressions though, hopefully, none
  97of them are serious.
  98
  99Once a stable release is made, its ongoing maintenance is passed off to the
 100"stable team," currently consisting of Greg Kroah-Hartman.  The stable team
 101will release occasional updates to the stable release using the 2.6.x.y
 102numbering scheme.  To be considered for an update release, a patch must (1)
 103fix a significant bug, and (2) already be merged into the mainline for the
 104next development kernel.  Kernels will typically receive stable updates for
 105a little more than one development cycle past their initial release.  So,
 106for example, the 2.6.36 kernel's history looked like:
 107
 108        October 10      2.6.36 stable release
 109        November 22     2.6.36.1
 110        December 9      2.6.36.2
 111        January 7       2.6.36.3
 112        February 17     2.6.36.4
 113
 1142.6.36.4 was the final stable update for the 2.6.36 release.
 115
 116Some kernels are designated "long term" kernels; they will receive support
 117for a longer period.  As of this writing, the current long term kernels
 118and their maintainers are:
 119
 120        2.6.27  Willy Tarreau           (Deep-frozen stable kernel)
 121        2.6.32  Greg Kroah-Hartman
 122        2.6.35  Andi Kleen              (Embedded flag kernel)
 123
 124The selection of a kernel for long-term support is purely a matter of a
 125maintainer having the need and the time to maintain that release.  There
 126are no known plans for long-term support for any specific upcoming
 127release.
 128
 129
 1302.2: THE LIFECYCLE OF A PATCH
 131
 132Patches do not go directly from the developer's keyboard into the mainline
 133kernel.  There is, instead, a somewhat involved (if somewhat informal)
 134process designed to ensure that each patch is reviewed for quality and that
 135each patch implements a change which is desirable to have in the mainline.
 136This process can happen quickly for minor fixes, or, in the case of large
 137and controversial changes, go on for years.  Much developer frustration
 138comes from a lack of understanding of this process or from attempts to
 139circumvent it.
 140
 141In the hopes of reducing that frustration, this document will describe how
 142a patch gets into the kernel.  What follows below is an introduction which
 143describes the process in a somewhat idealized way.  A much more detailed
 144treatment will come in later sections.
 145
 146The stages that a patch goes through are, generally:
 147
 148 - Design.  This is where the real requirements for the patch - and the way
 149   those requirements will be met - are laid out.  Design work is often
 150   done without involving the community, but it is better to do this work
 151   in the open if at all possible; it can save a lot of time redesigning
 152   things later.
 153
 154 - Early review.  Patches are posted to the relevant mailing list, and
 155   developers on that list reply with any comments they may have.  This
 156   process should turn up any major problems with a patch if all goes
 157   well.
 158
 159 - Wider review.  When the patch is getting close to ready for mainline
 160   inclusion, it should be accepted by a relevant subsystem maintainer -
 161   though this acceptance is not a guarantee that the patch will make it
 162   all the way to the mainline.  The patch will show up in the maintainer's
 163   subsystem tree and into the -next trees (described below).  When the
 164   process works, this step leads to more extensive review of the patch and
 165   the discovery of any problems resulting from the integration of this
 166   patch with work being done by others.
 167
 168-  Please note that most maintainers also have day jobs, so merging
 169   your patch may not be their highest priority.  If your patch is
 170   getting feedback about changes that are needed, you should either
 171   make those changes or justify why they should not be made.  If your
 172   patch has no review complaints but is not being merged by its
 173   appropriate subsystem or driver maintainer, you should be persistent
 174   in updating the patch to the current kernel so that it applies cleanly
 175   and keep sending it for review and merging.
 176
 177 - Merging into the mainline.  Eventually, a successful patch will be
 178   merged into the mainline repository managed by Linus Torvalds.  More
 179   comments and/or problems may surface at this time; it is important that
 180   the developer be responsive to these and fix any issues which arise.
 181
 182 - Stable release.  The number of users potentially affected by the patch
 183   is now large, so, once again, new problems may arise.
 184
 185 - Long-term maintenance.  While it is certainly possible for a developer
 186   to forget about code after merging it, that sort of behavior tends to
 187   leave a poor impression in the development community.  Merging code
 188   eliminates some of the maintenance burden, in that others will fix
 189   problems caused by API changes.  But the original developer should
 190   continue to take responsibility for the code if it is to remain useful
 191   in the longer term.
 192
 193One of the largest mistakes made by kernel developers (or their employers)
 194is to try to cut the process down to a single "merging into the mainline"
 195step.  This approach invariably leads to frustration for everybody
 196involved.
 197
 198
 1992.3: HOW PATCHES GET INTO THE KERNEL
 200
 201There is exactly one person who can merge patches into the mainline kernel
 202repository: Linus Torvalds.  But, of the over 9,500 patches which went
 203into the 2.6.38 kernel, only 112 (around 1.3%) were directly chosen by Linus
 204himself.  The kernel project has long since grown to a size where no single
 205developer could possibly inspect and select every patch unassisted.  The
 206way the kernel developers have addressed this growth is through the use of
 207a lieutenant system built around a chain of trust.
 208
 209The kernel code base is logically broken down into a set of subsystems:
 210networking, specific architecture support, memory management, video
 211devices, etc.  Most subsystems have a designated maintainer, a developer
 212who has overall responsibility for the code within that subsystem.  These
 213subsystem maintainers are the gatekeepers (in a loose way) for the portion
 214of the kernel they manage; they are the ones who will (usually) accept a
 215patch for inclusion into the mainline kernel.
 216
 217Subsystem maintainers each manage their own version of the kernel source
 218tree, usually (but certainly not always) using the git source management
 219tool.  Tools like git (and related tools like quilt or mercurial) allow
 220maintainers to track a list of patches, including authorship information
 221and other metadata.  At any given time, the maintainer can identify which
 222patches in his or her repository are not found in the mainline.
 223
 224When the merge window opens, top-level maintainers will ask Linus to "pull"
 225the patches they have selected for merging from their repositories.  If
 226Linus agrees, the stream of patches will flow up into his repository,
 227becoming part of the mainline kernel.  The amount of attention that Linus
 228pays to specific patches received in a pull operation varies.  It is clear
 229that, sometimes, he looks quite closely.  But, as a general rule, Linus
 230trusts the subsystem maintainers to not send bad patches upstream.
 231
 232Subsystem maintainers, in turn, can pull patches from other maintainers.
 233For example, the networking tree is built from patches which accumulated
 234first in trees dedicated to network device drivers, wireless networking,
 235etc.  This chain of repositories can be arbitrarily long, though it rarely
 236exceeds two or three links.  Since each maintainer in the chain trusts
 237those managing lower-level trees, this process is known as the "chain of
 238trust."
 239
 240Clearly, in a system like this, getting patches into the kernel depends on
 241finding the right maintainer.  Sending patches directly to Linus is not
 242normally the right way to go.
 243
 244
 2452.4: NEXT TREES
 246
 247The chain of subsystem trees guides the flow of patches into the kernel,
 248but it also raises an interesting question: what if somebody wants to look
 249at all of the patches which are being prepared for the next merge window?
 250Developers will be interested in what other changes are pending to see
 251whether there are any conflicts to worry about; a patch which changes a
 252core kernel function prototype, for example, will conflict with any other
 253patches which use the older form of that function.  Reviewers and testers
 254want access to the changes in their integrated form before all of those
 255changes land in the mainline kernel.  One could pull changes from all of
 256the interesting subsystem trees, but that would be a big and error-prone
 257job.
 258
 259The answer comes in the form of -next trees, where subsystem trees are
 260collected for testing and review.  The older of these trees, maintained by
 261Andrew Morton, is called "-mm" (for memory management, which is how it got
 262started).  The -mm tree integrates patches from a long list of subsystem
 263trees; it also has some patches aimed at helping with debugging.
 264
 265Beyond that, -mm contains a significant collection of patches which have
 266been selected by Andrew directly.  These patches may have been posted on a
 267mailing list, or they may apply to a part of the kernel for which there is
 268no designated subsystem tree.  As a result, -mm operates as a sort of
 269subsystem tree of last resort; if there is no other obvious path for a
 270patch into the mainline, it is likely to end up in -mm.  Miscellaneous
 271patches which accumulate in -mm will eventually either be forwarded on to
 272an appropriate subsystem tree or be sent directly to Linus.  In a typical
 273development cycle, approximately 5-10% of the patches going into the
 274mainline get there via -mm.
 275
 276The current -mm patch is available in the "mmotm" (-mm of the moment)
 277directory at:
 278
 279        http://userweb.kernel.org/~akpm/mmotm/
 280
 281Use of the MMOTM tree is likely to be a frustrating experience, though;
 282there is a definite chance that it will not even compile.
 283
 284The primary tree for next-cycle patch merging is linux-next, maintained by
 285Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
 286the mainline is expected to look like after the next merge window closes.
 287Linux-next trees are announced on the linux-kernel and linux-next mailing
 288lists when they are assembled; they can be downloaded from:
 289
 290        http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/
 291
 292Some information about linux-next has been gathered at:
 293
 294        http://linux.f-seidel.de/linux-next/pmwiki/
 295
 296Linux-next has become an integral part of the kernel development process;
 297all patches merged during a given merge window should really have found
 298their way into linux-next some time before the merge window opens.
 299
 300
 3012.4.1: STAGING TREES
 302
 303The kernel source tree contains the drivers/staging/ directory, where
 304many sub-directories for drivers or filesystems that are on their way to
 305being added to the kernel tree live.  They remain in drivers/staging while
 306they still need more work; once complete, they can be moved into the
 307kernel proper.  This is a way to keep track of drivers that aren't
 308up to Linux kernel coding or quality standards, but people may want to use
 309them and track development.
 310
 311Greg Kroah-Hartman currently maintains the staging tree.  Drivers that
 312still need work are sent to him, with each driver having its own
 313subdirectory in drivers/staging/.  Along with the driver source files, a
 314TODO file should be present in the directory as well.  The TODO file lists
 315the pending work that the driver needs for acceptance into the kernel
 316proper, as well as a list of people that should be Cc'd for any patches to
 317the driver.  Current rules require that drivers contributed to staging
 318must, at a minimum, compile properly.
 319
 320Staging can be a relatively easy way to get new drivers into the mainline
 321where, with luck, they will come to the attention of other developers and
 322improve quickly.  Entry into staging is not the end of the story, though;
 323code in staging which is not seeing regular progress will eventually be
 324removed.  Distributors also tend to be relatively reluctant to enable
 325staging drivers.  So staging is, at best, a stop on the way toward becoming
 326a proper mainline driver.
 327
 328
 3292.5: TOOLS
 330
 331As can be seen from the above text, the kernel development process depends
 332heavily on the ability to herd collections of patches in various
 333directions.  The whole thing would not work anywhere near as well as it
 334does without suitably powerful tools.  Tutorials on how to use these tools
 335are well beyond the scope of this document, but there is space for a few
 336pointers.
 337
 338By far the dominant source code management system used by the kernel
 339community is git.  Git is one of a number of distributed version control
 340systems being developed in the free software community.  It is well tuned
 341for kernel development, in that it performs quite well when dealing with
 342large repositories and large numbers of patches.  It also has a reputation
 343for being difficult to learn and use, though it has gotten better over
 344time.  Some sort of familiarity with git is almost a requirement for kernel
 345developers; even if they do not use it for their own work, they'll need git
 346to keep up with what other developers (and the mainline) are doing.
 347
 348Git is now packaged by almost all Linux distributions.  There is a home
 349page at:
 350
 351        http://git-scm.com/
 352
 353That page has pointers to documentation and tutorials.
 354
 355Among the kernel developers who do not use git, the most popular choice is
 356almost certainly Mercurial:
 357
 358        http://www.selenic.com/mercurial/
 359
 360Mercurial shares many features with git, but it provides an interface which
 361many find easier to use.
 362
 363The other tool worth knowing about is Quilt:
 364
 365        http://savannah.nongnu.org/projects/quilt/
 366
 367Quilt is a patch management system, rather than a source code management
 368system.  It does not track history over time; it is, instead, oriented
 369toward tracking a specific set of changes against an evolving code base.
 370Some major subsystem maintainers use quilt to manage patches intended to go
 371upstream.  For the management of certain kinds of trees (-mm, for example),
 372quilt is the best tool for the job.
 373
 374
 3752.6: MAILING LISTS
 376
 377A great deal of Linux kernel development work is done by way of mailing
 378lists.  It is hard to be a fully-functioning member of the community
 379without joining at least one list somewhere.  But Linux mailing lists also
 380represent a potential hazard to developers, who risk getting buried under a
 381load of electronic mail, running afoul of the conventions used on the Linux
 382lists, or both.
 383
 384Most kernel mailing lists are run on vger.kernel.org; the master list can
 385be found at:
 386
 387        http://vger.kernel.org/vger-lists.html
 388
 389There are lists hosted elsewhere, though; a number of them are at
 390lists.redhat.com.
 391
 392The core mailing list for kernel development is, of course, linux-kernel.
 393This list is an intimidating place to be; volume can reach 500 messages per
 394day, the amount of noise is high, the conversation can be severely
 395technical, and participants are not always concerned with showing a high
 396degree of politeness.  But there is no other place where the kernel
 397development community comes together as a whole; developers who avoid this
 398list will miss important information.
 399
 400There are a few hints which can help with linux-kernel survival:
 401
 402- Have the list delivered to a separate folder, rather than your main
 403  mailbox.  One must be able to ignore the stream for sustained periods of
 404  time.
 405
 406- Do not try to follow every conversation - nobody else does.  It is
 407  important to filter on both the topic of interest (though note that
 408  long-running conversations can drift away from the original subject
 409  without changing the email subject line) and the people who are
 410  participating.
 411
 412- Do not feed the trolls.  If somebody is trying to stir up an angry
 413  response, ignore them.
 414
 415- When responding to linux-kernel email (or that on other lists) preserve
 416  the Cc: header for all involved.  In the absence of a strong reason (such
 417  as an explicit request), you should never remove recipients.  Always make
 418  sure that the person you are responding to is in the Cc: list.  This
 419  convention also makes it unnecessary to explicitly ask to be copied on
 420  replies to your postings.
 421
 422- Search the list archives (and the net as a whole) before asking
 423  questions.  Some developers can get impatient with people who clearly
 424  have not done their homework.
 425
 426- Avoid top-posting (the practice of putting your answer above the quoted
 427  text you are responding to).  It makes your response harder to read and
 428  makes a poor impression.
 429
 430- Ask on the correct mailing list.  Linux-kernel may be the general meeting
 431  point, but it is not the best place to find developers from all
 432  subsystems.
 433
 434The last point - finding the correct mailing list - is a common place for
 435beginning developers to go wrong.  Somebody who asks a networking-related
 436question on linux-kernel will almost certainly receive a polite suggestion
 437to ask on the netdev list instead, as that is the list frequented by most
 438networking developers.  Other lists exist for the SCSI, video4linux, IDE,
 439filesystem, etc. subsystems.  The best place to look for mailing lists is
 440in the MAINTAINERS file packaged with the kernel source.
 441
 442
 4432.7: GETTING STARTED WITH KERNEL DEVELOPMENT
 444
 445Questions about how to get started with the kernel development process are
 446common - from both individuals and companies.  Equally common are missteps
 447which make the beginning of the relationship harder than it has to be.
 448
 449Companies often look to hire well-known developers to get a development
 450group started.  This can, in fact, be an effective technique.  But it also
 451tends to be expensive and does not do much to grow the pool of experienced
 452kernel developers.  It is possible to bring in-house developers up to speed
 453on Linux kernel development, given the investment of a bit of time.  Taking
 454this time can endow an employer with a group of developers who understand
 455the kernel and the company both, and who can help to train others as well.
 456Over the medium term, this is often the more profitable approach.
 457
 458Individual developers are often, understandably, at a loss for a place to
 459start.  Beginning with a large project can be intimidating; one often wants
 460to test the waters with something smaller first.  This is the point where
 461some developers jump into the creation of patches fixing spelling errors or
 462minor coding style issues.  Unfortunately, such patches create a level of
 463noise which is distracting for the development community as a whole, so,
 464increasingly, they are looked down upon.  New developers wishing to
 465introduce themselves to the community will not get the sort of reception
 466they wish for by these means.
 467
 468Andrew Morton gives this advice for aspiring kernel developers
 469
 470        The #1 project for all kernel beginners should surely be "make sure
 471        that the kernel runs perfectly at all times on all machines which
 472        you can lay your hands on".  Usually the way to do this is to work
 473        with others on getting things fixed up (this can require
 474        persistence!) but that's fine - it's a part of kernel development.
 475
 476(http://lwn.net/Articles/283982/).
 477
 478In the absence of obvious problems to fix, developers are advised to look
 479at the current lists of regressions and open bugs in general.  There is
 480never any shortage of issues in need of fixing; by addressing these issues,
 481developers will gain experience with the process while, at the same time,
 482building respect with the rest of the development community.
 483
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.