1.. _development_process:
2
3How the development process works
4=================================
5
6Linux kernel development in the early 1990's was a pretty loose affair,
7with relatively small numbers of users and developers involved.  With a
8user base in the millions and with some 2,000 developers involved over the
9course of one year, the kernel has since had to evolve a number of
10processes to keep development happening smoothly.  A solid understanding of
11how the process works is required in order to be an effective part of it.
12
13The big picture
14---------------
15
16The kernel developers use a loosely time-based release process, with a new
17major kernel release happening every two or three months.  The recent
18release history looks like this:
19
20	======  =================
21	5.0	March 3, 2019
22	5.1	May 5, 2019
23	5.2	July 7, 2019
24	5.3	September 15, 2019
25	5.4	November 24, 2019
26	5.5	January 6, 2020
27	======  =================
28
29Every 5.x release is a major kernel release with new features, internal
30API changes, and more.  A typical release can contain about 13,000
31changesets with changes to several hundred thousand lines of code.  5.x is
32the leading edge of Linux kernel development; the kernel uses a
33rolling development model which is continually integrating major changes.
34
35A relatively straightforward discipline is followed with regard to the
36merging of patches for each release.  At the beginning of each development
37cycle, the "merge window" is said to be open.  At that time, code which is
38deemed to be sufficiently stable (and which is accepted by the development
39community) is merged into the mainline kernel.  The bulk of changes for a
40new development cycle (and all of the major changes) will be merged during
41this time, at a rate approaching 1,000 changes ("patches," or "changesets")
42per day.
43
44(As an aside, it is worth noting that the changes integrated during the
45merge window do not come out of thin air; they have been collected, tested,
46and staged ahead of time.  How that process works will be described in
47detail later on).
48
49The merge window lasts for approximately two weeks.  At the end of this
50time, Linus Torvalds will declare that the window is closed and release the
51first of the "rc" kernels.  For the kernel which is destined to be 5.6,
52for example, the release which happens at the end of the merge window will
53be called 5.6-rc1.  The -rc1 release is the signal that the time to
54merge new features has passed, and that the time to stabilize the next
55kernel has begun.
56
57Over the next six to ten weeks, only patches which fix problems should be
58submitted to the mainline.  On occasion a more significant change will be
59allowed, but such occasions are rare; developers who try to merge new
60features outside of the merge window tend to get an unfriendly reception.
61As a general rule, if you miss the merge window for a given feature, the
62best thing to do is to wait for the next development cycle.  (An occasional
63exception is made for drivers for previously-unsupported hardware; if they
64touch no in-tree code, they cannot cause regressions and should be safe to
65add at any time).
66
67As fixes make their way into the mainline, the patch rate will slow over
68time.  Linus releases new -rc kernels about once a week; a normal series
69will get up to somewhere between -rc6 and -rc9 before the kernel is
70considered to be sufficiently stable and the final release is made.
71At that point the whole process starts over again.
72
73As an example, here is how the 5.4 development cycle went (all dates in
742019):
75
76	==============  ===============================
77	September 15	5.3 stable release
78	September 30	5.4-rc1, merge window closes
79	October 6	5.4-rc2
80	October 13	5.4-rc3
81	October 20	5.4-rc4
82	October 27	5.4-rc5
83	November 3	5.4-rc6
84	November 10	5.4-rc7
85	November 17	5.4-rc8
86	November 24	5.4 stable release
87	==============  ===============================
88
89How do the developers decide when to close the development cycle and create
90the stable release?  The most significant metric used is the list of
91regressions from previous releases.  No bugs are welcome, but those which
92break systems which worked in the past are considered to be especially
93serious.  For this reason, patches which cause regressions are looked upon
94unfavorably and are quite likely to be reverted during the stabilization
95period.
96
97The developers' goal is to fix all known regressions before the stable
98release is made.  In the real world, this kind of perfection is hard to
99achieve; there are just too many variables in a project of this size.
100There comes a point where delaying the final release just makes the problem
101worse; the pile of changes waiting for the next merge window will grow
102larger, creating even more regressions the next time around.  So most 5.x
103kernels go out with a handful of known regressions though, hopefully, none
104of them are serious.
105
106Once a stable release is made, its ongoing maintenance is passed off to the
107"stable team," currently Greg Kroah-Hartman. The stable team will release
108occasional updates to the stable release using the 5.x.y numbering scheme.
109To be considered for an update release, a patch must (1) fix a significant
110bug, and (2) already be merged into the mainline for the next development
111kernel. Kernels will typically receive stable updates for a little more
112than one development cycle past their initial release. So, for example, the
1135.2 kernel's history looked like this (all dates in 2019):
114
115	==============  ===============================
116	September 15 	5.2 stable release
117	July 14		5.2.1
118	July 21		5.2.2
119	July 26		5.2.3
120	July 28		5.2.4
121	July 31  	5.2.5
122	...		...
123	October 11	5.2.21
124	==============  ===============================
125
1265.2.21 was the final stable update of the 5.2 release.
127
128Some kernels are designated "long term" kernels; they will receive support
129for a longer period.  As of this writing, the current long term kernels
130and their maintainers are:
131
132	======  ================================	=======================
133	3.16	Ben Hutchings				(very long-term kernel)
134	4.4	Greg Kroah-Hartman & Sasha Levin	(very long-term kernel)
135	4.9	Greg Kroah-Hartman & Sasha Levin
136	4.14	Greg Kroah-Hartman & Sasha Levin
137	4.19	Greg Kroah-Hartman & Sasha Levin
138	5.4	Greg Kroah-Hartman & Sasha Levin
139	======  ================================	=======================
140
141The selection of a kernel for long-term support is purely a matter of a
142maintainer having the need and the time to maintain that release.  There
143are no known plans for long-term support for any specific upcoming
144release.
145
146
147The lifecycle of a patch
148------------------------
149
150Patches do not go directly from the developer's keyboard into the mainline
151kernel.  There is, instead, a somewhat involved (if somewhat informal)
152process designed to ensure that each patch is reviewed for quality and that
153each patch implements a change which is desirable to have in the mainline.
154This process can happen quickly for minor fixes, or, in the case of large
155and controversial changes, go on for years.  Much developer frustration
156comes from a lack of understanding of this process or from attempts to
157circumvent it.
158
159In the hopes of reducing that frustration, this document will describe how
160a patch gets into the kernel.  What follows below is an introduction which
161describes the process in a somewhat idealized way.  A much more detailed
162treatment will come in later sections.
163
164The stages that a patch goes through are, generally:
165
166 - Design.  This is where the real requirements for the patch - and the way
167   those requirements will be met - are laid out.  Design work is often
168   done without involving the community, but it is better to do this work
169   in the open if at all possible; it can save a lot of time redesigning
170   things later.
171
172 - Early review.  Patches are posted to the relevant mailing list, and
173   developers on that list reply with any comments they may have.  This
174   process should turn up any major problems with a patch if all goes
175   well.
176
177 - Wider review.  When the patch is getting close to ready for mainline
178   inclusion, it should be accepted by a relevant subsystem maintainer -
179   though this acceptance is not a guarantee that the patch will make it
180   all the way to the mainline.  The patch will show up in the maintainer's
181   subsystem tree and into the -next trees (described below).  When the
182   process works, this step leads to more extensive review of the patch and
183   the discovery of any problems resulting from the integration of this
184   patch with work being done by others.
185
186-  Please note that most maintainers also have day jobs, so merging
187   your patch may not be their highest priority.  If your patch is
188   getting feedback about changes that are needed, you should either
189   make those changes or justify why they should not be made.  If your
190   patch has no review complaints but is not being merged by its
191   appropriate subsystem or driver maintainer, you should be persistent
192   in updating the patch to the current kernel so that it applies cleanly
193   and keep sending it for review and merging.
194
195 - Merging into the mainline.  Eventually, a successful patch will be
196   merged into the mainline repository managed by Linus Torvalds.  More
197   comments and/or problems may surface at this time; it is important that
198   the developer be responsive to these and fix any issues which arise.
199
200 - Stable release.  The number of users potentially affected by the patch
201   is now large, so, once again, new problems may arise.
202
203 - Long-term maintenance.  While it is certainly possible for a developer
204   to forget about code after merging it, that sort of behavior tends to
205   leave a poor impression in the development community.  Merging code
206   eliminates some of the maintenance burden, in that others will fix
207   problems caused by API changes.  But the original developer should
208   continue to take responsibility for the code if it is to remain useful
209   in the longer term.
210
211One of the largest mistakes made by kernel developers (or their employers)
212is to try to cut the process down to a single "merging into the mainline"
213step.  This approach invariably leads to frustration for everybody
214involved.
215
216How patches get into the Kernel
217-------------------------------
218
219There is exactly one person who can merge patches into the mainline kernel
220repository: Linus Torvalds. But, for example, of the over 9,500 patches
221which went into the 2.6.38 kernel, only 112 (around 1.3%) were directly
222chosen by Linus himself. The kernel project has long since grown to a size
223where no single developer could possibly inspect and select every patch
224unassisted. The way the kernel developers have addressed this growth is
225through the use of a lieutenant system built around a chain of trust.
226
227The kernel code base is logically broken down into a set of subsystems:
228networking, specific architecture support, memory management, video
229devices, etc.  Most subsystems have a designated maintainer, a developer
230who has overall responsibility for the code within that subsystem.  These
231subsystem maintainers are the gatekeepers (in a loose way) for the portion
232of the kernel they manage; they are the ones who will (usually) accept a
233patch for inclusion into the mainline kernel.
234
235Subsystem maintainers each manage their own version of the kernel source
236tree, usually (but certainly not always) using the git source management
237tool.  Tools like git (and related tools like quilt or mercurial) allow
238maintainers to track a list of patches, including authorship information
239and other metadata.  At any given time, the maintainer can identify which
240patches in his or her repository are not found in the mainline.
241
242When the merge window opens, top-level maintainers will ask Linus to "pull"
243the patches they have selected for merging from their repositories.  If
244Linus agrees, the stream of patches will flow up into his repository,
245becoming part of the mainline kernel.  The amount of attention that Linus
246pays to specific patches received in a pull operation varies.  It is clear
247that, sometimes, he looks quite closely.  But, as a general rule, Linus
248trusts the subsystem maintainers to not send bad patches upstream.
249
250Subsystem maintainers, in turn, can pull patches from other maintainers.
251For example, the networking tree is built from patches which accumulated
252first in trees dedicated to network device drivers, wireless networking,
253etc.  This chain of repositories can be arbitrarily long, though it rarely
254exceeds two or three links.  Since each maintainer in the chain trusts
255those managing lower-level trees, this process is known as the "chain of
256trust."
257
258Clearly, in a system like this, getting patches into the kernel depends on
259finding the right maintainer.  Sending patches directly to Linus is not
260normally the right way to go.
261
262
263Next trees
264----------
265
266The chain of subsystem trees guides the flow of patches into the kernel,
267but it also raises an interesting question: what if somebody wants to look
268at all of the patches which are being prepared for the next merge window?
269Developers will be interested in what other changes are pending to see
270whether there are any conflicts to worry about; a patch which changes a
271core kernel function prototype, for example, will conflict with any other
272patches which use the older form of that function.  Reviewers and testers
273want access to the changes in their integrated form before all of those
274changes land in the mainline kernel.  One could pull changes from all of
275the interesting subsystem trees, but that would be a big and error-prone
276job.
277
278The answer comes in the form of -next trees, where subsystem trees are
279collected for testing and review.  The older of these trees, maintained by
280Andrew Morton, is called "-mm" (for memory management, which is how it got
281started).  The -mm tree integrates patches from a long list of subsystem
282trees; it also has some patches aimed at helping with debugging.
283
284Beyond that, -mm contains a significant collection of patches which have
285been selected by Andrew directly.  These patches may have been posted on a
286mailing list, or they may apply to a part of the kernel for which there is
287no designated subsystem tree.  As a result, -mm operates as a sort of
288subsystem tree of last resort; if there is no other obvious path for a
289patch into the mainline, it is likely to end up in -mm.  Miscellaneous
290patches which accumulate in -mm will eventually either be forwarded on to
291an appropriate subsystem tree or be sent directly to Linus.  In a typical
292development cycle, approximately 5-10% of the patches going into the
293mainline get there via -mm.
294
295The current -mm patch is available in the "mmotm" (-mm of the moment)
296directory at:
297
298	http://www.ozlabs.org/~akpm/mmotm/
299
300Use of the MMOTM tree is likely to be a frustrating experience, though;
301there is a definite chance that it will not even compile.
302
303The primary tree for next-cycle patch merging is linux-next, maintained by
304Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
305the mainline is expected to look like after the next merge window closes.
306Linux-next trees are announced on the linux-kernel and linux-next mailing
307lists when they are assembled; they can be downloaded from:
308
309	http://www.kernel.org/pub/linux/kernel/next/
310
311Linux-next has become an integral part of the kernel development process;
312all patches merged during a given merge window should really have found
313their way into linux-next some time before the merge window opens.
314
315
316Staging trees
317-------------
318
319The kernel source tree contains the drivers/staging/ directory, where
320many sub-directories for drivers or filesystems that are on their way to
321being added to the kernel tree live.  They remain in drivers/staging while
322they still need more work; once complete, they can be moved into the
323kernel proper.  This is a way to keep track of drivers that aren't
324up to Linux kernel coding or quality standards, but people may want to use
325them and track development.
326
327Greg Kroah-Hartman currently maintains the staging tree.  Drivers that
328still need work are sent to him, with each driver having its own
329subdirectory in drivers/staging/.  Along with the driver source files, a
330TODO file should be present in the directory as well.  The TODO file lists
331the pending work that the driver needs for acceptance into the kernel
332proper, as well as a list of people that should be Cc'd for any patches to
333the driver.  Current rules require that drivers contributed to staging
334must, at a minimum, compile properly.
335
336Staging can be a relatively easy way to get new drivers into the mainline
337where, with luck, they will come to the attention of other developers and
338improve quickly.  Entry into staging is not the end of the story, though;
339code in staging which is not seeing regular progress will eventually be
340removed.  Distributors also tend to be relatively reluctant to enable
341staging drivers.  So staging is, at best, a stop on the way toward becoming
342a proper mainline driver.
343
344
345Tools
346-----
347
348As can be seen from the above text, the kernel development process depends
349heavily on the ability to herd collections of patches in various
350directions.  The whole thing would not work anywhere near as well as it
351does without suitably powerful tools.  Tutorials on how to use these tools
352are well beyond the scope of this document, but there is space for a few
353pointers.
354
355By far the dominant source code management system used by the kernel
356community is git.  Git is one of a number of distributed version control
357systems being developed in the free software community.  It is well tuned
358for kernel development, in that it performs quite well when dealing with
359large repositories and large numbers of patches.  It also has a reputation
360for being difficult to learn and use, though it has gotten better over
361time.  Some sort of familiarity with git is almost a requirement for kernel
362developers; even if they do not use it for their own work, they'll need git
363to keep up with what other developers (and the mainline) are doing.
364
365Git is now packaged by almost all Linux distributions.  There is a home
366page at:
367
368	http://git-scm.com/
369
370That page has pointers to documentation and tutorials.
371
372Among the kernel developers who do not use git, the most popular choice is
373almost certainly Mercurial:
374
375	http://www.selenic.com/mercurial/
376
377Mercurial shares many features with git, but it provides an interface which
378many find easier to use.
379
380The other tool worth knowing about is Quilt:
381
382	http://savannah.nongnu.org/projects/quilt/
383
384Quilt is a patch management system, rather than a source code management
385system.  It does not track history over time; it is, instead, oriented
386toward tracking a specific set of changes against an evolving code base.
387Some major subsystem maintainers use quilt to manage patches intended to go
388upstream.  For the management of certain kinds of trees (-mm, for example),
389quilt is the best tool for the job.
390
391
392Mailing lists
393-------------
394
395A great deal of Linux kernel development work is done by way of mailing
396lists.  It is hard to be a fully-functioning member of the community
397without joining at least one list somewhere.  But Linux mailing lists also
398represent a potential hazard to developers, who risk getting buried under a
399load of electronic mail, running afoul of the conventions used on the Linux
400lists, or both.
401
402Most kernel mailing lists are run on vger.kernel.org; the master list can
403be found at:
404
405	http://vger.kernel.org/vger-lists.html
406
407There are lists hosted elsewhere, though; a number of them are at
408lists.redhat.com.
409
410The core mailing list for kernel development is, of course, linux-kernel.
411This list is an intimidating place to be; volume can reach 500 messages per
412day, the amount of noise is high, the conversation can be severely
413technical, and participants are not always concerned with showing a high
414degree of politeness.  But there is no other place where the kernel
415development community comes together as a whole; developers who avoid this
416list will miss important information.
417
418There are a few hints which can help with linux-kernel survival:
419
420- Have the list delivered to a separate folder, rather than your main
421  mailbox.  One must be able to ignore the stream for sustained periods of
422  time.
423
424- Do not try to follow every conversation - nobody else does.  It is
425  important to filter on both the topic of interest (though note that
426  long-running conversations can drift away from the original subject
427  without changing the email subject line) and the people who are
428  participating.
429
430- Do not feed the trolls.  If somebody is trying to stir up an angry
431  response, ignore them.
432
433- When responding to linux-kernel email (or that on other lists) preserve
434  the Cc: header for all involved.  In the absence of a strong reason (such
435  as an explicit request), you should never remove recipients.  Always make
436  sure that the person you are responding to is in the Cc: list.  This
437  convention also makes it unnecessary to explicitly ask to be copied on
438  replies to your postings.
439
440- Search the list archives (and the net as a whole) before asking
441  questions.  Some developers can get impatient with people who clearly
442  have not done their homework.
443
444- Avoid top-posting (the practice of putting your answer above the quoted
445  text you are responding to).  It makes your response harder to read and
446  makes a poor impression.
447
448- Ask on the correct mailing list.  Linux-kernel may be the general meeting
449  point, but it is not the best place to find developers from all
450  subsystems.
451
452The last point - finding the correct mailing list - is a common place for
453beginning developers to go wrong.  Somebody who asks a networking-related
454question on linux-kernel will almost certainly receive a polite suggestion
455to ask on the netdev list instead, as that is the list frequented by most
456networking developers.  Other lists exist for the SCSI, video4linux, IDE,
457filesystem, etc. subsystems.  The best place to look for mailing lists is
458in the MAINTAINERS file packaged with the kernel source.
459
460
461Getting started with Kernel development
462---------------------------------------
463
464Questions about how to get started with the kernel development process are
465common - from both individuals and companies.  Equally common are missteps
466which make the beginning of the relationship harder than it has to be.
467
468Companies often look to hire well-known developers to get a development
469group started.  This can, in fact, be an effective technique.  But it also
470tends to be expensive and does not do much to grow the pool of experienced
471kernel developers.  It is possible to bring in-house developers up to speed
472on Linux kernel development, given the investment of a bit of time.  Taking
473this time can endow an employer with a group of developers who understand
474the kernel and the company both, and who can help to train others as well.
475Over the medium term, this is often the more profitable approach.
476
477Individual developers are often, understandably, at a loss for a place to
478start.  Beginning with a large project can be intimidating; one often wants
479to test the waters with something smaller first.  This is the point where
480some developers jump into the creation of patches fixing spelling errors or
481minor coding style issues.  Unfortunately, such patches create a level of
482noise which is distracting for the development community as a whole, so,
483increasingly, they are looked down upon.  New developers wishing to
484introduce themselves to the community will not get the sort of reception
485they wish for by these means.
486
487Andrew Morton gives this advice for aspiring kernel developers
488
489::
490
491	The #1 project for all kernel beginners should surely be "make sure
492	that the kernel runs perfectly at all times on all machines which
493	you can lay your hands on".  Usually the way to do this is to work
494	with others on getting things fixed up (this can require
495	persistence!) but that's fine - it's a part of kernel development.
496
497(http://lwn.net/Articles/283982/).
498
499In the absence of obvious problems to fix, developers are advised to look
500at the current lists of regressions and open bugs in general.  There is
501never any shortage of issues in need of fixing; by addressing these issues,
502developers will gain experience with the process while, at the same time,
503building respect with the rest of the development community.
504