1.. _development_process:
2
3How the development process works
4=================================
5
6Linux kernel development in the early 1990's was a pretty loose affair,
7with relatively small numbers of users and developers involved.  With a
8user base in the millions and with some 2,000 developers involved over the
9course of one year, the kernel has since had to evolve a number of
10processes to keep development happening smoothly.  A solid understanding of
11how the process works is required in order to be an effective part of it.
12
13The big picture
14---------------
15
16The kernel developers use a loosely time-based release process, with a new
17major kernel release happening every two or three months.  The recent
18release history looks like this:
19
20	======  =================
21	4.11	April 30, 2017
22	4.12	July 2, 2017
23	4.13	September 3, 2017
24	4.14	November 12, 2017
25	4.15	January 28, 2018
26	4.16	April 1, 2018
27	======  =================
28
29Every 4.x release is a major kernel release with new features, internal
30API changes, and more.  A typical 4.x release contain about 13,000
31changesets with changes to several hundred thousand lines of code.  4.x is
32thus the leading edge of Linux kernel development; the kernel uses a
33rolling development model which is continually integrating major changes.
34
35A relatively straightforward discipline is followed with regard to the
36merging of patches for each release.  At the beginning of each development
37cycle, the "merge window" is said to be open.  At that time, code which is
38deemed to be sufficiently stable (and which is accepted by the development
39community) is merged into the mainline kernel.  The bulk of changes for a
40new development cycle (and all of the major changes) will be merged during
41this time, at a rate approaching 1,000 changes ("patches," or "changesets")
42per day.
43
44(As an aside, it is worth noting that the changes integrated during the
45merge window do not come out of thin air; they have been collected, tested,
46and staged ahead of time.  How that process works will be described in
47detail later on).
48
49The merge window lasts for approximately two weeks.  At the end of this
50time, Linus Torvalds will declare that the window is closed and release the
51first of the "rc" kernels.  For the kernel which is destined to be 2.6.40,
52for example, the release which happens at the end of the merge window will
53be called 2.6.40-rc1.  The -rc1 release is the signal that the time to
54merge new features has passed, and that the time to stabilize the next
55kernel has begun.
56
57Over the next six to ten weeks, only patches which fix problems should be
58submitted to the mainline.  On occasion a more significant change will be
59allowed, but such occasions are rare; developers who try to merge new
60features outside of the merge window tend to get an unfriendly reception.
61As a general rule, if you miss the merge window for a given feature, the
62best thing to do is to wait for the next development cycle.  (An occasional
63exception is made for drivers for previously-unsupported hardware; if they
64touch no in-tree code, they cannot cause regressions and should be safe to
65add at any time).
66
67As fixes make their way into the mainline, the patch rate will slow over
68time.  Linus releases new -rc kernels about once a week; a normal series
69will get up to somewhere between -rc6 and -rc9 before the kernel is
70considered to be sufficiently stable and the final 2.6.x release is made.
71At that point the whole process starts over again.
72
73As an example, here is how the 4.16 development cycle went (all dates in
742018):
75
76	==============  ===============================
77	January 28	4.15 stable release
78	February 11	4.16-rc1, merge window closes
79	February 18	4.16-rc2
80	February 25	4.16-rc3
81	March 4		4.16-rc4
82	March 11	4.16-rc5
83	March 18	4.16-rc6
84	March 25	4.16-rc7
85	April 1		4.16 stable release
86	==============  ===============================
87
88How do the developers decide when to close the development cycle and create
89the stable release?  The most significant metric used is the list of
90regressions from previous releases.  No bugs are welcome, but those which
91break systems which worked in the past are considered to be especially
92serious.  For this reason, patches which cause regressions are looked upon
93unfavorably and are quite likely to be reverted during the stabilization
94period.
95
96The developers' goal is to fix all known regressions before the stable
97release is made.  In the real world, this kind of perfection is hard to
98achieve; there are just too many variables in a project of this size.
99There comes a point where delaying the final release just makes the problem
100worse; the pile of changes waiting for the next merge window will grow
101larger, creating even more regressions the next time around.  So most 4.x
102kernels go out with a handful of known regressions though, hopefully, none
103of them are serious.
104
105Once a stable release is made, its ongoing maintenance is passed off to the
106"stable team," currently consisting of Greg Kroah-Hartman.  The stable team
107will release occasional updates to the stable release using the 4.x.y
108numbering scheme.  To be considered for an update release, a patch must (1)
109fix a significant bug, and (2) already be merged into the mainline for the
110next development kernel.  Kernels will typically receive stable updates for
111a little more than one development cycle past their initial release.  So,
112for example, the 4.13 kernel's history looked like:
113
114	==============  ===============================
115	September 3 	4.13 stable release
116	September 13	4.13.1
117	September 20	4.13.2
118	September 27	4.13.3
119	October 5	4.13.4
120	October 12  	4.13.5
121	...		...
122	November 24	4.13.16
123	==============  ===============================
124
1254.13.16 was the final stable update of the 4.13 release.
126
127Some kernels are designated "long term" kernels; they will receive support
128for a longer period.  As of this writing, the current long term kernels
129and their maintainers are:
130
131	======  ======================  ==============================
132	3.16	Ben Hutchings		(very long-term stable kernel)
133	4.1	Sasha Levin
134	4.4	Greg Kroah-Hartman	(very long-term stable kernel)
135	4.9	Greg Kroah-Hartman
136	4.14	Greg Kroah-Hartman
137	======  ======================  ==============================
138
139The selection of a kernel for long-term support is purely a matter of a
140maintainer having the need and the time to maintain that release.  There
141are no known plans for long-term support for any specific upcoming
142release.
143
144
145The lifecycle of a patch
146------------------------
147
148Patches do not go directly from the developer's keyboard into the mainline
149kernel.  There is, instead, a somewhat involved (if somewhat informal)
150process designed to ensure that each patch is reviewed for quality and that
151each patch implements a change which is desirable to have in the mainline.
152This process can happen quickly for minor fixes, or, in the case of large
153and controversial changes, go on for years.  Much developer frustration
154comes from a lack of understanding of this process or from attempts to
155circumvent it.
156
157In the hopes of reducing that frustration, this document will describe how
158a patch gets into the kernel.  What follows below is an introduction which
159describes the process in a somewhat idealized way.  A much more detailed
160treatment will come in later sections.
161
162The stages that a patch goes through are, generally:
163
164 - Design.  This is where the real requirements for the patch - and the way
165   those requirements will be met - are laid out.  Design work is often
166   done without involving the community, but it is better to do this work
167   in the open if at all possible; it can save a lot of time redesigning
168   things later.
169
170 - Early review.  Patches are posted to the relevant mailing list, and
171   developers on that list reply with any comments they may have.  This
172   process should turn up any major problems with a patch if all goes
173   well.
174
175 - Wider review.  When the patch is getting close to ready for mainline
176   inclusion, it should be accepted by a relevant subsystem maintainer -
177   though this acceptance is not a guarantee that the patch will make it
178   all the way to the mainline.  The patch will show up in the maintainer's
179   subsystem tree and into the -next trees (described below).  When the
180   process works, this step leads to more extensive review of the patch and
181   the discovery of any problems resulting from the integration of this
182   patch with work being done by others.
183
184-  Please note that most maintainers also have day jobs, so merging
185   your patch may not be their highest priority.  If your patch is
186   getting feedback about changes that are needed, you should either
187   make those changes or justify why they should not be made.  If your
188   patch has no review complaints but is not being merged by its
189   appropriate subsystem or driver maintainer, you should be persistent
190   in updating the patch to the current kernel so that it applies cleanly
191   and keep sending it for review and merging.
192
193 - Merging into the mainline.  Eventually, a successful patch will be
194   merged into the mainline repository managed by Linus Torvalds.  More
195   comments and/or problems may surface at this time; it is important that
196   the developer be responsive to these and fix any issues which arise.
197
198 - Stable release.  The number of users potentially affected by the patch
199   is now large, so, once again, new problems may arise.
200
201 - Long-term maintenance.  While it is certainly possible for a developer
202   to forget about code after merging it, that sort of behavior tends to
203   leave a poor impression in the development community.  Merging code
204   eliminates some of the maintenance burden, in that others will fix
205   problems caused by API changes.  But the original developer should
206   continue to take responsibility for the code if it is to remain useful
207   in the longer term.
208
209One of the largest mistakes made by kernel developers (or their employers)
210is to try to cut the process down to a single "merging into the mainline"
211step.  This approach invariably leads to frustration for everybody
212involved.
213
214How patches get into the Kernel
215-------------------------------
216
217There is exactly one person who can merge patches into the mainline kernel
218repository: Linus Torvalds.  But, of the over 9,500 patches which went
219into the 2.6.38 kernel, only 112 (around 1.3%) were directly chosen by Linus
220himself.  The kernel project has long since grown to a size where no single
221developer could possibly inspect and select every patch unassisted.  The
222way the kernel developers have addressed this growth is through the use of
223a lieutenant system built around a chain of trust.
224
225The kernel code base is logically broken down into a set of subsystems:
226networking, specific architecture support, memory management, video
227devices, etc.  Most subsystems have a designated maintainer, a developer
228who has overall responsibility for the code within that subsystem.  These
229subsystem maintainers are the gatekeepers (in a loose way) for the portion
230of the kernel they manage; they are the ones who will (usually) accept a
231patch for inclusion into the mainline kernel.
232
233Subsystem maintainers each manage their own version of the kernel source
234tree, usually (but certainly not always) using the git source management
235tool.  Tools like git (and related tools like quilt or mercurial) allow
236maintainers to track a list of patches, including authorship information
237and other metadata.  At any given time, the maintainer can identify which
238patches in his or her repository are not found in the mainline.
239
240When the merge window opens, top-level maintainers will ask Linus to "pull"
241the patches they have selected for merging from their repositories.  If
242Linus agrees, the stream of patches will flow up into his repository,
243becoming part of the mainline kernel.  The amount of attention that Linus
244pays to specific patches received in a pull operation varies.  It is clear
245that, sometimes, he looks quite closely.  But, as a general rule, Linus
246trusts the subsystem maintainers to not send bad patches upstream.
247
248Subsystem maintainers, in turn, can pull patches from other maintainers.
249For example, the networking tree is built from patches which accumulated
250first in trees dedicated to network device drivers, wireless networking,
251etc.  This chain of repositories can be arbitrarily long, though it rarely
252exceeds two or three links.  Since each maintainer in the chain trusts
253those managing lower-level trees, this process is known as the "chain of
254trust."
255
256Clearly, in a system like this, getting patches into the kernel depends on
257finding the right maintainer.  Sending patches directly to Linus is not
258normally the right way to go.
259
260
261Next trees
262----------
263
264The chain of subsystem trees guides the flow of patches into the kernel,
265but it also raises an interesting question: what if somebody wants to look
266at all of the patches which are being prepared for the next merge window?
267Developers will be interested in what other changes are pending to see
268whether there are any conflicts to worry about; a patch which changes a
269core kernel function prototype, for example, will conflict with any other
270patches which use the older form of that function.  Reviewers and testers
271want access to the changes in their integrated form before all of those
272changes land in the mainline kernel.  One could pull changes from all of
273the interesting subsystem trees, but that would be a big and error-prone
274job.
275
276The answer comes in the form of -next trees, where subsystem trees are
277collected for testing and review.  The older of these trees, maintained by
278Andrew Morton, is called "-mm" (for memory management, which is how it got
279started).  The -mm tree integrates patches from a long list of subsystem
280trees; it also has some patches aimed at helping with debugging.
281
282Beyond that, -mm contains a significant collection of patches which have
283been selected by Andrew directly.  These patches may have been posted on a
284mailing list, or they may apply to a part of the kernel for which there is
285no designated subsystem tree.  As a result, -mm operates as a sort of
286subsystem tree of last resort; if there is no other obvious path for a
287patch into the mainline, it is likely to end up in -mm.  Miscellaneous
288patches which accumulate in -mm will eventually either be forwarded on to
289an appropriate subsystem tree or be sent directly to Linus.  In a typical
290development cycle, approximately 5-10% of the patches going into the
291mainline get there via -mm.
292
293The current -mm patch is available in the "mmotm" (-mm of the moment)
294directory at:
295
296	http://www.ozlabs.org/~akpm/mmotm/
297
298Use of the MMOTM tree is likely to be a frustrating experience, though;
299there is a definite chance that it will not even compile.
300
301The primary tree for next-cycle patch merging is linux-next, maintained by
302Stephen Rothwell.  The linux-next tree is, by design, a snapshot of what
303the mainline is expected to look like after the next merge window closes.
304Linux-next trees are announced on the linux-kernel and linux-next mailing
305lists when they are assembled; they can be downloaded from:
306
307	http://www.kernel.org/pub/linux/kernel/next/
308
309Linux-next has become an integral part of the kernel development process;
310all patches merged during a given merge window should really have found
311their way into linux-next some time before the merge window opens.
312
313
314Staging trees
315-------------
316
317The kernel source tree contains the drivers/staging/ directory, where
318many sub-directories for drivers or filesystems that are on their way to
319being added to the kernel tree live.  They remain in drivers/staging while
320they still need more work; once complete, they can be moved into the
321kernel proper.  This is a way to keep track of drivers that aren't
322up to Linux kernel coding or quality standards, but people may want to use
323them and track development.
324
325Greg Kroah-Hartman currently maintains the staging tree.  Drivers that
326still need work are sent to him, with each driver having its own
327subdirectory in drivers/staging/.  Along with the driver source files, a
328TODO file should be present in the directory as well.  The TODO file lists
329the pending work that the driver needs for acceptance into the kernel
330proper, as well as a list of people that should be Cc'd for any patches to
331the driver.  Current rules require that drivers contributed to staging
332must, at a minimum, compile properly.
333
334Staging can be a relatively easy way to get new drivers into the mainline
335where, with luck, they will come to the attention of other developers and
336improve quickly.  Entry into staging is not the end of the story, though;
337code in staging which is not seeing regular progress will eventually be
338removed.  Distributors also tend to be relatively reluctant to enable
339staging drivers.  So staging is, at best, a stop on the way toward becoming
340a proper mainline driver.
341
342
343Tools
344-----
345
346As can be seen from the above text, the kernel development process depends
347heavily on the ability to herd collections of patches in various
348directions.  The whole thing would not work anywhere near as well as it
349does without suitably powerful tools.  Tutorials on how to use these tools
350are well beyond the scope of this document, but there is space for a few
351pointers.
352
353By far the dominant source code management system used by the kernel
354community is git.  Git is one of a number of distributed version control
355systems being developed in the free software community.  It is well tuned
356for kernel development, in that it performs quite well when dealing with
357large repositories and large numbers of patches.  It also has a reputation
358for being difficult to learn and use, though it has gotten better over
359time.  Some sort of familiarity with git is almost a requirement for kernel
360developers; even if they do not use it for their own work, they'll need git
361to keep up with what other developers (and the mainline) are doing.
362
363Git is now packaged by almost all Linux distributions.  There is a home
364page at:
365
366	http://git-scm.com/
367
368That page has pointers to documentation and tutorials.
369
370Among the kernel developers who do not use git, the most popular choice is
371almost certainly Mercurial:
372
373	http://www.selenic.com/mercurial/
374
375Mercurial shares many features with git, but it provides an interface which
376many find easier to use.
377
378The other tool worth knowing about is Quilt:
379
380	http://savannah.nongnu.org/projects/quilt/
381
382Quilt is a patch management system, rather than a source code management
383system.  It does not track history over time; it is, instead, oriented
384toward tracking a specific set of changes against an evolving code base.
385Some major subsystem maintainers use quilt to manage patches intended to go
386upstream.  For the management of certain kinds of trees (-mm, for example),
387quilt is the best tool for the job.
388
389
390Mailing lists
391-------------
392
393A great deal of Linux kernel development work is done by way of mailing
394lists.  It is hard to be a fully-functioning member of the community
395without joining at least one list somewhere.  But Linux mailing lists also
396represent a potential hazard to developers, who risk getting buried under a
397load of electronic mail, running afoul of the conventions used on the Linux
398lists, or both.
399
400Most kernel mailing lists are run on vger.kernel.org; the master list can
401be found at:
402
403	http://vger.kernel.org/vger-lists.html
404
405There are lists hosted elsewhere, though; a number of them are at
406lists.redhat.com.
407
408The core mailing list for kernel development is, of course, linux-kernel.
409This list is an intimidating place to be; volume can reach 500 messages per
410day, the amount of noise is high, the conversation can be severely
411technical, and participants are not always concerned with showing a high
412degree of politeness.  But there is no other place where the kernel
413development community comes together as a whole; developers who avoid this
414list will miss important information.
415
416There are a few hints which can help with linux-kernel survival:
417
418- Have the list delivered to a separate folder, rather than your main
419  mailbox.  One must be able to ignore the stream for sustained periods of
420  time.
421
422- Do not try to follow every conversation - nobody else does.  It is
423  important to filter on both the topic of interest (though note that
424  long-running conversations can drift away from the original subject
425  without changing the email subject line) and the people who are
426  participating.
427
428- Do not feed the trolls.  If somebody is trying to stir up an angry
429  response, ignore them.
430
431- When responding to linux-kernel email (or that on other lists) preserve
432  the Cc: header for all involved.  In the absence of a strong reason (such
433  as an explicit request), you should never remove recipients.  Always make
434  sure that the person you are responding to is in the Cc: list.  This
435  convention also makes it unnecessary to explicitly ask to be copied on
436  replies to your postings.
437
438- Search the list archives (and the net as a whole) before asking
439  questions.  Some developers can get impatient with people who clearly
440  have not done their homework.
441
442- Avoid top-posting (the practice of putting your answer above the quoted
443  text you are responding to).  It makes your response harder to read and
444  makes a poor impression.
445
446- Ask on the correct mailing list.  Linux-kernel may be the general meeting
447  point, but it is not the best place to find developers from all
448  subsystems.
449
450The last point - finding the correct mailing list - is a common place for
451beginning developers to go wrong.  Somebody who asks a networking-related
452question on linux-kernel will almost certainly receive a polite suggestion
453to ask on the netdev list instead, as that is the list frequented by most
454networking developers.  Other lists exist for the SCSI, video4linux, IDE,
455filesystem, etc. subsystems.  The best place to look for mailing lists is
456in the MAINTAINERS file packaged with the kernel source.
457
458
459Getting started with Kernel development
460---------------------------------------
461
462Questions about how to get started with the kernel development process are
463common - from both individuals and companies.  Equally common are missteps
464which make the beginning of the relationship harder than it has to be.
465
466Companies often look to hire well-known developers to get a development
467group started.  This can, in fact, be an effective technique.  But it also
468tends to be expensive and does not do much to grow the pool of experienced
469kernel developers.  It is possible to bring in-house developers up to speed
470on Linux kernel development, given the investment of a bit of time.  Taking
471this time can endow an employer with a group of developers who understand
472the kernel and the company both, and who can help to train others as well.
473Over the medium term, this is often the more profitable approach.
474
475Individual developers are often, understandably, at a loss for a place to
476start.  Beginning with a large project can be intimidating; one often wants
477to test the waters with something smaller first.  This is the point where
478some developers jump into the creation of patches fixing spelling errors or
479minor coding style issues.  Unfortunately, such patches create a level of
480noise which is distracting for the development community as a whole, so,
481increasingly, they are looked down upon.  New developers wishing to
482introduce themselves to the community will not get the sort of reception
483they wish for by these means.
484
485Andrew Morton gives this advice for aspiring kernel developers
486
487::
488
489	The #1 project for all kernel beginners should surely be "make sure
490	that the kernel runs perfectly at all times on all machines which
491	you can lay your hands on".  Usually the way to do this is to work
492	with others on getting things fixed up (this can require
493	persistence!) but that's fine - it's a part of kernel development.
494
495(http://lwn.net/Articles/283982/).
496
497In the absence of obvious problems to fix, developers are advised to look
498at the current lists of regressions and open bugs in general.  There is
499never any shortage of issues in need of fixing; by addressing these issues,
500developers will gain experience with the process while, at the same time,
501building respect with the rest of the development community.
502