1.. SPDX-License-Identifier: GPL-2.0
2.. include:: <isonum.txt>
3
4.. |struct cpuidle_state| replace:: :c:type:`struct cpuidle_state <cpuidle_state>`
5.. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>`
6
7========================
8CPU Idle Time Management
9========================
10
11:Copyright: |copy| 2018 Intel Corporation
12
13:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
14
15
16Concepts
17========
18
19Modern processors are generally able to enter states in which the execution of
20a program is suspended and instructions belonging to it are not fetched from
21memory or executed.  Those states are the *idle* states of the processor.
22
23Since part of the processor hardware is not used in idle states, entering them
24generally allows power drawn by the processor to be reduced and, in consequence,
25it is an opportunity to save energy.
26
27CPU idle time management is an energy-efficiency feature concerned about using
28the idle states of processors for this purpose.
29
30Logical CPUs
31------------
32
33CPU idle time management operates on CPUs as seen by the *CPU scheduler* (that
34is the part of the kernel responsible for the distribution of computational
35work in the system).  In its view, CPUs are *logical* units.  That is, they need
36not be separate physical entities and may just be interfaces appearing to
37software as individual single-core processors.  In other words, a CPU is an
38entity which appears to be fetching instructions that belong to one sequence
39(program) from memory and executing them, but it need not work this way
40physically.  Generally, three different cases can be consider here.
41
42First, if the whole processor can only follow one sequence of instructions (one
43program) at a time, it is a CPU.  In that case, if the hardware is asked to
44enter an idle state, that applies to the processor as a whole.
45
46Second, if the processor is multi-core, each core in it is able to follow at
47least one program at a time.  The cores need not be entirely independent of each
48other (for example, they may share caches), but still most of the time they
49work physically in parallel with each other, so if each of them executes only
50one program, those programs run mostly independently of each other at the same
51time.  The entire cores are CPUs in that case and if the hardware is asked to
52enter an idle state, that applies to the core that asked for it in the first
53place, but it also may apply to a larger unit (say a "package" or a "cluster")
54that the core belongs to (in fact, it may apply to an entire hierarchy of larger
55units containing the core).  Namely, if all of the cores in the larger unit
56except for one have been put into idle states at the "core level" and the
57remaining core asks the processor to enter an idle state, that may trigger it
58to put the whole larger unit into an idle state which also will affect the
59other cores in that unit.
60
61Finally, each core in a multi-core processor may be able to follow more than one
62program in the same time frame (that is, each core may be able to fetch
63instructions from multiple locations in memory and execute them in the same time
64frame, but not necessarily entirely in parallel with each other).  In that case
65the cores present themselves to software as "bundles" each consisting of
66multiple individual single-core "processors", referred to as *hardware threads*
67(or hyper-threads specifically on Intel hardware), that each can follow one
68sequence of instructions.  Then, the hardware threads are CPUs from the CPU idle
69time management perspective and if the processor is asked to enter an idle state
70by one of them, the hardware thread (or CPU) that asked for it is stopped, but
71nothing more happens, unless all of the other hardware threads within the same
72core also have asked the processor to enter an idle state.  In that situation,
73the core may be put into an idle state individually or a larger unit containing
74it may be put into an idle state as a whole (if the other cores within the
75larger unit are in idle states already).
76
77Idle CPUs
78---------
79
80Logical CPUs, simply referred to as "CPUs" in what follows, are regarded as
81*idle* by the Linux kernel when there are no tasks to run on them except for the
82special "idle" task.
83
84Tasks are the CPU scheduler's representation of work.  Each task consists of a
85sequence of instructions to execute, or code, data to be manipulated while
86running that code, and some context information that needs to be loaded into the
87processor every time the task's code is run by a CPU.  The CPU scheduler
88distributes work by assigning tasks to run to the CPUs present in the system.
89
90Tasks can be in various states.  In particular, they are *runnable* if there are
91no specific conditions preventing their code from being run by a CPU as long as
92there is a CPU available for that (for example, they are not waiting for any
93events to occur or similar).  When a task becomes runnable, the CPU scheduler
94assigns it to one of the available CPUs to run and if there are no more runnable
95tasks assigned to it, the CPU will load the given task's context and run its
96code (from the instruction following the last one executed so far, possibly by
97another CPU).  [If there are multiple runnable tasks assigned to one CPU
98simultaneously, they will be subject to prioritization and time sharing in order
99to allow them to make some progress over time.]
100
101The special "idle" task becomes runnable if there are no other runnable tasks
102assigned to the given CPU and the CPU is then regarded as idle.  In other words,
103in Linux idle CPUs run the code of the "idle" task called *the idle loop*.  That
104code may cause the processor to be put into one of its idle states, if they are
105supported, in order to save energy, but if the processor does not support any
106idle states, or there is not enough time to spend in an idle state before the
107next wakeup event, or there are strict latency constraints preventing any of the
108available idle states from being used, the CPU will simply execute more or less
109useless instructions in a loop until it is assigned a new task to run.
110
111
112.. _idle-loop:
113
114The Idle Loop
115=============
116
117The idle loop code takes two major steps in every iteration of it.  First, it
118calls into a code module referred to as the *governor* that belongs to the CPU
119idle time management subsystem called ``CPUIdle`` to select an idle state for
120the CPU to ask the hardware to enter.  Second, it invokes another code module
121from the ``CPUIdle`` subsystem, called the *driver*, to actually ask the
122processor hardware to enter the idle state selected by the governor.
123
124The role of the governor is to find an idle state most suitable for the
125conditions at hand.  For this purpose, idle states that the hardware can be
126asked to enter by logical CPUs are represented in an abstract way independent of
127the platform or the processor architecture and organized in a one-dimensional
128(linear) array.  That array has to be prepared and supplied by the ``CPUIdle``
129driver matching the platform the kernel is running on at the initialization
130time.  This allows ``CPUIdle`` governors to be independent of the underlying
131hardware and to work with any platforms that the Linux kernel can run on.
132
133Each idle state present in that array is characterized by two parameters to be
134taken into account by the governor, the *target residency* and the (worst-case)
135*exit latency*.  The target residency is the minimum time the hardware must
136spend in the given state, including the time needed to enter it (which may be
137substantial), in order to save more energy than it would save by entering one of
138the shallower idle states instead.  [The "depth" of an idle state roughly
139corresponds to the power drawn by the processor in that state.]  The exit
140latency, in turn, is the maximum time it will take a CPU asking the processor
141hardware to enter an idle state to start executing the first instruction after a
142wakeup from that state.  Note that in general the exit latency also must cover
143the time needed to enter the given state in case the wakeup occurs when the
144hardware is entering it and it must be entered completely to be exited in an
145ordered manner.
146
147There are two types of information that can influence the governor's decisions.
148First of all, the governor knows the time until the closest timer event.  That
149time is known exactly, because the kernel programs timers and it knows exactly
150when they will trigger, and it is the maximum time the hardware that the given
151CPU depends on can spend in an idle state, including the time necessary to enter
152and exit it.  However, the CPU may be woken up by a non-timer event at any time
153(in particular, before the closest timer triggers) and it generally is not known
154when that may happen.  The governor can only see how much time the CPU actually
155was idle after it has been woken up (that time will be referred to as the *idle
156duration* from now on) and it can use that information somehow along with the
157time until the closest timer to estimate the idle duration in future.  How the
158governor uses that information depends on what algorithm is implemented by it
159and that is the primary reason for having more than one governor in the
160``CPUIdle`` subsystem.
161
162There are three ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_
163and ``ladder``.  Which of them is used by default depends on the configuration
164of the kernel and in particular on whether or not the scheduler tick can be
165`stopped by the idle loop <idle-cpus-and-tick_>`_.  It is possible to change the
166governor at run time if the ``cpuidle_sysfs_switch`` command line parameter has
167been passed to the kernel, but that is not safe in general, so it should not be
168done on production systems (that may change in the future, though).  The name of
169the ``CPUIdle`` governor currently used by the kernel can be read from the
170:file:`current_governor_ro` (or :file:`current_governor` if
171``cpuidle_sysfs_switch`` is present in the kernel command line) file under
172:file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``.
173
174Which ``CPUIdle`` driver is used, on the other hand, usually depends on the
175platform the kernel is running on, but there are platforms with more than one
176matching driver.  For example, there are two drivers that can work with the
177majority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with
178hardcoded idle states information and the other able to read that information
179from the system's ACPI tables, respectively.  Still, even in those cases, the
180driver chosen at the system initialization time cannot be replaced later, so the
181decision on which one of them to use has to be made early (on Intel platforms
182the ``acpi_idle`` driver will be used if ``intel_idle`` is disabled for some
183reason or if it does not recognize the processor).  The name of the ``CPUIdle``
184driver currently used by the kernel can be read from the :file:`current_driver`
185file under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``.
186
187
188.. _idle-cpus-and-tick:
189
190Idle CPUs and The Scheduler Tick
191================================
192
193The scheduler tick is a timer that triggers periodically in order to implement
194the time sharing strategy of the CPU scheduler.  Of course, if there are
195multiple runnable tasks assigned to one CPU at the same time, the only way to
196allow them to make reasonable progress in a given time frame is to make them
197share the available CPU time.  Namely, in rough approximation, each task is
198given a slice of the CPU time to run its code, subject to the scheduling class,
199prioritization and so on and when that time slice is used up, the CPU should be
200switched over to running (the code of) another task.  The currently running task
201may not want to give the CPU away voluntarily, however, and the scheduler tick
202is there to make the switch happen regardless.  That is not the only role of the
203tick, but it is the primary reason for using it.
204
205The scheduler tick is problematic from the CPU idle time management perspective,
206because it triggers periodically and relatively often (depending on the kernel
207configuration, the length of the tick period is between 1 ms and 10 ms).
208Thus, if the tick is allowed to trigger on idle CPUs, it will not make sense
209for them to ask the hardware to enter idle states with target residencies above
210the tick period length.  Moreover, in that case the idle duration of any CPU
211will never exceed the tick period length and the energy used for entering and
212exiting idle states due to the tick wakeups on idle CPUs will be wasted.
213
214Fortunately, it is not really necessary to allow the tick to trigger on idle
215CPUs, because (by definition) they have no tasks to run except for the special
216"idle" one.  In other words, from the CPU scheduler perspective, the only user
217of the CPU time on them is the idle loop.  Since the time of an idle CPU need
218not be shared between multiple runnable tasks, the primary reason for using the
219tick goes away if the given CPU is idle.  Consequently, it is possible to stop
220the scheduler tick entirely on idle CPUs in principle, even though that may not
221always be worth the effort.
222
223Whether or not it makes sense to stop the scheduler tick in the idle loop
224depends on what is expected by the governor.  First, if there is another
225(non-tick) timer due to trigger within the tick range, stopping the tick clearly
226would be a waste of time, even though the timer hardware may not need to be
227reprogrammed in that case.  Second, if the governor is expecting a non-timer
228wakeup within the tick range, stopping the tick is not necessary and it may even
229be harmful.  Namely, in that case the governor will select an idle state with
230the target residency within the time until the expected wakeup, so that state is
231going to be relatively shallow.  The governor really cannot select a deep idle
232state then, as that would contradict its own expectation of a wakeup in short
233order.  Now, if the wakeup really occurs shortly, stopping the tick would be a
234waste of time and in this case the timer hardware would need to be reprogrammed,
235which is expensive.  On the other hand, if the tick is stopped and the wakeup
236does not occur any time soon, the hardware may spend indefinite amount of time
237in the shallow idle state selected by the governor, which will be a waste of
238energy.  Hence, if the governor is expecting a wakeup of any kind within the
239tick range, it is better to allow the tick trigger.  Otherwise, however, the
240governor will select a relatively deep idle state, so the tick should be stopped
241so that it does not wake up the CPU too early.
242
243In any case, the governor knows what it is expecting and the decision on whether
244or not to stop the scheduler tick belongs to it.  Still, if the tick has been
245stopped already (in one of the previous iterations of the loop), it is better
246to leave it as is and the governor needs to take that into account.
247
248The kernel can be configured to disable stopping the scheduler tick in the idle
249loop altogether.  That can be done through the build-time configuration of it
250(by unsetting the ``CONFIG_NO_HZ_IDLE`` configuration option) or by passing
251``nohz=off`` to it in the command line.  In both cases, as the stopping of the
252scheduler tick is disabled, the governor's decisions regarding it are simply
253ignored by the idle loop code and the tick is never stopped.
254
255The systems that run kernels configured to allow the scheduler tick to be
256stopped on idle CPUs are referred to as *tickless* systems and they are
257generally regarded as more energy-efficient than the systems running kernels in
258which the tick cannot be stopped.  If the given system is tickless, it will use
259the ``menu`` governor by default and if it is not tickless, the default
260``CPUIdle`` governor on it will be ``ladder``.
261
262
263.. _menu-gov:
264
265The ``menu`` Governor
266=====================
267
268The ``menu`` governor is the default ``CPUIdle`` governor for tickless systems.
269It is quite complex, but the basic principle of its design is straightforward.
270Namely, when invoked to select an idle state for a CPU (i.e. an idle state that
271the CPU will ask the processor hardware to enter), it attempts to predict the
272idle duration and uses the predicted value for idle state selection.
273
274It first obtains the time until the closest timer event with the assumption
275that the scheduler tick will be stopped.  That time, referred to as the *sleep
276length* in what follows, is the upper bound on the time before the next CPU
277wakeup.  It is used to determine the sleep length range, which in turn is needed
278to get the sleep length correction factor.
279
280The ``menu`` governor maintains two arrays of sleep length correction factors.
281One of them is used when tasks previously running on the given CPU are waiting
282for some I/O operations to complete and the other one is used when that is not
283the case.  Each array contains several correction factor values that correspond
284to different sleep length ranges organized so that each range represented in the
285array is approximately 10 times wider than the previous one.
286
287The correction factor for the given sleep length range (determined before
288selecting the idle state for the CPU) is updated after the CPU has been woken
289up and the closer the sleep length is to the observed idle duration, the closer
290to 1 the correction factor becomes (it must fall between 0 and 1 inclusive).
291The sleep length is multiplied by the correction factor for the range that it
292falls into to obtain the first approximation of the predicted idle duration.
293
294Next, the governor uses a simple pattern recognition algorithm to refine its
295idle duration prediction.  Namely, it saves the last 8 observed idle duration
296values and, when predicting the idle duration next time, it computes the average
297and variance of them.  If the variance is small (smaller than 400 square
298milliseconds) or it is small relative to the average (the average is greater
299that 6 times the standard deviation), the average is regarded as the "typical
300interval" value.  Otherwise, the longest of the saved observed idle duration
301values is discarded and the computation is repeated for the remaining ones.
302Again, if the variance of them is small (in the above sense), the average is
303taken as the "typical interval" value and so on, until either the "typical
304interval" is determined or too many data points are disregarded, in which case
305the "typical interval" is assumed to equal "infinity" (the maximum unsigned
306integer value).  The "typical interval" computed this way is compared with the
307sleep length multiplied by the correction factor and the minimum of the two is
308taken as the predicted idle duration.
309
310Then, the governor computes an extra latency limit to help "interactive"
311workloads.  It uses the observation that if the exit latency of the selected
312idle state is comparable with the predicted idle duration, the total time spent
313in that state probably will be very short and the amount of energy to save by
314entering it will be relatively small, so likely it is better to avoid the
315overhead related to entering that state and exiting it.  Thus selecting a
316shallower state is likely to be a better option then.   The first approximation
317of the extra latency limit is the predicted idle duration itself which
318additionally is divided by a value depending on the number of tasks that
319previously ran on the given CPU and now they are waiting for I/O operations to
320complete.  The result of that division is compared with the latency limit coming
321from the power management quality of service, or `PM QoS <cpu-pm-qos_>`_,
322framework and the minimum of the two is taken as the limit for the idle states'
323exit latency.
324
325Now, the governor is ready to walk the list of idle states and choose one of
326them.  For this purpose, it compares the target residency of each state with
327the predicted idle duration and the exit latency of it with the computed latency
328limit.  It selects the state with the target residency closest to the predicted
329idle duration, but still below it, and exit latency that does not exceed the
330limit.
331
332In the final step the governor may still need to refine the idle state selection
333if it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_.  That
334happens if the idle duration predicted by it is less than the tick period and
335the tick has not been stopped already (in a previous iteration of the idle
336loop).  Then, the sleep length used in the previous computations may not reflect
337the real time until the closest timer event and if it really is greater than
338that time, the governor may need to select a shallower state with a suitable
339target residency.
340
341
342.. _teo-gov:
343
344The Timer Events Oriented (TEO) Governor
345========================================
346
347The timer events oriented (TEO) governor is an alternative ``CPUIdle`` governor
348for tickless systems.  It follows the same basic strategy as the ``menu`` `one
349<menu-gov_>`_: it always tries to find the deepest idle state suitable for the
350given conditions.  However, it applies a different approach to that problem.
351
352First, it does not use sleep length correction factors, but instead it attempts
353to correlate the observed idle duration values with the available idle states
354and use that information to pick up the idle state that is most likely to
355"match" the upcoming CPU idle interval.   Second, it does not take the tasks
356that were running on the given CPU in the past and are waiting on some I/O
357operations to complete now at all (there is no guarantee that they will run on
358the same CPU when they become runnable again) and the pattern detection code in
359it avoids taking timer wakeups into account.  It also only uses idle duration
360values less than the current time till the closest timer (with the scheduler
361tick excluded) for that purpose.
362
363Like in the ``menu`` governor `case <menu-gov_>`_, the first step is to obtain
364the *sleep length*, which is the time until the closest timer event with the
365assumption that the scheduler tick will be stopped (that also is the upper bound
366on the time until the next CPU wakeup).  That value is then used to preselect an
367idle state on the basis of three metrics maintained for each idle state provided
368by the ``CPUIdle`` driver: ``hits``, ``misses`` and ``early_hits``.
369
370The ``hits`` and ``misses`` metrics measure the likelihood that a given idle
371state will "match" the observed (post-wakeup) idle duration if it "matches" the
372sleep length.  They both are subject to decay (after a CPU wakeup) every time
373the target residency of the idle state corresponding to them is less than or
374equal to the sleep length and the target residency of the next idle state is
375greater than the sleep length (that is, when the idle state corresponding to
376them "matches" the sleep length).  The ``hits`` metric is increased if the
377former condition is satisfied and the target residency of the given idle state
378is less than or equal to the observed idle duration and the target residency of
379the next idle state is greater than the observed idle duration at the same time
380(that is, it is increased when the given idle state "matches" both the sleep
381length and the observed idle duration).  In turn, the ``misses`` metric is
382increased when the given idle state "matches" the sleep length only and the
383observed idle duration is too short for its target residency.
384
385The ``early_hits`` metric measures the likelihood that a given idle state will
386"match" the observed (post-wakeup) idle duration if it does not "match" the
387sleep length.  It is subject to decay on every CPU wakeup and it is increased
388when the idle state corresponding to it "matches" the observed (post-wakeup)
389idle duration and the target residency of the next idle state is less than or
390equal to the sleep length (i.e. the idle state "matching" the sleep length is
391deeper than the given one).
392
393The governor walks the list of idle states provided by the ``CPUIdle`` driver
394and finds the last (deepest) one with the target residency less than or equal
395to the sleep length.  Then, the ``hits`` and ``misses`` metrics of that idle
396state are compared with each other and it is preselected if the ``hits`` one is
397greater (which means that that idle state is likely to "match" the observed idle
398duration after CPU wakeup).  If the ``misses`` one is greater, the governor
399preselects the shallower idle state with the maximum ``early_hits`` metric
400(or if there are multiple shallower idle states with equal ``early_hits``
401metric which also is the maximum, the shallowest of them will be preselected).
402[If there is a wakeup latency constraint coming from the `PM QoS framework
403<cpu-pm-qos_>`_ which is hit before reaching the deepest idle state with the
404target residency within the sleep length, the deepest idle state with the exit
405latency within the constraint is preselected without consulting the ``hits``,
406``misses`` and ``early_hits`` metrics.]
407
408Next, the governor takes several idle duration values observed most recently
409into consideration and if at least a half of them are greater than or equal to
410the target residency of the preselected idle state, that idle state becomes the
411final candidate to ask for.  Otherwise, the average of the most recent idle
412duration values below the target residency of the preselected idle state is
413computed and the governor walks the idle states shallower than the preselected
414one and finds the deepest of them with the target residency within that average.
415That idle state is then taken as the final candidate to ask for.
416
417Still, at this point the governor may need to refine the idle state selection if
418it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_.  That
419generally happens if the target residency of the idle state selected so far is
420less than the tick period and the tick has not been stopped already (in a
421previous iteration of the idle loop).  Then, like in the ``menu`` governor
422`case <menu-gov_>`_, the sleep length used in the previous computations may not
423reflect the real time until the closest timer event and if it really is greater
424than that time, a shallower state with a suitable target residency may need to
425be selected.
426
427
428.. _idle-states-representation:
429
430Representation of Idle States
431=============================
432
433For the CPU idle time management purposes all of the physical idle states
434supported by the processor have to be represented as a one-dimensional array of
435|struct cpuidle_state| objects each allowing an individual (logical) CPU to ask
436the processor hardware to enter an idle state of certain properties.  If there
437is a hierarchy of units in the processor, one |struct cpuidle_state| object can
438cover a combination of idle states supported by the units at different levels of
439the hierarchy.  In that case, the `target residency and exit latency parameters
440of it <idle-loop_>`_, must reflect the properties of the idle state at the
441deepest level (i.e. the idle state of the unit containing all of the other
442units).
443
444For example, take a processor with two cores in a larger unit referred to as
445a "module" and suppose that asking the hardware to enter a specific idle state
446(say "X") at the "core" level by one core will trigger the module to try to
447enter a specific idle state of its own (say "MX") if the other core is in idle
448state "X" already.  In other words, asking for idle state "X" at the "core"
449level gives the hardware a license to go as deep as to idle state "MX" at the
450"module" level, but there is no guarantee that this is going to happen (the core
451asking for idle state "X" may just end up in that state by itself instead).
452Then, the target residency of the |struct cpuidle_state| object representing
453idle state "X" must reflect the minimum time to spend in idle state "MX" of
454the module (including the time needed to enter it), because that is the minimum
455time the CPU needs to be idle to save any energy in case the hardware enters
456that state.  Analogously, the exit latency parameter of that object must cover
457the exit time of idle state "MX" of the module (and usually its entry time too),
458because that is the maximum delay between a wakeup signal and the time the CPU
459will start to execute the first new instruction (assuming that both cores in the
460module will always be ready to execute instructions as soon as the module
461becomes operational as a whole).
462
463There are processors without direct coordination between different levels of the
464hierarchy of units inside them, however.  In those cases asking for an idle
465state at the "core" level does not automatically affect the "module" level, for
466example, in any way and the ``CPUIdle`` driver is responsible for the entire
467handling of the hierarchy.  Then, the definition of the idle state objects is
468entirely up to the driver, but still the physical properties of the idle state
469that the processor hardware finally goes into must always follow the parameters
470used by the governor for idle state selection (for instance, the actual exit
471latency of that idle state must not exceed the exit latency parameter of the
472idle state object selected by the governor).
473
474In addition to the target residency and exit latency idle state parameters
475discussed above, the objects representing idle states each contain a few other
476parameters describing the idle state and a pointer to the function to run in
477order to ask the hardware to enter that state.  Also, for each
478|struct cpuidle_state| object, there is a corresponding
479:c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage
480statistics of the given idle state.  That information is exposed by the kernel
481via ``sysfs``.
482
483For each CPU in the system, there is a :file:`/sys/devices/system/cpu<N>/cpuidle/`
484directory in ``sysfs``, where the number ``<N>`` is assigned to the given
485CPU at the initialization time.  That directory contains a set of subdirectories
486called :file:`state0`, :file:`state1` and so on, up to the number of idle state
487objects defined for the given CPU minus one.  Each of these directories
488corresponds to one idle state object and the larger the number in its name, the
489deeper the (effective) idle state represented by it.  Each of them contains
490a number of files (attributes) representing the properties of the idle state
491object corresponding to it, as follows:
492
493``above``
494	Total number of times this idle state had been asked for, but the
495	observed idle duration was certainly too short to match its target
496	residency.
497
498``below``
499	Total number of times this idle state had been asked for, but cerainly
500	a deeper idle state would have been a better match for the observed idle
501	duration.
502
503``desc``
504	Description of the idle state.
505
506``disable``
507	Whether or not this idle state is disabled.
508
509``latency``
510	Exit latency of the idle state in microseconds.
511
512``name``
513	Name of the idle state.
514
515``power``
516	Power drawn by hardware in this idle state in milliwatts (if specified,
517	0 otherwise).
518
519``residency``
520	Target residency of the idle state in microseconds.
521
522``time``
523	Total time spent in this idle state by the given CPU (as measured by the
524	kernel) in microseconds.
525
526``usage``
527	Total number of times the hardware has been asked by the given CPU to
528	enter this idle state.
529
530The :file:`desc` and :file:`name` files both contain strings.  The difference
531between them is that the name is expected to be more concise, while the
532description may be longer and it may contain white space or special characters.
533The other files listed above contain integer numbers.
534
535The :file:`disable` attribute is the only writeable one.  If it contains 1, the
536given idle state is disabled for this particular CPU, which means that the
537governor will never select it for this particular CPU and the ``CPUIdle``
538driver will never ask the hardware to enter it for that CPU as a result.
539However, disabling an idle state for one CPU does not prevent it from being
540asked for by the other CPUs, so it must be disabled for all of them in order to
541never be asked for by any of them.  [Note that, due to the way the ``ladder``
542governor is implemented, disabling an idle state prevents that governor from
543selecting any idle states deeper than the disabled one too.]
544
545If the :file:`disable` attribute contains 0, the given idle state is enabled for
546this particular CPU, but it still may be disabled for some or all of the other
547CPUs in the system at the same time.  Writing 1 to it causes the idle state to
548be disabled for this particular CPU and writing 0 to it allows the governor to
549take it into consideration for the given CPU and the driver to ask for it,
550unless that state was disabled globally in the driver (in which case it cannot
551be used at all).
552
553The :file:`power` attribute is not defined very well, especially for idle state
554objects representing combinations of idle states at different levels of the
555hierarchy of units in the processor, and it generally is hard to obtain idle
556state power numbers for complex hardware, so :file:`power` often contains 0 (not
557available) and if it contains a nonzero number, that number may not be very
558accurate and it should not be relied on for anything meaningful.
559
560The number in the :file:`time` file generally may be greater than the total time
561really spent by the given CPU in the given idle state, because it is measured by
562the kernel and it may not cover the cases in which the hardware refused to enter
563this idle state and entered a shallower one instead of it (or even it did not
564enter any idle state at all).  The kernel can only measure the time span between
565asking the hardware to enter an idle state and the subsequent wakeup of the CPU
566and it cannot say what really happened in the meantime at the hardware level.
567Moreover, if the idle state object in question represents a combination of idle
568states at different levels of the hierarchy of units in the processor,
569the kernel can never say how deep the hardware went down the hierarchy in any
570particular case.  For these reasons, the only reliable way to find out how
571much time has been spent by the hardware in different idle states supported by
572it is to use idle state residency counters in the hardware, if available.
573
574
575.. _cpu-pm-qos:
576
577Power Management Quality of Service for CPUs
578============================================
579
580The power management quality of service (PM QoS) framework in the Linux kernel
581allows kernel code and user space processes to set constraints on various
582energy-efficiency features of the kernel to prevent performance from dropping
583below a required level.  The PM QoS constraints can be set globally, in
584predefined categories referred to as PM QoS classes, or against individual
585devices.
586
587CPU idle time management can be affected by PM QoS in two ways, through the
588global constraint in the ``PM_QOS_CPU_DMA_LATENCY`` class and through the
589resume latency constraints for individual CPUs.  Kernel code (e.g. device
590drivers) can set both of them with the help of special internal interfaces
591provided by the PM QoS framework.  User space can modify the former by opening
592the :file:`cpu_dma_latency` special device file under :file:`/dev/` and writing
593a binary value (interpreted as a signed 32-bit integer) to it.  In turn, the
594resume latency constraint for a CPU can be modified by user space by writing a
595string (representing a signed 32-bit integer) to the
596:file:`power/pm_qos_resume_latency_us` file under
597:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number
598``<N>`` is allocated at the system initialization time.  Negative values
599will be rejected in both cases and, also in both cases, the written integer
600number will be interpreted as a requested PM QoS constraint in microseconds.
601
602The requested value is not automatically applied as a new constraint, however,
603as it may be less restrictive (greater in this particular case) than another
604constraint previously requested by someone else.  For this reason, the PM QoS
605framework maintains a list of requests that have been made so far in each
606global class and for each device, aggregates them and applies the effective
607(minimum in this particular case) value as the new constraint.
608
609In fact, opening the :file:`cpu_dma_latency` special device file causes a new
610PM QoS request to be created and added to the priority list of requests in the
611``PM_QOS_CPU_DMA_LATENCY`` class and the file descriptor coming from the
612"open" operation represents that request.  If that file descriptor is then
613used for writing, the number written to it will be associated with the PM QoS
614request represented by it as a new requested constraint value.  Next, the
615priority list mechanism will be used to determine the new effective value of
616the entire list of requests and that effective value will be set as a new
617constraint.  Thus setting a new requested constraint value will only change the
618real constraint if the effective "list" value is affected by it.  In particular,
619for the ``PM_QOS_CPU_DMA_LATENCY`` class it only affects the real constraint if
620it is the minimum of the requested constraints in the list.  The process holding
621a file descriptor obtained by opening the :file:`cpu_dma_latency` special device
622file controls the PM QoS request associated with that file descriptor, but it
623controls this particular PM QoS request only.
624
625Closing the :file:`cpu_dma_latency` special device file or, more precisely, the
626file descriptor obtained while opening it, causes the PM QoS request associated
627with that file descriptor to be removed from the ``PM_QOS_CPU_DMA_LATENCY``
628class priority list and destroyed.  If that happens, the priority list mechanism
629will be used, again, to determine the new effective value for the whole list
630and that value will become the new real constraint.
631
632In turn, for each CPU there is only one resume latency PM QoS request
633associated with the :file:`power/pm_qos_resume_latency_us` file under
634:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes
635this single PM QoS request to be updated regardless of which user space
636process does that.  In other words, this PM QoS request is shared by the entire
637user space, so access to the file associated with it needs to be arbitrated
638to avoid confusion.  [Arguably, the only legitimate use of this mechanism in
639practice is to pin a process to the CPU in question and let it use the
640``sysfs`` interface to control the resume latency constraint for it.]  It
641still only is a request, however.  It is a member of a priority list used to
642determine the effective value to be set as the resume latency constraint for the
643CPU in question every time the list of requests is updated this way or another
644(there may be other requests coming from kernel code in that list).
645
646CPU idle time governors are expected to regard the minimum of the global
647effective ``PM_QOS_CPU_DMA_LATENCY`` class constraint and the effective
648resume latency constraint for the given CPU as the upper limit for the exit
649latency of the idle states they can select for that CPU.  They should never
650select any idle states with exit latency beyond that limit.
651
652
653Idle States Control Via Kernel Command Line
654===========================================
655
656In addition to the ``sysfs`` interface allowing individual idle states to be
657`disabled for individual CPUs <idle-states-representation_>`_, there are kernel
658command line parameters affecting CPU idle time management.
659
660The ``cpuidle.off=1`` kernel command line option can be used to disable the
661CPU idle time management entirely.  It does not prevent the idle loop from
662running on idle CPUs, but it prevents the CPU idle time governors and drivers
663from being invoked.  If it is added to the kernel command line, the idle loop
664will ask the hardware to enter idle states on idle CPUs via the CPU architecture
665support code that is expected to provide a default mechanism for this purpose.
666That default mechanism usually is the least common denominator for all of the
667processors implementing the architecture (i.e. CPU instruction set) in question,
668however, so it is rather crude and not very energy-efficient.  For this reason,
669it is not recommended for production use.
670
671The ``cpuidle.governor=`` kernel command line switch allows the ``CPUIdle``
672governor to use to be specified.  It has to be appended with a string matching
673the name of an available governor (e.g. ``cpuidle.governor=menu``) and that
674governor will be used instead of the default one.  It is possible to force
675the ``menu`` governor to be used on the systems that use the ``ladder`` governor
676by default this way, for example.
677
678The other kernel command line parameters controlling CPU idle time management
679described below are only relevant for the *x86* architecture and some of
680them affect Intel processors only.
681
682The *x86* architecture support code recognizes three kernel command line
683options related to CPU idle time management: ``idle=poll``, ``idle=halt``,
684and ``idle=nomwait``.  The first two of them disable the ``acpi_idle`` and
685``intel_idle`` drivers altogether, which effectively causes the entire
686``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the
687architecture support code to deal with idle CPUs.  How it does that depends on
688which of the two parameters is added to the kernel command line.  In the
689``idle=halt`` case, the architecture support code will use the ``HLT``
690instruction of the CPUs (which, as a rule, suspends the execution of the program
691and causes the hardware to attempt to enter the shallowest available idle state)
692for this purpose, and if ``idle=poll`` is used, idle CPUs will execute a
693more or less ``lightweight'' sequence of instructions in a tight loop.  [Note
694that using ``idle=poll`` is somewhat drastic in many cases, as preventing idle
695CPUs from saving almost any energy at all may not be the only effect of it.
696For example, on Intel hardware it effectively prevents CPUs from using
697P-states (see |cpufreq|) that require any number of CPUs in a package to be
698idle, so it very well may hurt single-thread computations performance as well as
699energy-efficiency.  Thus using it for performance reasons may not be a good idea
700at all.]
701
702The ``idle=nomwait`` option disables the ``intel_idle`` driver and causes
703``acpi_idle`` to be used (as long as all of the information needed by it is
704there in the system's ACPI tables), but it is not allowed to use the
705``MWAIT`` instruction of the CPUs to ask the hardware to enter idle states.
706
707In addition to the architecture-level kernel command line options affecting CPU
708idle time management, there are parameters affecting individual ``CPUIdle``
709drivers that can be passed to them via the kernel command line.  Specifically,
710the ``intel_idle.max_cstate=<n>`` and ``processor.max_cstate=<n>`` parameters,
711where ``<n>`` is an idle state index also used in the name of the given
712state's directory in ``sysfs`` (see
713`Representation of Idle States <idle-states-representation_>`_), causes the
714``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the
715idle states deeper than idle state ``<n>``.  In that case, they will never ask
716for any of those idle states or expose them to the governor.  [The behavior of
717the two drivers is different for ``<n>`` equal to ``0``.  Adding
718``intel_idle.max_cstate=0`` to the kernel command line disables the
719``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas
720``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``.
721Also, the ``acpi_idle`` driver is part of the ``processor`` kernel module that
722can be loaded separately and ``max_cstate=<n>`` can be passed to it as a module
723parameter when it is loaded.]
724