1fc7db767SRafael J. Wysocki.. SPDX-License-Identifier: GPL-2.0 2fc1860d6SRafael J. Wysocki.. include:: <isonum.txt> 3fc7db767SRafael J. Wysocki 4aa5eee35SRafael J. Wysocki.. |struct cpuidle_state| replace:: :c:type:`struct cpuidle_state <cpuidle_state>` 5aa5eee35SRafael J. Wysocki.. |cpufreq| replace:: :doc:`CPU Performance Scaling <cpufreq>` 6aa5eee35SRafael J. Wysocki 7aa5eee35SRafael J. Wysocki======================== 8aa5eee35SRafael J. WysockiCPU Idle Time Management 9aa5eee35SRafael J. Wysocki======================== 10aa5eee35SRafael J. Wysocki 11fc1860d6SRafael J. Wysocki:Copyright: |copy| 2018 Intel Corporation 12aa5eee35SRafael J. Wysocki 13fc1860d6SRafael J. Wysocki:Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> 14fc1860d6SRafael J. Wysocki 15aa5eee35SRafael J. Wysocki 16aa5eee35SRafael J. WysockiConcepts 17aa5eee35SRafael J. Wysocki======== 18aa5eee35SRafael J. Wysocki 19aa5eee35SRafael J. WysockiModern processors are generally able to enter states in which the execution of 20aa5eee35SRafael J. Wysockia program is suspended and instructions belonging to it are not fetched from 21aa5eee35SRafael J. Wysockimemory or executed. Those states are the *idle* states of the processor. 22aa5eee35SRafael J. Wysocki 23aa5eee35SRafael J. WysockiSince part of the processor hardware is not used in idle states, entering them 24aa5eee35SRafael J. Wysockigenerally allows power drawn by the processor to be reduced and, in consequence, 25aa5eee35SRafael J. Wysockiit is an opportunity to save energy. 26aa5eee35SRafael J. Wysocki 27aa5eee35SRafael J. WysockiCPU idle time management is an energy-efficiency feature concerned about using 28aa5eee35SRafael J. Wysockithe idle states of processors for this purpose. 29aa5eee35SRafael J. Wysocki 30aa5eee35SRafael J. WysockiLogical CPUs 31aa5eee35SRafael J. Wysocki------------ 32aa5eee35SRafael J. Wysocki 33aa5eee35SRafael J. WysockiCPU idle time management operates on CPUs as seen by the *CPU scheduler* (that 34aa5eee35SRafael J. Wysockiis the part of the kernel responsible for the distribution of computational 35aa5eee35SRafael J. Wysockiwork in the system). In its view, CPUs are *logical* units. That is, they need 36aa5eee35SRafael J. Wysockinot be separate physical entities and may just be interfaces appearing to 37aa5eee35SRafael J. Wysockisoftware as individual single-core processors. In other words, a CPU is an 38aa5eee35SRafael J. Wysockientity which appears to be fetching instructions that belong to one sequence 39aa5eee35SRafael J. Wysocki(program) from memory and executing them, but it need not work this way 40aa5eee35SRafael J. Wysockiphysically. Generally, three different cases can be consider here. 41aa5eee35SRafael J. Wysocki 42aa5eee35SRafael J. WysockiFirst, if the whole processor can only follow one sequence of instructions (one 43aa5eee35SRafael J. Wysockiprogram) at a time, it is a CPU. In that case, if the hardware is asked to 44aa5eee35SRafael J. Wysockienter an idle state, that applies to the processor as a whole. 45aa5eee35SRafael J. Wysocki 46aa5eee35SRafael J. WysockiSecond, if the processor is multi-core, each core in it is able to follow at 47aa5eee35SRafael J. Wysockileast one program at a time. The cores need not be entirely independent of each 48aa5eee35SRafael J. Wysockiother (for example, they may share caches), but still most of the time they 49aa5eee35SRafael J. Wysockiwork physically in parallel with each other, so if each of them executes only 50aa5eee35SRafael J. Wysockione program, those programs run mostly independently of each other at the same 51aa5eee35SRafael J. Wysockitime. The entire cores are CPUs in that case and if the hardware is asked to 52aa5eee35SRafael J. Wysockienter an idle state, that applies to the core that asked for it in the first 53aa5eee35SRafael J. Wysockiplace, but it also may apply to a larger unit (say a "package" or a "cluster") 54aa5eee35SRafael J. Wysockithat the core belongs to (in fact, it may apply to an entire hierarchy of larger 55aa5eee35SRafael J. Wysockiunits containing the core). Namely, if all of the cores in the larger unit 56aa5eee35SRafael J. Wysockiexcept for one have been put into idle states at the "core level" and the 57aa5eee35SRafael J. Wysockiremaining core asks the processor to enter an idle state, that may trigger it 58aa5eee35SRafael J. Wysockito put the whole larger unit into an idle state which also will affect the 59aa5eee35SRafael J. Wysockiother cores in that unit. 60aa5eee35SRafael J. Wysocki 61aa5eee35SRafael J. WysockiFinally, each core in a multi-core processor may be able to follow more than one 62aa5eee35SRafael J. Wysockiprogram in the same time frame (that is, each core may be able to fetch 63aa5eee35SRafael J. Wysockiinstructions from multiple locations in memory and execute them in the same time 64aa5eee35SRafael J. Wysockiframe, but not necessarily entirely in parallel with each other). In that case 65aa5eee35SRafael J. Wysockithe cores present themselves to software as "bundles" each consisting of 66aa5eee35SRafael J. Wysockimultiple individual single-core "processors", referred to as *hardware threads* 67aa5eee35SRafael J. Wysocki(or hyper-threads specifically on Intel hardware), that each can follow one 68aa5eee35SRafael J. Wysockisequence of instructions. Then, the hardware threads are CPUs from the CPU idle 69aa5eee35SRafael J. Wysockitime management perspective and if the processor is asked to enter an idle state 70aa5eee35SRafael J. Wysockiby one of them, the hardware thread (or CPU) that asked for it is stopped, but 71aa5eee35SRafael J. Wysockinothing more happens, unless all of the other hardware threads within the same 72aa5eee35SRafael J. Wysockicore also have asked the processor to enter an idle state. In that situation, 73aa5eee35SRafael J. Wysockithe core may be put into an idle state individually or a larger unit containing 74aa5eee35SRafael J. Wysockiit may be put into an idle state as a whole (if the other cores within the 75aa5eee35SRafael J. Wysockilarger unit are in idle states already). 76aa5eee35SRafael J. Wysocki 77aa5eee35SRafael J. WysockiIdle CPUs 78aa5eee35SRafael J. Wysocki--------- 79aa5eee35SRafael J. Wysocki 80aa5eee35SRafael J. WysockiLogical CPUs, simply referred to as "CPUs" in what follows, are regarded as 81aa5eee35SRafael J. Wysocki*idle* by the Linux kernel when there are no tasks to run on them except for the 82aa5eee35SRafael J. Wysockispecial "idle" task. 83aa5eee35SRafael J. Wysocki 84aa5eee35SRafael J. WysockiTasks are the CPU scheduler's representation of work. Each task consists of a 85aa5eee35SRafael J. Wysockisequence of instructions to execute, or code, data to be manipulated while 86aa5eee35SRafael J. Wysockirunning that code, and some context information that needs to be loaded into the 87aa5eee35SRafael J. Wysockiprocessor every time the task's code is run by a CPU. The CPU scheduler 88aa5eee35SRafael J. Wysockidistributes work by assigning tasks to run to the CPUs present in the system. 89aa5eee35SRafael J. Wysocki 90aa5eee35SRafael J. WysockiTasks can be in various states. In particular, they are *runnable* if there are 91aa5eee35SRafael J. Wysockino specific conditions preventing their code from being run by a CPU as long as 92aa5eee35SRafael J. Wysockithere is a CPU available for that (for example, they are not waiting for any 93aa5eee35SRafael J. Wysockievents to occur or similar). When a task becomes runnable, the CPU scheduler 94aa5eee35SRafael J. Wysockiassigns it to one of the available CPUs to run and if there are no more runnable 95aa5eee35SRafael J. Wysockitasks assigned to it, the CPU will load the given task's context and run its 96aa5eee35SRafael J. Wysockicode (from the instruction following the last one executed so far, possibly by 97aa5eee35SRafael J. Wysockianother CPU). [If there are multiple runnable tasks assigned to one CPU 98aa5eee35SRafael J. Wysockisimultaneously, they will be subject to prioritization and time sharing in order 99aa5eee35SRafael J. Wysockito allow them to make some progress over time.] 100aa5eee35SRafael J. Wysocki 101aa5eee35SRafael J. WysockiThe special "idle" task becomes runnable if there are no other runnable tasks 102aa5eee35SRafael J. Wysockiassigned to the given CPU and the CPU is then regarded as idle. In other words, 103aa5eee35SRafael J. Wysockiin Linux idle CPUs run the code of the "idle" task called *the idle loop*. That 104aa5eee35SRafael J. Wysockicode may cause the processor to be put into one of its idle states, if they are 105aa5eee35SRafael J. Wysockisupported, in order to save energy, but if the processor does not support any 106aa5eee35SRafael J. Wysockiidle states, or there is not enough time to spend in an idle state before the 107aa5eee35SRafael J. Wysockinext wakeup event, or there are strict latency constraints preventing any of the 108aa5eee35SRafael J. Wysockiavailable idle states from being used, the CPU will simply execute more or less 109aa5eee35SRafael J. Wysockiuseless instructions in a loop until it is assigned a new task to run. 110aa5eee35SRafael J. Wysocki 111aa5eee35SRafael J. Wysocki 112aa5eee35SRafael J. Wysocki.. _idle-loop: 113aa5eee35SRafael J. Wysocki 114aa5eee35SRafael J. WysockiThe Idle Loop 115aa5eee35SRafael J. Wysocki============= 116aa5eee35SRafael J. Wysocki 117aa5eee35SRafael J. WysockiThe idle loop code takes two major steps in every iteration of it. First, it 118aa5eee35SRafael J. Wysockicalls into a code module referred to as the *governor* that belongs to the CPU 119aa5eee35SRafael J. Wysockiidle time management subsystem called ``CPUIdle`` to select an idle state for 120aa5eee35SRafael J. Wysockithe CPU to ask the hardware to enter. Second, it invokes another code module 121aa5eee35SRafael J. Wysockifrom the ``CPUIdle`` subsystem, called the *driver*, to actually ask the 122aa5eee35SRafael J. Wysockiprocessor hardware to enter the idle state selected by the governor. 123aa5eee35SRafael J. Wysocki 124aa5eee35SRafael J. WysockiThe role of the governor is to find an idle state most suitable for the 125aa5eee35SRafael J. Wysockiconditions at hand. For this purpose, idle states that the hardware can be 126aa5eee35SRafael J. Wysockiasked to enter by logical CPUs are represented in an abstract way independent of 127aa5eee35SRafael J. Wysockithe platform or the processor architecture and organized in a one-dimensional 128aa5eee35SRafael J. Wysocki(linear) array. That array has to be prepared and supplied by the ``CPUIdle`` 129aa5eee35SRafael J. Wysockidriver matching the platform the kernel is running on at the initialization 130aa5eee35SRafael J. Wysockitime. This allows ``CPUIdle`` governors to be independent of the underlying 131aa5eee35SRafael J. Wysockihardware and to work with any platforms that the Linux kernel can run on. 132aa5eee35SRafael J. Wysocki 133aa5eee35SRafael J. WysockiEach idle state present in that array is characterized by two parameters to be 134aa5eee35SRafael J. Wysockitaken into account by the governor, the *target residency* and the (worst-case) 135aa5eee35SRafael J. Wysocki*exit latency*. The target residency is the minimum time the hardware must 136aa5eee35SRafael J. Wysockispend in the given state, including the time needed to enter it (which may be 137aa5eee35SRafael J. Wysockisubstantial), in order to save more energy than it would save by entering one of 138aa5eee35SRafael J. Wysockithe shallower idle states instead. [The "depth" of an idle state roughly 139aa5eee35SRafael J. Wysockicorresponds to the power drawn by the processor in that state.] The exit 140aa5eee35SRafael J. Wysockilatency, in turn, is the maximum time it will take a CPU asking the processor 141aa5eee35SRafael J. Wysockihardware to enter an idle state to start executing the first instruction after a 142aa5eee35SRafael J. Wysockiwakeup from that state. Note that in general the exit latency also must cover 143aa5eee35SRafael J. Wysockithe time needed to enter the given state in case the wakeup occurs when the 144aa5eee35SRafael J. Wysockihardware is entering it and it must be entered completely to be exited in an 145aa5eee35SRafael J. Wysockiordered manner. 146aa5eee35SRafael J. Wysocki 147aa5eee35SRafael J. WysockiThere are two types of information that can influence the governor's decisions. 148aa5eee35SRafael J. WysockiFirst of all, the governor knows the time until the closest timer event. That 149aa5eee35SRafael J. Wysockitime is known exactly, because the kernel programs timers and it knows exactly 150aa5eee35SRafael J. Wysockiwhen they will trigger, and it is the maximum time the hardware that the given 151aa5eee35SRafael J. WysockiCPU depends on can spend in an idle state, including the time necessary to enter 152aa5eee35SRafael J. Wysockiand exit it. However, the CPU may be woken up by a non-timer event at any time 153aa5eee35SRafael J. Wysocki(in particular, before the closest timer triggers) and it generally is not known 154aa5eee35SRafael J. Wysockiwhen that may happen. The governor can only see how much time the CPU actually 155aa5eee35SRafael J. Wysockiwas idle after it has been woken up (that time will be referred to as the *idle 156aa5eee35SRafael J. Wysockiduration* from now on) and it can use that information somehow along with the 157aa5eee35SRafael J. Wysockitime until the closest timer to estimate the idle duration in future. How the 158aa5eee35SRafael J. Wysockigovernor uses that information depends on what algorithm is implemented by it 159aa5eee35SRafael J. Wysockiand that is the primary reason for having more than one governor in the 160aa5eee35SRafael J. Wysocki``CPUIdle`` subsystem. 161aa5eee35SRafael J. Wysocki 1627395683aSHanjun GuoThere are four ``CPUIdle`` governors available, ``menu``, `TEO <teo-gov_>`_, 1637395683aSHanjun Guo``ladder`` and ``haltpoll``. Which of them is used by default depends on the 1647395683aSHanjun Guoconfiguration of the kernel and in particular on whether or not the scheduler 1657395683aSHanjun Guotick can be `stopped by the idle loop <idle-cpus-and-tick_>`_. Available 1667395683aSHanjun Guogovernors can be read from the :file:`available_governors`, and the governor 1677395683aSHanjun Guocan be changed at runtime. The name of the ``CPUIdle`` governor currently 1687395683aSHanjun Guoused by the kernel can be read from the :file:`current_governor_ro` or 1697395683aSHanjun Guo:file:`current_governor` file under :file:`/sys/devices/system/cpu/cpuidle/` 1707395683aSHanjun Guoin ``sysfs``. 171aa5eee35SRafael J. Wysocki 172aa5eee35SRafael J. WysockiWhich ``CPUIdle`` driver is used, on the other hand, usually depends on the 173aa5eee35SRafael J. Wysockiplatform the kernel is running on, but there are platforms with more than one 174aa5eee35SRafael J. Wysockimatching driver. For example, there are two drivers that can work with the 175aa5eee35SRafael J. Wysockimajority of Intel platforms, ``intel_idle`` and ``acpi_idle``, one with 176aa5eee35SRafael J. Wysockihardcoded idle states information and the other able to read that information 177aa5eee35SRafael J. Wysockifrom the system's ACPI tables, respectively. Still, even in those cases, the 178aa5eee35SRafael J. Wysockidriver chosen at the system initialization time cannot be replaced later, so the 179aa5eee35SRafael J. Wysockidecision on which one of them to use has to be made early (on Intel platforms 180aa5eee35SRafael J. Wysockithe ``acpi_idle`` driver will be used if ``intel_idle`` is disabled for some 181aa5eee35SRafael J. Wysockireason or if it does not recognize the processor). The name of the ``CPUIdle`` 182aa5eee35SRafael J. Wysockidriver currently used by the kernel can be read from the :file:`current_driver` 183aa5eee35SRafael J. Wysockifile under :file:`/sys/devices/system/cpu/cpuidle/` in ``sysfs``. 184aa5eee35SRafael J. Wysocki 185aa5eee35SRafael J. Wysocki 186aa5eee35SRafael J. Wysocki.. _idle-cpus-and-tick: 187aa5eee35SRafael J. Wysocki 188aa5eee35SRafael J. WysockiIdle CPUs and The Scheduler Tick 189aa5eee35SRafael J. Wysocki================================ 190aa5eee35SRafael J. Wysocki 191aa5eee35SRafael J. WysockiThe scheduler tick is a timer that triggers periodically in order to implement 192aa5eee35SRafael J. Wysockithe time sharing strategy of the CPU scheduler. Of course, if there are 193aa5eee35SRafael J. Wysockimultiple runnable tasks assigned to one CPU at the same time, the only way to 194aa5eee35SRafael J. Wysockiallow them to make reasonable progress in a given time frame is to make them 195aa5eee35SRafael J. Wysockishare the available CPU time. Namely, in rough approximation, each task is 196aa5eee35SRafael J. Wysockigiven a slice of the CPU time to run its code, subject to the scheduling class, 197aa5eee35SRafael J. Wysockiprioritization and so on and when that time slice is used up, the CPU should be 198aa5eee35SRafael J. Wysockiswitched over to running (the code of) another task. The currently running task 199aa5eee35SRafael J. Wysockimay not want to give the CPU away voluntarily, however, and the scheduler tick 200aa5eee35SRafael J. Wysockiis there to make the switch happen regardless. That is not the only role of the 201aa5eee35SRafael J. Wysockitick, but it is the primary reason for using it. 202aa5eee35SRafael J. Wysocki 203aa5eee35SRafael J. WysockiThe scheduler tick is problematic from the CPU idle time management perspective, 204aa5eee35SRafael J. Wysockibecause it triggers periodically and relatively often (depending on the kernel 205aa5eee35SRafael J. Wysockiconfiguration, the length of the tick period is between 1 ms and 10 ms). 206aa5eee35SRafael J. WysockiThus, if the tick is allowed to trigger on idle CPUs, it will not make sense 207aa5eee35SRafael J. Wysockifor them to ask the hardware to enter idle states with target residencies above 208aa5eee35SRafael J. Wysockithe tick period length. Moreover, in that case the idle duration of any CPU 209aa5eee35SRafael J. Wysockiwill never exceed the tick period length and the energy used for entering and 210aa5eee35SRafael J. Wysockiexiting idle states due to the tick wakeups on idle CPUs will be wasted. 211aa5eee35SRafael J. Wysocki 212aa5eee35SRafael J. WysockiFortunately, it is not really necessary to allow the tick to trigger on idle 213aa5eee35SRafael J. WysockiCPUs, because (by definition) they have no tasks to run except for the special 214aa5eee35SRafael J. Wysocki"idle" one. In other words, from the CPU scheduler perspective, the only user 215aa5eee35SRafael J. Wysockiof the CPU time on them is the idle loop. Since the time of an idle CPU need 216aa5eee35SRafael J. Wysockinot be shared between multiple runnable tasks, the primary reason for using the 217aa5eee35SRafael J. Wysockitick goes away if the given CPU is idle. Consequently, it is possible to stop 218aa5eee35SRafael J. Wysockithe scheduler tick entirely on idle CPUs in principle, even though that may not 219aa5eee35SRafael J. Wysockialways be worth the effort. 220aa5eee35SRafael J. Wysocki 221aa5eee35SRafael J. WysockiWhether or not it makes sense to stop the scheduler tick in the idle loop 222aa5eee35SRafael J. Wysockidepends on what is expected by the governor. First, if there is another 223aa5eee35SRafael J. Wysocki(non-tick) timer due to trigger within the tick range, stopping the tick clearly 224aa5eee35SRafael J. Wysockiwould be a waste of time, even though the timer hardware may not need to be 225aa5eee35SRafael J. Wysockireprogrammed in that case. Second, if the governor is expecting a non-timer 226aa5eee35SRafael J. Wysockiwakeup within the tick range, stopping the tick is not necessary and it may even 227aa5eee35SRafael J. Wysockibe harmful. Namely, in that case the governor will select an idle state with 228aa5eee35SRafael J. Wysockithe target residency within the time until the expected wakeup, so that state is 229aa5eee35SRafael J. Wysockigoing to be relatively shallow. The governor really cannot select a deep idle 230aa5eee35SRafael J. Wysockistate then, as that would contradict its own expectation of a wakeup in short 231aa5eee35SRafael J. Wysockiorder. Now, if the wakeup really occurs shortly, stopping the tick would be a 232aa5eee35SRafael J. Wysockiwaste of time and in this case the timer hardware would need to be reprogrammed, 233aa5eee35SRafael J. Wysockiwhich is expensive. On the other hand, if the tick is stopped and the wakeup 234aa5eee35SRafael J. Wysockidoes not occur any time soon, the hardware may spend indefinite amount of time 235aa5eee35SRafael J. Wysockiin the shallow idle state selected by the governor, which will be a waste of 236aa5eee35SRafael J. Wysockienergy. Hence, if the governor is expecting a wakeup of any kind within the 237aa5eee35SRafael J. Wysockitick range, it is better to allow the tick trigger. Otherwise, however, the 238aa5eee35SRafael J. Wysockigovernor will select a relatively deep idle state, so the tick should be stopped 239aa5eee35SRafael J. Wysockiso that it does not wake up the CPU too early. 240aa5eee35SRafael J. Wysocki 241aa5eee35SRafael J. WysockiIn any case, the governor knows what it is expecting and the decision on whether 242aa5eee35SRafael J. Wysockior not to stop the scheduler tick belongs to it. Still, if the tick has been 243aa5eee35SRafael J. Wysockistopped already (in one of the previous iterations of the loop), it is better 244aa5eee35SRafael J. Wysockito leave it as is and the governor needs to take that into account. 245aa5eee35SRafael J. Wysocki 246aa5eee35SRafael J. WysockiThe kernel can be configured to disable stopping the scheduler tick in the idle 247aa5eee35SRafael J. Wysockiloop altogether. That can be done through the build-time configuration of it 248aa5eee35SRafael J. Wysocki(by unsetting the ``CONFIG_NO_HZ_IDLE`` configuration option) or by passing 249aa5eee35SRafael J. Wysocki``nohz=off`` to it in the command line. In both cases, as the stopping of the 250aa5eee35SRafael J. Wysockischeduler tick is disabled, the governor's decisions regarding it are simply 251aa5eee35SRafael J. Wysockiignored by the idle loop code and the tick is never stopped. 252aa5eee35SRafael J. Wysocki 253aa5eee35SRafael J. WysockiThe systems that run kernels configured to allow the scheduler tick to be 254aa5eee35SRafael J. Wysockistopped on idle CPUs are referred to as *tickless* systems and they are 255aa5eee35SRafael J. Wysockigenerally regarded as more energy-efficient than the systems running kernels in 256aa5eee35SRafael J. Wysockiwhich the tick cannot be stopped. If the given system is tickless, it will use 257aa5eee35SRafael J. Wysockithe ``menu`` governor by default and if it is not tickless, the default 258aa5eee35SRafael J. Wysocki``CPUIdle`` governor on it will be ``ladder``. 259aa5eee35SRafael J. Wysocki 260aa5eee35SRafael J. Wysocki 261b26bf6abSRafael J. Wysocki.. _menu-gov: 262b26bf6abSRafael J. Wysocki 263aa5eee35SRafael J. WysockiThe ``menu`` Governor 264aa5eee35SRafael J. Wysocki===================== 265aa5eee35SRafael J. Wysocki 266aa5eee35SRafael J. WysockiThe ``menu`` governor is the default ``CPUIdle`` governor for tickless systems. 267aa5eee35SRafael J. WysockiIt is quite complex, but the basic principle of its design is straightforward. 268aa5eee35SRafael J. WysockiNamely, when invoked to select an idle state for a CPU (i.e. an idle state that 269aa5eee35SRafael J. Wysockithe CPU will ask the processor hardware to enter), it attempts to predict the 270aa5eee35SRafael J. Wysockiidle duration and uses the predicted value for idle state selection. 271aa5eee35SRafael J. Wysocki 272aa5eee35SRafael J. WysockiIt first obtains the time until the closest timer event with the assumption 273aa5eee35SRafael J. Wysockithat the scheduler tick will be stopped. That time, referred to as the *sleep 274aa5eee35SRafael J. Wysockilength* in what follows, is the upper bound on the time before the next CPU 275aa5eee35SRafael J. Wysockiwakeup. It is used to determine the sleep length range, which in turn is needed 276aa5eee35SRafael J. Wysockito get the sleep length correction factor. 277aa5eee35SRafael J. Wysocki 278aa5eee35SRafael J. WysockiThe ``menu`` governor maintains two arrays of sleep length correction factors. 279aa5eee35SRafael J. WysockiOne of them is used when tasks previously running on the given CPU are waiting 280aa5eee35SRafael J. Wysockifor some I/O operations to complete and the other one is used when that is not 281aa5eee35SRafael J. Wysockithe case. Each array contains several correction factor values that correspond 282aa5eee35SRafael J. Wysockito different sleep length ranges organized so that each range represented in the 283aa5eee35SRafael J. Wysockiarray is approximately 10 times wider than the previous one. 284aa5eee35SRafael J. Wysocki 285aa5eee35SRafael J. WysockiThe correction factor for the given sleep length range (determined before 286aa5eee35SRafael J. Wysockiselecting the idle state for the CPU) is updated after the CPU has been woken 287aa5eee35SRafael J. Wysockiup and the closer the sleep length is to the observed idle duration, the closer 288aa5eee35SRafael J. Wysockito 1 the correction factor becomes (it must fall between 0 and 1 inclusive). 289aa5eee35SRafael J. WysockiThe sleep length is multiplied by the correction factor for the range that it 290aa5eee35SRafael J. Wysockifalls into to obtain the first approximation of the predicted idle duration. 291aa5eee35SRafael J. Wysocki 292aa5eee35SRafael J. WysockiNext, the governor uses a simple pattern recognition algorithm to refine its 293aa5eee35SRafael J. Wysockiidle duration prediction. Namely, it saves the last 8 observed idle duration 294aa5eee35SRafael J. Wysockivalues and, when predicting the idle duration next time, it computes the average 295aa5eee35SRafael J. Wysockiand variance of them. If the variance is small (smaller than 400 square 296aa5eee35SRafael J. Wysockimilliseconds) or it is small relative to the average (the average is greater 297aa5eee35SRafael J. Wysockithat 6 times the standard deviation), the average is regarded as the "typical 298aa5eee35SRafael J. Wysockiinterval" value. Otherwise, the longest of the saved observed idle duration 299aa5eee35SRafael J. Wysockivalues is discarded and the computation is repeated for the remaining ones. 300aa5eee35SRafael J. WysockiAgain, if the variance of them is small (in the above sense), the average is 301aa5eee35SRafael J. Wysockitaken as the "typical interval" value and so on, until either the "typical 302aa5eee35SRafael J. Wysockiinterval" is determined or too many data points are disregarded, in which case 303aa5eee35SRafael J. Wysockithe "typical interval" is assumed to equal "infinity" (the maximum unsigned 304aa5eee35SRafael J. Wysockiinteger value). The "typical interval" computed this way is compared with the 305aa5eee35SRafael J. Wysockisleep length multiplied by the correction factor and the minimum of the two is 306aa5eee35SRafael J. Wysockitaken as the predicted idle duration. 307aa5eee35SRafael J. Wysocki 308aa5eee35SRafael J. WysockiThen, the governor computes an extra latency limit to help "interactive" 309aa5eee35SRafael J. Wysockiworkloads. It uses the observation that if the exit latency of the selected 310aa5eee35SRafael J. Wysockiidle state is comparable with the predicted idle duration, the total time spent 311aa5eee35SRafael J. Wysockiin that state probably will be very short and the amount of energy to save by 312aa5eee35SRafael J. Wysockientering it will be relatively small, so likely it is better to avoid the 313aa5eee35SRafael J. Wysockioverhead related to entering that state and exiting it. Thus selecting a 314aa5eee35SRafael J. Wysockishallower state is likely to be a better option then. The first approximation 315aa5eee35SRafael J. Wysockiof the extra latency limit is the predicted idle duration itself which 316aa5eee35SRafael J. Wysockiadditionally is divided by a value depending on the number of tasks that 317aa5eee35SRafael J. Wysockipreviously ran on the given CPU and now they are waiting for I/O operations to 318aa5eee35SRafael J. Wysockicomplete. The result of that division is compared with the latency limit coming 319aa5eee35SRafael J. Wysockifrom the power management quality of service, or `PM QoS <cpu-pm-qos_>`_, 320aa5eee35SRafael J. Wysockiframework and the minimum of the two is taken as the limit for the idle states' 321aa5eee35SRafael J. Wysockiexit latency. 322aa5eee35SRafael J. Wysocki 323aa5eee35SRafael J. WysockiNow, the governor is ready to walk the list of idle states and choose one of 324aa5eee35SRafael J. Wysockithem. For this purpose, it compares the target residency of each state with 325aa5eee35SRafael J. Wysockithe predicted idle duration and the exit latency of it with the computed latency 326aa5eee35SRafael J. Wysockilimit. It selects the state with the target residency closest to the predicted 327aa5eee35SRafael J. Wysockiidle duration, but still below it, and exit latency that does not exceed the 328aa5eee35SRafael J. Wysockilimit. 329aa5eee35SRafael J. Wysocki 330aa5eee35SRafael J. WysockiIn the final step the governor may still need to refine the idle state selection 331aa5eee35SRafael J. Wysockiif it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That 332aa5eee35SRafael J. Wysockihappens if the idle duration predicted by it is less than the tick period and 333aa5eee35SRafael J. Wysockithe tick has not been stopped already (in a previous iteration of the idle 334aa5eee35SRafael J. Wysockiloop). Then, the sleep length used in the previous computations may not reflect 335aa5eee35SRafael J. Wysockithe real time until the closest timer event and if it really is greater than 336aa5eee35SRafael J. Wysockithat time, the governor may need to select a shallower state with a suitable 337aa5eee35SRafael J. Wysockitarget residency. 338aa5eee35SRafael J. Wysocki 339aa5eee35SRafael J. Wysocki 340b26bf6abSRafael J. Wysocki.. _teo-gov: 341b26bf6abSRafael J. Wysocki 342b26bf6abSRafael J. WysockiThe Timer Events Oriented (TEO) Governor 343b26bf6abSRafael J. Wysocki======================================== 344b26bf6abSRafael J. Wysocki 345b26bf6abSRafael J. WysockiThe timer events oriented (TEO) governor is an alternative ``CPUIdle`` governor 346b26bf6abSRafael J. Wysockifor tickless systems. It follows the same basic strategy as the ``menu`` `one 347b26bf6abSRafael J. Wysocki<menu-gov_>`_: it always tries to find the deepest idle state suitable for the 348b26bf6abSRafael J. Wysockigiven conditions. However, it applies a different approach to that problem. 349b26bf6abSRafael J. Wysocki 350*154ae8bbSRafael J. Wysocki.. kernel-doc:: drivers/cpuidle/governors/teo.c 351*154ae8bbSRafael J. Wysocki :doc: teo-description 352b26bf6abSRafael J. Wysocki 353aa5eee35SRafael J. Wysocki.. _idle-states-representation: 354aa5eee35SRafael J. Wysocki 355aa5eee35SRafael J. WysockiRepresentation of Idle States 356aa5eee35SRafael J. Wysocki============================= 357aa5eee35SRafael J. Wysocki 358aa5eee35SRafael J. WysockiFor the CPU idle time management purposes all of the physical idle states 359aa5eee35SRafael J. Wysockisupported by the processor have to be represented as a one-dimensional array of 360aa5eee35SRafael J. Wysocki|struct cpuidle_state| objects each allowing an individual (logical) CPU to ask 361aa5eee35SRafael J. Wysockithe processor hardware to enter an idle state of certain properties. If there 362aa5eee35SRafael J. Wysockiis a hierarchy of units in the processor, one |struct cpuidle_state| object can 363aa5eee35SRafael J. Wysockicover a combination of idle states supported by the units at different levels of 364aa5eee35SRafael J. Wysockithe hierarchy. In that case, the `target residency and exit latency parameters 365aa5eee35SRafael J. Wysockiof it <idle-loop_>`_, must reflect the properties of the idle state at the 366aa5eee35SRafael J. Wysockideepest level (i.e. the idle state of the unit containing all of the other 367aa5eee35SRafael J. Wysockiunits). 368aa5eee35SRafael J. Wysocki 369aa5eee35SRafael J. WysockiFor example, take a processor with two cores in a larger unit referred to as 370aa5eee35SRafael J. Wysockia "module" and suppose that asking the hardware to enter a specific idle state 371aa5eee35SRafael J. Wysocki(say "X") at the "core" level by one core will trigger the module to try to 372aa5eee35SRafael J. Wysockienter a specific idle state of its own (say "MX") if the other core is in idle 373aa5eee35SRafael J. Wysockistate "X" already. In other words, asking for idle state "X" at the "core" 374aa5eee35SRafael J. Wysockilevel gives the hardware a license to go as deep as to idle state "MX" at the 375aa5eee35SRafael J. Wysocki"module" level, but there is no guarantee that this is going to happen (the core 376aa5eee35SRafael J. Wysockiasking for idle state "X" may just end up in that state by itself instead). 377aa5eee35SRafael J. WysockiThen, the target residency of the |struct cpuidle_state| object representing 378aa5eee35SRafael J. Wysockiidle state "X" must reflect the minimum time to spend in idle state "MX" of 379aa5eee35SRafael J. Wysockithe module (including the time needed to enter it), because that is the minimum 380aa5eee35SRafael J. Wysockitime the CPU needs to be idle to save any energy in case the hardware enters 381aa5eee35SRafael J. Wysockithat state. Analogously, the exit latency parameter of that object must cover 382aa5eee35SRafael J. Wysockithe exit time of idle state "MX" of the module (and usually its entry time too), 383aa5eee35SRafael J. Wysockibecause that is the maximum delay between a wakeup signal and the time the CPU 384aa5eee35SRafael J. Wysockiwill start to execute the first new instruction (assuming that both cores in the 385aa5eee35SRafael J. Wysockimodule will always be ready to execute instructions as soon as the module 386aa5eee35SRafael J. Wysockibecomes operational as a whole). 387aa5eee35SRafael J. Wysocki 388aa5eee35SRafael J. WysockiThere are processors without direct coordination between different levels of the 389aa5eee35SRafael J. Wysockihierarchy of units inside them, however. In those cases asking for an idle 390aa5eee35SRafael J. Wysockistate at the "core" level does not automatically affect the "module" level, for 391aa5eee35SRafael J. Wysockiexample, in any way and the ``CPUIdle`` driver is responsible for the entire 392aa5eee35SRafael J. Wysockihandling of the hierarchy. Then, the definition of the idle state objects is 393aa5eee35SRafael J. Wysockientirely up to the driver, but still the physical properties of the idle state 394aa5eee35SRafael J. Wysockithat the processor hardware finally goes into must always follow the parameters 395aa5eee35SRafael J. Wysockiused by the governor for idle state selection (for instance, the actual exit 396aa5eee35SRafael J. Wysockilatency of that idle state must not exceed the exit latency parameter of the 397aa5eee35SRafael J. Wysockiidle state object selected by the governor). 398aa5eee35SRafael J. Wysocki 399aa5eee35SRafael J. WysockiIn addition to the target residency and exit latency idle state parameters 400aa5eee35SRafael J. Wysockidiscussed above, the objects representing idle states each contain a few other 401aa5eee35SRafael J. Wysockiparameters describing the idle state and a pointer to the function to run in 402aa5eee35SRafael J. Wysockiorder to ask the hardware to enter that state. Also, for each 403aa5eee35SRafael J. Wysocki|struct cpuidle_state| object, there is a corresponding 404aa5eee35SRafael J. Wysocki:c:type:`struct cpuidle_state_usage <cpuidle_state_usage>` one containing usage 405aa5eee35SRafael J. Wysockistatistics of the given idle state. That information is exposed by the kernel 406aa5eee35SRafael J. Wysockivia ``sysfs``. 407aa5eee35SRafael J. Wysocki 40823d18dcfSJulia LawallFor each CPU in the system, there is a :file:`/sys/devices/system/cpu/cpu<N>/cpuidle/` 409aa5eee35SRafael J. Wysockidirectory in ``sysfs``, where the number ``<N>`` is assigned to the given 410aa5eee35SRafael J. WysockiCPU at the initialization time. That directory contains a set of subdirectories 411aa5eee35SRafael J. Wysockicalled :file:`state0`, :file:`state1` and so on, up to the number of idle state 412aa5eee35SRafael J. Wysockiobjects defined for the given CPU minus one. Each of these directories 413aa5eee35SRafael J. Wysockicorresponds to one idle state object and the larger the number in its name, the 414aa5eee35SRafael J. Wysockideeper the (effective) idle state represented by it. Each of them contains 415aa5eee35SRafael J. Wysockia number of files (attributes) representing the properties of the idle state 416aa5eee35SRafael J. Wysockiobject corresponding to it, as follows: 417aa5eee35SRafael J. Wysocki 41804dab58aSRafael J. Wysocki``above`` 41904dab58aSRafael J. Wysocki Total number of times this idle state had been asked for, but the 42004dab58aSRafael J. Wysocki observed idle duration was certainly too short to match its target 42104dab58aSRafael J. Wysocki residency. 42204dab58aSRafael J. Wysocki 42304dab58aSRafael J. Wysocki``below`` 4246841ca15SJulia Lawall Total number of times this idle state had been asked for, but certainly 42504dab58aSRafael J. Wysocki a deeper idle state would have been a better match for the observed idle 42604dab58aSRafael J. Wysocki duration. 42704dab58aSRafael J. Wysocki 428aa5eee35SRafael J. Wysocki``desc`` 429aa5eee35SRafael J. Wysocki Description of the idle state. 430aa5eee35SRafael J. Wysocki 431aa5eee35SRafael J. Wysocki``disable`` 432aa5eee35SRafael J. Wysocki Whether or not this idle state is disabled. 433aa5eee35SRafael J. Wysocki 43475a80267SRafael J. Wysocki``default_status`` 43575a80267SRafael J. Wysocki The default status of this state, "enabled" or "disabled". 43675a80267SRafael J. Wysocki 437aa5eee35SRafael J. Wysocki``latency`` 438aa5eee35SRafael J. Wysocki Exit latency of the idle state in microseconds. 439aa5eee35SRafael J. Wysocki 440aa5eee35SRafael J. Wysocki``name`` 441aa5eee35SRafael J. Wysocki Name of the idle state. 442aa5eee35SRafael J. Wysocki 443aa5eee35SRafael J. Wysocki``power`` 444aa5eee35SRafael J. Wysocki Power drawn by hardware in this idle state in milliwatts (if specified, 445aa5eee35SRafael J. Wysocki 0 otherwise). 446aa5eee35SRafael J. Wysocki 447aa5eee35SRafael J. Wysocki``residency`` 448aa5eee35SRafael J. Wysocki Target residency of the idle state in microseconds. 449aa5eee35SRafael J. Wysocki 450aa5eee35SRafael J. Wysocki``time`` 451aa5eee35SRafael J. Wysocki Total time spent in this idle state by the given CPU (as measured by the 452aa5eee35SRafael J. Wysocki kernel) in microseconds. 453aa5eee35SRafael J. Wysocki 454aa5eee35SRafael J. Wysocki``usage`` 455aa5eee35SRafael J. Wysocki Total number of times the hardware has been asked by the given CPU to 456aa5eee35SRafael J. Wysocki enter this idle state. 457aa5eee35SRafael J. Wysocki 458f49735f4SLina Iyer``rejected`` 459f49735f4SLina Iyer Total number of times a request to enter this idle state on the given 460f49735f4SLina Iyer CPU was rejected. 461f49735f4SLina Iyer 462aa5eee35SRafael J. WysockiThe :file:`desc` and :file:`name` files both contain strings. The difference 463aa5eee35SRafael J. Wysockibetween them is that the name is expected to be more concise, while the 464aa5eee35SRafael J. Wysockidescription may be longer and it may contain white space or special characters. 465aa5eee35SRafael J. WysockiThe other files listed above contain integer numbers. 466aa5eee35SRafael J. Wysocki 467aa5eee35SRafael J. WysockiThe :file:`disable` attribute is the only writeable one. If it contains 1, the 468aa5eee35SRafael J. Wysockigiven idle state is disabled for this particular CPU, which means that the 469aa5eee35SRafael J. Wysockigovernor will never select it for this particular CPU and the ``CPUIdle`` 470aa5eee35SRafael J. Wysockidriver will never ask the hardware to enter it for that CPU as a result. 471aa5eee35SRafael J. WysockiHowever, disabling an idle state for one CPU does not prevent it from being 472aa5eee35SRafael J. Wysockiasked for by the other CPUs, so it must be disabled for all of them in order to 473aa5eee35SRafael J. Wysockinever be asked for by any of them. [Note that, due to the way the ``ladder`` 474aa5eee35SRafael J. Wysockigovernor is implemented, disabling an idle state prevents that governor from 475aa5eee35SRafael J. Wysockiselecting any idle states deeper than the disabled one too.] 476aa5eee35SRafael J. Wysocki 477aa5eee35SRafael J. WysockiIf the :file:`disable` attribute contains 0, the given idle state is enabled for 478aa5eee35SRafael J. Wysockithis particular CPU, but it still may be disabled for some or all of the other 479aa5eee35SRafael J. WysockiCPUs in the system at the same time. Writing 1 to it causes the idle state to 480aa5eee35SRafael J. Wysockibe disabled for this particular CPU and writing 0 to it allows the governor to 481aa5eee35SRafael J. Wysockitake it into consideration for the given CPU and the driver to ask for it, 482aa5eee35SRafael J. Wysockiunless that state was disabled globally in the driver (in which case it cannot 483aa5eee35SRafael J. Wysockibe used at all). 484aa5eee35SRafael J. Wysocki 485aa5eee35SRafael J. WysockiThe :file:`power` attribute is not defined very well, especially for idle state 486aa5eee35SRafael J. Wysockiobjects representing combinations of idle states at different levels of the 487aa5eee35SRafael J. Wysockihierarchy of units in the processor, and it generally is hard to obtain idle 488aa5eee35SRafael J. Wysockistate power numbers for complex hardware, so :file:`power` often contains 0 (not 489aa5eee35SRafael J. Wysockiavailable) and if it contains a nonzero number, that number may not be very 490aa5eee35SRafael J. Wysockiaccurate and it should not be relied on for anything meaningful. 491aa5eee35SRafael J. Wysocki 492aa5eee35SRafael J. WysockiThe number in the :file:`time` file generally may be greater than the total time 493aa5eee35SRafael J. Wysockireally spent by the given CPU in the given idle state, because it is measured by 494aa5eee35SRafael J. Wysockithe kernel and it may not cover the cases in which the hardware refused to enter 495aa5eee35SRafael J. Wysockithis idle state and entered a shallower one instead of it (or even it did not 496aa5eee35SRafael J. Wysockienter any idle state at all). The kernel can only measure the time span between 497aa5eee35SRafael J. Wysockiasking the hardware to enter an idle state and the subsequent wakeup of the CPU 498aa5eee35SRafael J. Wysockiand it cannot say what really happened in the meantime at the hardware level. 499aa5eee35SRafael J. WysockiMoreover, if the idle state object in question represents a combination of idle 500aa5eee35SRafael J. Wysockistates at different levels of the hierarchy of units in the processor, 501aa5eee35SRafael J. Wysockithe kernel can never say how deep the hardware went down the hierarchy in any 502aa5eee35SRafael J. Wysockiparticular case. For these reasons, the only reliable way to find out how 503aa5eee35SRafael J. Wysockimuch time has been spent by the hardware in different idle states supported by 504aa5eee35SRafael J. Wysockiit is to use idle state residency counters in the hardware, if available. 505aa5eee35SRafael J. Wysocki 506f49735f4SLina IyerGenerally, an interrupt received when trying to enter an idle state causes the 507f49735f4SLina Iyeridle state entry request to be rejected, in which case the ``CPUIdle`` driver 508f49735f4SLina Iyermay return an error code to indicate that this was the case. The :file:`usage` 509f49735f4SLina Iyerand :file:`rejected` files report the number of times the given idle state 510f49735f4SLina Iyerwas entered successfully or rejected, respectively. 511aa5eee35SRafael J. Wysocki 512aa5eee35SRafael J. Wysocki.. _cpu-pm-qos: 513aa5eee35SRafael J. Wysocki 514aa5eee35SRafael J. WysockiPower Management Quality of Service for CPUs 515aa5eee35SRafael J. Wysocki============================================ 516aa5eee35SRafael J. Wysocki 517aa5eee35SRafael J. WysockiThe power management quality of service (PM QoS) framework in the Linux kernel 518aa5eee35SRafael J. Wysockiallows kernel code and user space processes to set constraints on various 519aa5eee35SRafael J. Wysockienergy-efficiency features of the kernel to prevent performance from dropping 520b8e6e27cSRafael J. Wysockibelow a required level. 521aa5eee35SRafael J. Wysocki 522aa5eee35SRafael J. WysockiCPU idle time management can be affected by PM QoS in two ways, through the 523b8e6e27cSRafael J. Wysockiglobal CPU latency limit and through the resume latency constraints for 524b8e6e27cSRafael J. Wysockiindividual CPUs. Kernel code (e.g. device drivers) can set both of them with 525b8e6e27cSRafael J. Wysockithe help of special internal interfaces provided by the PM QoS framework. User 526b8e6e27cSRafael J. Wysockispace can modify the former by opening the :file:`cpu_dma_latency` special 527b8e6e27cSRafael J. Wysockidevice file under :file:`/dev/` and writing a binary value (interpreted as a 528b8e6e27cSRafael J. Wysockisigned 32-bit integer) to it. In turn, the resume latency constraint for a CPU 529b8e6e27cSRafael J. Wysockican be modified from user space by writing a string (representing a signed 530b8e6e27cSRafael J. Wysocki32-bit integer) to the :file:`power/pm_qos_resume_latency_us` file under 531aa5eee35SRafael J. Wysocki:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs``, where the CPU number 532aa5eee35SRafael J. Wysocki``<N>`` is allocated at the system initialization time. Negative values 533aa5eee35SRafael J. Wysockiwill be rejected in both cases and, also in both cases, the written integer 534aa5eee35SRafael J. Wysockinumber will be interpreted as a requested PM QoS constraint in microseconds. 535aa5eee35SRafael J. Wysocki 536aa5eee35SRafael J. WysockiThe requested value is not automatically applied as a new constraint, however, 537aa5eee35SRafael J. Wysockias it may be less restrictive (greater in this particular case) than another 538aa5eee35SRafael J. Wysockiconstraint previously requested by someone else. For this reason, the PM QoS 539b8e6e27cSRafael J. Wysockiframework maintains a list of requests that have been made so far for the 540b8e6e27cSRafael J. Wysockiglobal CPU latency limit and for each individual CPU, aggregates them and 541b8e6e27cSRafael J. Wysockiapplies the effective (minimum in this particular case) value as the new 542b8e6e27cSRafael J. Wysockiconstraint. 543aa5eee35SRafael J. Wysocki 544aa5eee35SRafael J. WysockiIn fact, opening the :file:`cpu_dma_latency` special device file causes a new 545b8e6e27cSRafael J. WysockiPM QoS request to be created and added to a global priority list of CPU latency 546b8e6e27cSRafael J. Wysockilimit requests and the file descriptor coming from the "open" operation 547b8e6e27cSRafael J. Wysockirepresents that request. If that file descriptor is then used for writing, the 548b8e6e27cSRafael J. Wysockinumber written to it will be associated with the PM QoS request represented by 549b8e6e27cSRafael J. Wysockiit as a new requested limit value. Next, the priority list mechanism will be 550b8e6e27cSRafael J. Wysockiused to determine the new effective value of the entire list of requests and 551b8e6e27cSRafael J. Wysockithat effective value will be set as a new CPU latency limit. Thus requesting a 552b8e6e27cSRafael J. Wysockinew limit value will only change the real limit if the effective "list" value is 553b8e6e27cSRafael J. Wysockiaffected by it, which is the case if it is the minimum of the requested values 554b8e6e27cSRafael J. Wysockiin the list. 555b8e6e27cSRafael J. Wysocki 556b8e6e27cSRafael J. WysockiThe process holding a file descriptor obtained by opening the 557b8e6e27cSRafael J. Wysocki:file:`cpu_dma_latency` special device file controls the PM QoS request 558b8e6e27cSRafael J. Wysockiassociated with that file descriptor, but it controls this particular PM QoS 559b8e6e27cSRafael J. Wysockirequest only. 560aa5eee35SRafael J. Wysocki 561aa5eee35SRafael J. WysockiClosing the :file:`cpu_dma_latency` special device file or, more precisely, the 562aa5eee35SRafael J. Wysockifile descriptor obtained while opening it, causes the PM QoS request associated 563b8e6e27cSRafael J. Wysockiwith that file descriptor to be removed from the global priority list of CPU 564b8e6e27cSRafael J. Wysockilatency limit requests and destroyed. If that happens, the priority list 565b8e6e27cSRafael J. Wysockimechanism will be used again, to determine the new effective value for the whole 566b8e6e27cSRafael J. Wysockilist and that value will become the new limit. 567aa5eee35SRafael J. Wysocki 568f06572efSRafael J. WysockiIn turn, for each CPU there is one resume latency PM QoS request associated with 569f06572efSRafael J. Wysockithe :file:`power/pm_qos_resume_latency_us` file under 570aa5eee35SRafael J. Wysocki:file:`/sys/devices/system/cpu/cpu<N>/` in ``sysfs`` and writing to it causes 571aa5eee35SRafael J. Wysockithis single PM QoS request to be updated regardless of which user space 572aa5eee35SRafael J. Wysockiprocess does that. In other words, this PM QoS request is shared by the entire 573aa5eee35SRafael J. Wysockiuser space, so access to the file associated with it needs to be arbitrated 574aa5eee35SRafael J. Wysockito avoid confusion. [Arguably, the only legitimate use of this mechanism in 575aa5eee35SRafael J. Wysockipractice is to pin a process to the CPU in question and let it use the 576f06572efSRafael J. Wysocki``sysfs`` interface to control the resume latency constraint for it.] It is 577f06572efSRafael J. Wysockistill only a request, however. It is an entry in a priority list used to 578aa5eee35SRafael J. Wysockidetermine the effective value to be set as the resume latency constraint for the 579aa5eee35SRafael J. WysockiCPU in question every time the list of requests is updated this way or another 580aa5eee35SRafael J. Wysocki(there may be other requests coming from kernel code in that list). 581aa5eee35SRafael J. Wysocki 582aa5eee35SRafael J. WysockiCPU idle time governors are expected to regard the minimum of the global 583b8e6e27cSRafael J. Wysocki(effective) CPU latency limit and the effective resume latency constraint for 584b8e6e27cSRafael J. Wysockithe given CPU as the upper limit for the exit latency of the idle states that 585b8e6e27cSRafael J. Wysockithey are allowed to select for that CPU. They should never select any idle 586b8e6e27cSRafael J. Wysockistates with exit latency beyond that limit. 587aa5eee35SRafael J. Wysocki 588aa5eee35SRafael J. Wysocki 589aa5eee35SRafael J. WysockiIdle States Control Via Kernel Command Line 590aa5eee35SRafael J. Wysocki=========================================== 591aa5eee35SRafael J. Wysocki 592aa5eee35SRafael J. WysockiIn addition to the ``sysfs`` interface allowing individual idle states to be 593aa5eee35SRafael J. Wysocki`disabled for individual CPUs <idle-states-representation_>`_, there are kernel 594aa5eee35SRafael J. Wysockicommand line parameters affecting CPU idle time management. 595aa5eee35SRafael J. Wysocki 596aa5eee35SRafael J. WysockiThe ``cpuidle.off=1`` kernel command line option can be used to disable the 597aa5eee35SRafael J. WysockiCPU idle time management entirely. It does not prevent the idle loop from 598aa5eee35SRafael J. Wysockirunning on idle CPUs, but it prevents the CPU idle time governors and drivers 599aa5eee35SRafael J. Wysockifrom being invoked. If it is added to the kernel command line, the idle loop 600aa5eee35SRafael J. Wysockiwill ask the hardware to enter idle states on idle CPUs via the CPU architecture 601aa5eee35SRafael J. Wysockisupport code that is expected to provide a default mechanism for this purpose. 602aa5eee35SRafael J. WysockiThat default mechanism usually is the least common denominator for all of the 603aa5eee35SRafael J. Wysockiprocessors implementing the architecture (i.e. CPU instruction set) in question, 604aa5eee35SRafael J. Wysockihowever, so it is rather crude and not very energy-efficient. For this reason, 605aa5eee35SRafael J. Wysockiit is not recommended for production use. 606aa5eee35SRafael J. Wysocki 60761cb5758SRafael J. WysockiThe ``cpuidle.governor=`` kernel command line switch allows the ``CPUIdle`` 60861cb5758SRafael J. Wysockigovernor to use to be specified. It has to be appended with a string matching 60961cb5758SRafael J. Wysockithe name of an available governor (e.g. ``cpuidle.governor=menu``) and that 61061cb5758SRafael J. Wysockigovernor will be used instead of the default one. It is possible to force 61161cb5758SRafael J. Wysockithe ``menu`` governor to be used on the systems that use the ``ladder`` governor 61261cb5758SRafael J. Wysockiby default this way, for example. 61361cb5758SRafael J. Wysocki 614aa5eee35SRafael J. WysockiThe other kernel command line parameters controlling CPU idle time management 615aa5eee35SRafael J. Wysockidescribed below are only relevant for the *x86* architecture and some of 616aa5eee35SRafael J. Wysockithem affect Intel processors only. 617aa5eee35SRafael J. Wysocki 618aa5eee35SRafael J. WysockiThe *x86* architecture support code recognizes three kernel command line 619aa5eee35SRafael J. Wysockioptions related to CPU idle time management: ``idle=poll``, ``idle=halt``, 620aa5eee35SRafael J. Wysockiand ``idle=nomwait``. The first two of them disable the ``acpi_idle`` and 621aa5eee35SRafael J. Wysocki``intel_idle`` drivers altogether, which effectively causes the entire 622aa5eee35SRafael J. Wysocki``CPUIdle`` subsystem to be disabled and makes the idle loop invoke the 623aa5eee35SRafael J. Wysockiarchitecture support code to deal with idle CPUs. How it does that depends on 624aa5eee35SRafael J. Wysockiwhich of the two parameters is added to the kernel command line. In the 625aa5eee35SRafael J. Wysocki``idle=halt`` case, the architecture support code will use the ``HLT`` 626aa5eee35SRafael J. Wysockiinstruction of the CPUs (which, as a rule, suspends the execution of the program 627aa5eee35SRafael J. Wysockiand causes the hardware to attempt to enter the shallowest available idle state) 628aa5eee35SRafael J. Wysockifor this purpose, and if ``idle=poll`` is used, idle CPUs will execute a 629eac53b3eSYoann Congalmore or less "lightweight" sequence of instructions in a tight loop. [Note 630aa5eee35SRafael J. Wysockithat using ``idle=poll`` is somewhat drastic in many cases, as preventing idle 631aa5eee35SRafael J. WysockiCPUs from saving almost any energy at all may not be the only effect of it. 632aa5eee35SRafael J. WysockiFor example, on Intel hardware it effectively prevents CPUs from using 633aa5eee35SRafael J. WysockiP-states (see |cpufreq|) that require any number of CPUs in a package to be 634aa5eee35SRafael J. Wysockiidle, so it very well may hurt single-thread computations performance as well as 635aa5eee35SRafael J. Wysockienergy-efficiency. Thus using it for performance reasons may not be a good idea 636aa5eee35SRafael J. Wysockiat all.] 637aa5eee35SRafael J. Wysocki 638aa5eee35SRafael J. WysockiThe ``idle=nomwait`` option disables the ``intel_idle`` driver and causes 639aa5eee35SRafael J. Wysocki``acpi_idle`` to be used (as long as all of the information needed by it is 640aa5eee35SRafael J. Wysockithere in the system's ACPI tables), but it is not allowed to use the 641aa5eee35SRafael J. Wysocki``MWAIT`` instruction of the CPUs to ask the hardware to enter idle states. 642aa5eee35SRafael J. Wysocki 643aa5eee35SRafael J. WysockiIn addition to the architecture-level kernel command line options affecting CPU 644aa5eee35SRafael J. Wysockiidle time management, there are parameters affecting individual ``CPUIdle`` 645aa5eee35SRafael J. Wysockidrivers that can be passed to them via the kernel command line. Specifically, 646aa5eee35SRafael J. Wysockithe ``intel_idle.max_cstate=<n>`` and ``processor.max_cstate=<n>`` parameters, 647aa5eee35SRafael J. Wysockiwhere ``<n>`` is an idle state index also used in the name of the given 648aa5eee35SRafael J. Wysockistate's directory in ``sysfs`` (see 649aa5eee35SRafael J. Wysocki`Representation of Idle States <idle-states-representation_>`_), causes the 650aa5eee35SRafael J. Wysocki``intel_idle`` and ``acpi_idle`` drivers, respectively, to discard all of the 651aa5eee35SRafael J. Wysockiidle states deeper than idle state ``<n>``. In that case, they will never ask 652aa5eee35SRafael J. Wysockifor any of those idle states or expose them to the governor. [The behavior of 653aa5eee35SRafael J. Wysockithe two drivers is different for ``<n>`` equal to ``0``. Adding 654aa5eee35SRafael J. Wysocki``intel_idle.max_cstate=0`` to the kernel command line disables the 655aa5eee35SRafael J. Wysocki``intel_idle`` driver and allows ``acpi_idle`` to be used, whereas 656aa5eee35SRafael J. Wysocki``processor.max_cstate=0`` is equivalent to ``processor.max_cstate=1``. 657aa5eee35SRafael J. WysockiAlso, the ``acpi_idle`` driver is part of the ``processor`` kernel module that 658aa5eee35SRafael J. Wysockican be loaded separately and ``max_cstate=<n>`` can be passed to it as a module 659aa5eee35SRafael J. Wysockiparameter when it is loaded.] 660