1=========================
2CPU hotplug in the Kernel
3=========================
4
5:Date: September, 2021
6:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
7         Rusty Russell <rusty@rustcorp.com.au>,
8         Srivatsa Vaddagiri <vatsa@in.ibm.com>,
9         Ashok Raj <ashok.raj@intel.com>,
10         Joel Schopp <jschopp@austin.ibm.com>,
11	 Thomas Gleixner <tglx@linutronix.de>
12
13Introduction
14============
15
16Modern advances in system architectures have introduced advanced error
17reporting and correction capabilities in processors. There are couple OEMS that
18support NUMA hardware which are hot pluggable as well, where physical node
19insertion and removal require support for CPU hotplug.
20
21Such advances require CPUs available to a kernel to be removed either for
22provisioning reasons, or for RAS purposes to keep an offending CPU off
23system execution path. Hence the need for CPU hotplug support in the
24Linux kernel.
25
26A more novel use of CPU-hotplug support is its use today in suspend resume
27support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
28which didn't support these methods.
29
30
31Command Line Switches
32=====================
33``maxcpus=n``
34  Restrict boot time CPUs to *n*. Say if you have four CPUs, using
35  ``maxcpus=2`` will only boot two. You can choose to bring the
36  other CPUs later online.
37
38``nr_cpus=n``
39  Restrict the total amount of CPUs the kernel will support. If the number
40  supplied here is lower than the number of physically available CPUs, then
41  those CPUs can not be brought online later.
42
43``additional_cpus=n``
44  Use this to limit hotpluggable CPUs. This option sets
45  ``cpu_possible_mask = cpu_present_mask + additional_cpus``
46
47  This option is limited to the IA64 architecture.
48
49``possible_cpus=n``
50  This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
51
52  This option is limited to the X86 and S390 architecture.
53
54``cpu0_hotplug``
55  Allow to shutdown CPU0.
56
57  This option is limited to the X86 architecture.
58
59CPU maps
60========
61
62``cpu_possible_mask``
63  Bitmap of possible CPUs that can ever be available in the
64  system. This is used to allocate some boot time memory for per_cpu variables
65  that aren't designed to grow/shrink as CPUs are made available or removed.
66  Once set during boot time discovery phase, the map is static, i.e no bits
67  are added or removed anytime. Trimming it accurately for your system needs
68  upfront can save some boot time memory.
69
70``cpu_online_mask``
71  Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
72  after a CPU is available for kernel scheduling and ready to receive
73  interrupts from devices. Its cleared when a CPU is brought down using
74  ``__cpu_disable()``, before which all OS services including interrupts are
75  migrated to another target CPU.
76
77``cpu_present_mask``
78  Bitmap of CPUs currently present in the system. Not all
79  of them may be online. When physical hotplug is processed by the relevant
80  subsystem (e.g ACPI) can change and new bit either be added or removed
81  from the map depending on the event is hot-add/hot-remove. There are currently
82  no locking rules as of now. Typical usage is to init topology during boot,
83  at which time hotplug is disabled.
84
85You really don't need to manipulate any of the system CPU maps. They should
86be read-only for most use. When setting up per-cpu resources almost always use
87``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
88``for_each_cpu()`` can be used to iterate over a custom CPU mask.
89
90Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
91
92
93Using CPU hotplug
94=================
95
96The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
97available on multiple architectures including ARM, MIPS, PowerPC and X86. The
98configuration is done via the sysfs interface::
99
100 $ ls -lh /sys/devices/system/cpu
101 total 0
102 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu0
103 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu1
104 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu2
105 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu3
106 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu4
107 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu5
108 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu6
109 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu7
110 drwxr-xr-x  2 root root    0 Dec 21 16:33 hotplug
111 -r--r--r--  1 root root 4.0K Dec 21 16:33 offline
112 -r--r--r--  1 root root 4.0K Dec 21 16:33 online
113 -r--r--r--  1 root root 4.0K Dec 21 16:33 possible
114 -r--r--r--  1 root root 4.0K Dec 21 16:33 present
115
116The files *offline*, *online*, *possible*, *present* represent the CPU masks.
117Each CPU folder contains an *online* file which controls the logical on (1) and
118off (0) state. To logically shutdown CPU4::
119
120 $ echo 0 > /sys/devices/system/cpu/cpu4/online
121  smpboot: CPU 4 is now offline
122
123Once the CPU is shutdown, it will be removed from */proc/interrupts*,
124*/proc/cpuinfo* and should also not be shown visible by the *top* command. To
125bring CPU4 back online::
126
127 $ echo 1 > /sys/devices/system/cpu/cpu4/online
128 smpboot: Booting Node 0 Processor 4 APIC 0x1
129
130The CPU is usable again. This should work on all CPUs, but CPU0 is often special
131and excluded from CPU hotplug.
132
133The CPU hotplug coordination
134============================
135
136The offline case
137----------------
138
139Once a CPU has been logically shutdown the teardown callbacks of registered
140hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
141at state ``CPUHP_OFFLINE``. This includes:
142
143* If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
144  will be set to true.
145* All processes are migrated away from this outgoing CPU to new CPUs.
146  The new CPU is chosen from each process' current cpuset, which may be
147  a subset of all online CPUs.
148* All interrupts targeted to this CPU are migrated to a new CPU
149* timers are also migrated to a new CPU
150* Once all services are migrated, kernel calls an arch specific routine
151  ``__cpu_disable()`` to perform arch specific cleanup.
152
153
154The CPU hotplug API
155===================
156
157CPU hotplug state machine
158-------------------------
159
160CPU hotplug uses a trivial state machine with a linear state space from
161CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown
162callback.
163
164When a CPU is onlined, the startup callbacks are invoked sequentially until
165the state CPUHP_ONLINE is reached. They can also be invoked when the
166callbacks of a state are set up or an instance is added to a multi-instance
167state.
168
169When a CPU is offlined the teardown callbacks are invoked in the reverse
170order sequentially until the state CPUHP_OFFLINE is reached. They can also
171be invoked when the callbacks of a state are removed or an instance is
172removed from a multi-instance state.
173
174If a usage site requires only a callback in one direction of the hotplug
175operations (CPU online or CPU offline) then the other not-required callback
176can be set to NULL when the state is set up.
177
178The state space is divided into three sections:
179
180* The PREPARE section
181
182  The PREPARE section covers the state space from CPUHP_OFFLINE to
183  CPUHP_BRINGUP_CPU.
184
185  The startup callbacks in this section are invoked before the CPU is
186  started during a CPU online operation. The teardown callbacks are invoked
187  after the CPU has become dysfunctional during a CPU offline operation.
188
189  The callbacks are invoked on a control CPU as they can't obviously run on
190  the hotplugged CPU which is either not yet started or has become
191  dysfunctional already.
192
193  The startup callbacks are used to setup resources which are required to
194  bring a CPU successfully online. The teardown callbacks are used to free
195  resources or to move pending work to an online CPU after the hotplugged
196  CPU became dysfunctional.
197
198  The startup callbacks are allowed to fail. If a callback fails, the CPU
199  online operation is aborted and the CPU is brought down to the previous
200  state (usually CPUHP_OFFLINE) again.
201
202  The teardown callbacks in this section are not allowed to fail.
203
204* The STARTING section
205
206  The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1
207  and CPUHP_AP_ONLINE.
208
209  The startup callbacks in this section are invoked on the hotplugged CPU
210  with interrupts disabled during a CPU online operation in the early CPU
211  setup code. The teardown callbacks are invoked with interrupts disabled
212  on the hotplugged CPU during a CPU offline operation shortly before the
213  CPU is completely shut down.
214
215  The callbacks in this section are not allowed to fail.
216
217  The callbacks are used for low level hardware initialization/shutdown and
218  for core subsystems.
219
220* The ONLINE section
221
222  The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and
223  CPUHP_ONLINE.
224
225  The startup callbacks in this section are invoked on the hotplugged CPU
226  during a CPU online operation. The teardown callbacks are invoked on the
227  hotplugged CPU during a CPU offline operation.
228
229  The callbacks are invoked in the context of the per CPU hotplug thread,
230  which is pinned on the hotplugged CPU. The callbacks are invoked with
231  interrupts and preemption enabled.
232
233  The callbacks are allowed to fail. When a callback fails the hotplug
234  operation is aborted and the CPU is brought back to the previous state.
235
236CPU online/offline operations
237-----------------------------
238
239A successful online operation looks like this::
240
241  [CPUHP_OFFLINE]
242  [CPUHP_OFFLINE + 1]->startup()       -> success
243  [CPUHP_OFFLINE + 2]->startup()       -> success
244  [CPUHP_OFFLINE + 3]                  -> skipped because startup == NULL
245  ...
246  [CPUHP_BRINGUP_CPU]->startup()       -> success
247  === End of PREPARE section
248  [CPUHP_BRINGUP_CPU + 1]->startup()   -> success
249  ...
250  [CPUHP_AP_ONLINE]->startup()         -> success
251  === End of STARTUP section
252  [CPUHP_AP_ONLINE + 1]->startup()     -> success
253  ...
254  [CPUHP_ONLINE - 1]->startup()        -> success
255  [CPUHP_ONLINE]
256
257A successful offline operation looks like this::
258
259  [CPUHP_ONLINE]
260  [CPUHP_ONLINE - 1]->teardown()       -> success
261  ...
262  [CPUHP_AP_ONLINE + 1]->teardown()    -> success
263  === Start of STARTUP section
264  [CPUHP_AP_ONLINE]->teardown()        -> success
265  ...
266  [CPUHP_BRINGUP_ONLINE - 1]->teardown()
267  ...
268  === Start of PREPARE section
269  [CPUHP_BRINGUP_CPU]->teardown()
270  [CPUHP_OFFLINE + 3]->teardown()
271  [CPUHP_OFFLINE + 2]                  -> skipped because teardown == NULL
272  [CPUHP_OFFLINE + 1]->teardown()
273  [CPUHP_OFFLINE]
274
275A failed online operation looks like this::
276
277  [CPUHP_OFFLINE]
278  [CPUHP_OFFLINE + 1]->startup()       -> success
279  [CPUHP_OFFLINE + 2]->startup()       -> success
280  [CPUHP_OFFLINE + 3]                  -> skipped because startup == NULL
281  ...
282  [CPUHP_BRINGUP_CPU]->startup()       -> success
283  === End of PREPARE section
284  [CPUHP_BRINGUP_CPU + 1]->startup()   -> success
285  ...
286  [CPUHP_AP_ONLINE]->startup()         -> success
287  === End of STARTUP section
288  [CPUHP_AP_ONLINE + 1]->startup()     -> success
289  ---
290  [CPUHP_AP_ONLINE + N]->startup()     -> fail
291  [CPUHP_AP_ONLINE + (N - 1)]->teardown()
292  ...
293  [CPUHP_AP_ONLINE + 1]->teardown()
294  === Start of STARTUP section
295  [CPUHP_AP_ONLINE]->teardown()
296  ...
297  [CPUHP_BRINGUP_ONLINE - 1]->teardown()
298  ...
299  === Start of PREPARE section
300  [CPUHP_BRINGUP_CPU]->teardown()
301  [CPUHP_OFFLINE + 3]->teardown()
302  [CPUHP_OFFLINE + 2]                  -> skipped because teardown == NULL
303  [CPUHP_OFFLINE + 1]->teardown()
304  [CPUHP_OFFLINE]
305
306A failed offline operation looks like this::
307
308  [CPUHP_ONLINE]
309  [CPUHP_ONLINE - 1]->teardown()       -> success
310  ...
311  [CPUHP_ONLINE - N]->teardown()       -> fail
312  [CPUHP_ONLINE - (N - 1)]->startup()
313  ...
314  [CPUHP_ONLINE - 1]->startup()
315  [CPUHP_ONLINE]
316
317Recursive failures cannot be handled sensibly. Look at the following
318example of a recursive fail due to a failed offline operation: ::
319
320  [CPUHP_ONLINE]
321  [CPUHP_ONLINE - 1]->teardown()       -> success
322  ...
323  [CPUHP_ONLINE - N]->teardown()       -> fail
324  [CPUHP_ONLINE - (N - 1)]->startup()  -> success
325  [CPUHP_ONLINE - (N - 2)]->startup()  -> fail
326
327The CPU hotplug state machine stops right here and does not try to go back
328down again because that would likely result in an endless loop::
329
330  [CPUHP_ONLINE - (N - 1)]->teardown() -> success
331  [CPUHP_ONLINE - N]->teardown()       -> fail
332  [CPUHP_ONLINE - (N - 1)]->startup()  -> success
333  [CPUHP_ONLINE - (N - 2)]->startup()  -> fail
334  [CPUHP_ONLINE - (N - 1)]->teardown() -> success
335  [CPUHP_ONLINE - N]->teardown()       -> fail
336
337Lather, rinse and repeat. In this case the CPU left in state::
338
339  [CPUHP_ONLINE - (N - 1)]
340
341which at least lets the system make progress and gives the user a chance to
342debug or even resolve the situation.
343
344Allocating a state
345------------------
346
347There are two ways to allocate a CPU hotplug state:
348
349* Static allocation
350
351  Static allocation has to be used when the subsystem or driver has
352  ordering requirements versus other CPU hotplug states. E.g. the PERF core
353  startup callback has to be invoked before the PERF driver startup
354  callbacks during a CPU online operation. During a CPU offline operation
355  the driver teardown callbacks have to be invoked before the core teardown
356  callback. The statically allocated states are described by constants in
357  the cpuhp_state enum which can be found in include/linux/cpuhotplug.h.
358
359  Insert the state into the enum at the proper place so the ordering
360  requirements are fulfilled. The state constant has to be used for state
361  setup and removal.
362
363  Static allocation is also required when the state callbacks are not set
364  up at runtime and are part of the initializer of the CPU hotplug state
365  array in kernel/cpu.c.
366
367* Dynamic allocation
368
369  When there are no ordering requirements for the state callbacks then
370  dynamic allocation is the preferred method. The state number is allocated
371  by the setup function and returned to the caller on success.
372
373  Only the PREPARE and ONLINE sections provide a dynamic allocation
374  range. The STARTING section does not as most of the callbacks in that
375  section have explicit ordering requirements.
376
377Setup of a CPU hotplug state
378----------------------------
379
380The core code provides the following functions to setup a state:
381
382* cpuhp_setup_state(state, name, startup, teardown)
383* cpuhp_setup_state_nocalls(state, name, startup, teardown)
384* cpuhp_setup_state_cpuslocked(state, name, startup, teardown)
385* cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown)
386
387For cases where a driver or a subsystem has multiple instances and the same
388CPU hotplug state callbacks need to be invoked for each instance, the CPU
389hotplug core provides multi-instance support. The advantage over driver
390specific instance lists is that the instance related functions are fully
391serialized against CPU hotplug operations and provide the automatic
392invocations of the state callbacks on add and removal. To set up such a
393multi-instance state the following function is available:
394
395* cpuhp_setup_state_multi(state, name, startup, teardown)
396
397The @state argument is either a statically allocated state or one of the
398constants for dynamically allocated states - CPUHP_PREPARE_DYN,
399CPUHP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for
400which a dynamic state should be allocated.
401
402The @name argument is used for sysfs output and for instrumentation. The
403naming convention is "subsys:mode" or "subsys/driver:mode",
404e.g. "perf:mode" or "perf/x86:mode". The common mode names are:
405
406======== =======================================================
407prepare  For states in the PREPARE section
408
409dead     For states in the PREPARE section which do not provide
410         a startup callback
411
412starting For states in the STARTING section
413
414dying    For states in the STARTING section which do not provide
415         a startup callback
416
417online   For states in the ONLINE section
418
419offline  For states in the ONLINE section which do not provide
420         a startup callback
421======== =======================================================
422
423As the @name argument is only used for sysfs and instrumentation other mode
424descriptors can be used as well if they describe the nature of the state
425better than the common ones.
426
427Examples for @name arguments: "perf/online", "perf/x86:prepare",
428"RCU/tree:dying", "sched/waitempty"
429
430The @startup argument is a function pointer to the callback which should be
431invoked during a CPU online operation. If the usage site does not require a
432startup callback set the pointer to NULL.
433
434The @teardown argument is a function pointer to the callback which should
435be invoked during a CPU offline operation. If the usage site does not
436require a teardown callback set the pointer to NULL.
437
438The functions differ in the way how the installed callbacks are treated:
439
440  * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked()
441    and cpuhp_setup_state_multi() only install the callbacks
442
443  * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the
444    callbacks and invoke the @startup callback (if not NULL) for all online
445    CPUs which have currently a state greater than the newly installed
446    state. Depending on the state section the callback is either invoked on
447    the current CPU (PREPARE section) or on each online CPU (ONLINE
448    section) in the context of the CPU's hotplug thread.
449
450    If a callback fails for CPU N then the teardown callback for CPU
451    0 .. N-1 is invoked to rollback the operation. The state setup fails,
452    the callbacks for the state are not installed and in case of dynamic
453    allocation the allocated state is freed.
454
455The state setup and the callback invocations are serialized against CPU
456hotplug operations. If the setup function has to be called from a CPU
457hotplug read locked region, then the _cpuslocked() variants have to be
458used. These functions cannot be used from within CPU hotplug callbacks.
459
460The function return values:
461  ======== ===================================================================
462  0        Statically allocated state was successfully set up
463
464  >0       Dynamically allocated state was successfully set up.
465
466           The returned number is the state number which was allocated. If
467           the state callbacks have to be removed later, e.g. module
468           removal, then this number has to be saved by the caller and used
469           as @state argument for the state remove function. For
470           multi-instance states the dynamically allocated state number is
471           also required as @state argument for the instance add/remove
472           operations.
473
474  <0	   Operation failed
475  ======== ===================================================================
476
477Removal of a CPU hotplug state
478------------------------------
479
480To remove a previously set up state, the following functions are provided:
481
482* cpuhp_remove_state(state)
483* cpuhp_remove_state_nocalls(state)
484* cpuhp_remove_state_nocalls_cpuslocked(state)
485* cpuhp_remove_multi_state(state)
486
487The @state argument is either a statically allocated state or the state
488number which was allocated in the dynamic range by cpuhp_setup_state*(). If
489the state is in the dynamic range, then the state number is freed and
490available for dynamic allocation again.
491
492The functions differ in the way how the installed callbacks are treated:
493
494  * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked()
495    and cpuhp_remove_multi_state() only remove the callbacks.
496
497  * cpuhp_remove_state() removes the callbacks and invokes the teardown
498    callback (if not NULL) for all online CPUs which have currently a state
499    greater than the removed state. Depending on the state section the
500    callback is either invoked on the current CPU (PREPARE section) or on
501    each online CPU (ONLINE section) in the context of the CPU's hotplug
502    thread.
503
504    In order to complete the removal, the teardown callback should not fail.
505
506The state removal and the callback invocations are serialized against CPU
507hotplug operations. If the remove function has to be called from a CPU
508hotplug read locked region, then the _cpuslocked() variants have to be
509used. These functions cannot be used from within CPU hotplug callbacks.
510
511If a multi-instance state is removed then the caller has to remove all
512instances first.
513
514Multi-Instance state instance management
515----------------------------------------
516
517Once the multi-instance state is set up, instances can be added to the
518state:
519
520  * cpuhp_state_add_instance(state, node)
521  * cpuhp_state_add_instance_nocalls(state, node)
522
523The @state argument is either a statically allocated state or the state
524number which was allocated in the dynamic range by cpuhp_setup_state_multi().
525
526The @node argument is a pointer to an hlist_node which is embedded in the
527instance's data structure. The pointer is handed to the multi-instance
528state callbacks and can be used by the callback to retrieve the instance
529via container_of().
530
531The functions differ in the way how the installed callbacks are treated:
532
533  * cpuhp_state_add_instance_nocalls() and only adds the instance to the
534    multi-instance state's node list.
535
536  * cpuhp_state_add_instance() adds the instance and invokes the startup
537    callback (if not NULL) associated with @state for all online CPUs which
538    have currently a state greater than @state. The callback is only
539    invoked for the to be added instance. Depending on the state section
540    the callback is either invoked on the current CPU (PREPARE section) or
541    on each online CPU (ONLINE section) in the context of the CPU's hotplug
542    thread.
543
544    If a callback fails for CPU N then the teardown callback for CPU
545    0 .. N-1 is invoked to rollback the operation, the function fails and
546    the instance is not added to the node list of the multi-instance state.
547
548To remove an instance from the state's node list these functions are
549available:
550
551  * cpuhp_state_remove_instance(state, node)
552  * cpuhp_state_remove_instance_nocalls(state, node)
553
554The arguments are the same as for the cpuhp_state_add_instance*()
555variants above.
556
557The functions differ in the way how the installed callbacks are treated:
558
559  * cpuhp_state_remove_instance_nocalls() only removes the instance from the
560    state's node list.
561
562  * cpuhp_state_remove_instance() removes the instance and invokes the
563    teardown callback (if not NULL) associated with @state for all online
564    CPUs which have currently a state greater than @state.  The callback is
565    only invoked for the to be removed instance.  Depending on the state
566    section the callback is either invoked on the current CPU (PREPARE
567    section) or on each online CPU (ONLINE section) in the context of the
568    CPU's hotplug thread.
569
570    In order to complete the removal, the teardown callback should not fail.
571
572The node list add/remove operations and the callback invocations are
573serialized against CPU hotplug operations. These functions cannot be used
574from within CPU hotplug callbacks and CPU hotplug read locked regions.
575
576Examples
577--------
578
579Setup and teardown a statically allocated state in the STARTING section for
580notifications on online and offline operations::
581
582   ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying);
583   if (ret < 0)
584        return ret;
585   ....
586   cpuhp_remove_state(CPUHP_SUBSYS_STARTING);
587
588Setup and teardown a dynamically allocated state in the ONLINE section
589for notifications on offline operations::
590
591   state = cpuhp_setup_state(CPUHP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline);
592   if (state < 0)
593       return state;
594   ....
595   cpuhp_remove_state(state);
596
597Setup and teardown a dynamically allocated state in the ONLINE section
598for notifications on online operations without invoking the callbacks::
599
600   state = cpuhp_setup_state_nocalls(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL);
601   if (state < 0)
602       return state;
603   ....
604   cpuhp_remove_state_nocalls(state);
605
606Setup, use and teardown a dynamically allocated multi-instance state in the
607ONLINE section for notifications on online and offline operation::
608
609   state = cpuhp_setup_state_multi(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline);
610   if (state < 0)
611       return state;
612   ....
613   ret = cpuhp_state_add_instance(state, &inst1->node);
614   if (ret)
615        return ret;
616   ....
617   ret = cpuhp_state_add_instance(state, &inst2->node);
618   if (ret)
619        return ret;
620   ....
621   cpuhp_remove_instance(state, &inst1->node);
622   ....
623   cpuhp_remove_instance(state, &inst2->node);
624   ....
625   remove_multi_state(state);
626
627
628Testing of hotplug states
629=========================
630
631One way to verify whether a custom state is working as expected or not is to
632shutdown a CPU and then put it online again. It is also possible to put the CPU
633to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
634*CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
635which would lead to rollback to the online state.
636
637All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states`` ::
638
639 $ tail /sys/devices/system/cpu/hotplug/states
640 138: mm/vmscan:online
641 139: mm/vmstat:online
642 140: lib/percpu_cnt:online
643 141: acpi/cpu-drv:online
644 142: base/cacheinfo:online
645 143: virtio/net:online
646 144: x86/mce:online
647 145: printk:online
648 168: sched:active
649 169: online
650
651To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue::
652
653  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
654  169
655  $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
656  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
657  140
658
659It is important to note that the teardown callback of state 140 have been
660invoked. And now get back online::
661
662  $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
663  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
664  169
665
666With trace events enabled, the individual steps are visible, too::
667
668  #  TASK-PID   CPU#    TIMESTAMP  FUNCTION
669  #     | |       |        |         |
670      bash-394  [001]  22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
671   cpuhp/4-31   [004]  22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
672   cpuhp/4-31   [004]  22.990: cpuhp_exit:  cpu: 0004  state: 168 step: 168 ret: 0
673   cpuhp/4-31   [004]  22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
674   cpuhp/4-31   [004]  22.992: cpuhp_exit:  cpu: 0004  state: 144 step: 144 ret: 0
675   cpuhp/4-31   [004]  22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
676   cpuhp/4-31   [004]  22.994: cpuhp_exit:  cpu: 0004  state: 143 step: 143 ret: 0
677   cpuhp/4-31   [004]  22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
678   cpuhp/4-31   [004]  22.996: cpuhp_exit:  cpu: 0004  state: 142 step: 142 ret: 0
679      bash-394  [001]  22.997: cpuhp_exit:  cpu: 0004  state: 140 step: 169 ret: 0
680      bash-394  [005]  95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
681   cpuhp/4-31   [004]  95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
682   cpuhp/4-31   [004]  95.542: cpuhp_exit:  cpu: 0004  state: 141 step: 141 ret: 0
683   cpuhp/4-31   [004]  95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
684   cpuhp/4-31   [004]  95.544: cpuhp_exit:  cpu: 0004  state: 142 step: 142 ret: 0
685   cpuhp/4-31   [004]  95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
686   cpuhp/4-31   [004]  95.546: cpuhp_exit:  cpu: 0004  state: 143 step: 143 ret: 0
687   cpuhp/4-31   [004]  95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
688   cpuhp/4-31   [004]  95.548: cpuhp_exit:  cpu: 0004  state: 144 step: 144 ret: 0
689   cpuhp/4-31   [004]  95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
690   cpuhp/4-31   [004]  95.550: cpuhp_exit:  cpu: 0004  state: 145 step: 145 ret: 0
691   cpuhp/4-31   [004]  95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
692   cpuhp/4-31   [004]  95.552: cpuhp_exit:  cpu: 0004  state: 168 step: 168 ret: 0
693      bash-394  [005]  95.553: cpuhp_exit:  cpu: 0004  state: 169 step: 140 ret: 0
694
695As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
69695.552. All invoked callbacks including their return codes are visible in the
697trace.
698
699Architecture's requirements
700===========================
701
702The following functions and configurations are required:
703
704``CONFIG_HOTPLUG_CPU``
705  This entry needs to be enabled in Kconfig
706
707``__cpu_up()``
708  Arch interface to bring up a CPU
709
710``__cpu_disable()``
711  Arch interface to shutdown a CPU, no more interrupts can be handled by the
712  kernel after the routine returns. This includes the shutdown of the timer.
713
714``__cpu_die()``
715  This actually supposed to ensure death of the CPU. Actually look at some
716  example code in other arch that implement CPU hotplug. The processor is taken
717  down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
718  typically waits for some per_cpu state to be set, to ensure the processor dead
719  routine is called to be sure positively.
720
721User Space Notification
722=======================
723
724After CPU successfully onlined or offline udev events are sent. A udev rule like::
725
726  SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
727
728will receive all events. A script like::
729
730  #!/bin/sh
731
732  if [ "${ACTION}" = "offline" ]
733  then
734      echo "CPU ${DEVPATH##*/} offline"
735
736  elif [ "${ACTION}" = "online" ]
737  then
738      echo "CPU ${DEVPATH##*/} online"
739
740  fi
741
742can process the event further.
743
744Kernel Inline Documentations Reference
745======================================
746
747.. kernel-doc:: include/linux/cpuhotplug.h
748