1=========================
2CPU hotplug in the Kernel
3=========================
4
5:Date: September, 2021
6:Author: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
7         Rusty Russell <rusty@rustcorp.com.au>,
8         Srivatsa Vaddagiri <vatsa@in.ibm.com>,
9         Ashok Raj <ashok.raj@intel.com>,
10         Joel Schopp <jschopp@austin.ibm.com>,
11	 Thomas Gleixner <tglx@linutronix.de>
12
13Introduction
14============
15
16Modern advances in system architectures have introduced advanced error
17reporting and correction capabilities in processors. There are couple OEMS that
18support NUMA hardware which are hot pluggable as well, where physical node
19insertion and removal require support for CPU hotplug.
20
21Such advances require CPUs available to a kernel to be removed either for
22provisioning reasons, or for RAS purposes to keep an offending CPU off
23system execution path. Hence the need for CPU hotplug support in the
24Linux kernel.
25
26A more novel use of CPU-hotplug support is its use today in suspend resume
27support for SMP. Dual-core and HT support makes even a laptop run SMP kernels
28which didn't support these methods.
29
30
31Command Line Switches
32=====================
33``maxcpus=n``
34  Restrict boot time CPUs to *n*. Say if you have four CPUs, using
35  ``maxcpus=2`` will only boot two. You can choose to bring the
36  other CPUs later online.
37
38``nr_cpus=n``
39  Restrict the total amount of CPUs the kernel will support. If the number
40  supplied here is lower than the number of physically available CPUs, then
41  those CPUs can not be brought online later.
42
43``additional_cpus=n``
44  Use this to limit hotpluggable CPUs. This option sets
45  ``cpu_possible_mask = cpu_present_mask + additional_cpus``
46
47  This option is limited to the IA64 architecture.
48
49``possible_cpus=n``
50  This option sets ``possible_cpus`` bits in ``cpu_possible_mask``.
51
52  This option is limited to the X86 and S390 architecture.
53
54``cpu0_hotplug``
55  Allow to shutdown CPU0.
56
57  This option is limited to the X86 architecture.
58
59CPU maps
60========
61
62``cpu_possible_mask``
63  Bitmap of possible CPUs that can ever be available in the
64  system. This is used to allocate some boot time memory for per_cpu variables
65  that aren't designed to grow/shrink as CPUs are made available or removed.
66  Once set during boot time discovery phase, the map is static, i.e no bits
67  are added or removed anytime. Trimming it accurately for your system needs
68  upfront can save some boot time memory.
69
70``cpu_online_mask``
71  Bitmap of all CPUs currently online. Its set in ``__cpu_up()``
72  after a CPU is available for kernel scheduling and ready to receive
73  interrupts from devices. Its cleared when a CPU is brought down using
74  ``__cpu_disable()``, before which all OS services including interrupts are
75  migrated to another target CPU.
76
77``cpu_present_mask``
78  Bitmap of CPUs currently present in the system. Not all
79  of them may be online. When physical hotplug is processed by the relevant
80  subsystem (e.g ACPI) can change and new bit either be added or removed
81  from the map depending on the event is hot-add/hot-remove. There are currently
82  no locking rules as of now. Typical usage is to init topology during boot,
83  at which time hotplug is disabled.
84
85You really don't need to manipulate any of the system CPU maps. They should
86be read-only for most use. When setting up per-cpu resources almost always use
87``cpu_possible_mask`` or ``for_each_possible_cpu()`` to iterate. To macro
88``for_each_cpu()`` can be used to iterate over a custom CPU mask.
89
90Never use anything other than ``cpumask_t`` to represent bitmap of CPUs.
91
92
93Using CPU hotplug
94=================
95
96The kernel option *CONFIG_HOTPLUG_CPU* needs to be enabled. It is currently
97available on multiple architectures including ARM, MIPS, PowerPC and X86. The
98configuration is done via the sysfs interface::
99
100 $ ls -lh /sys/devices/system/cpu
101 total 0
102 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu0
103 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu1
104 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu2
105 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu3
106 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu4
107 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu5
108 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu6
109 drwxr-xr-x  9 root root    0 Dec 21 16:33 cpu7
110 drwxr-xr-x  2 root root    0 Dec 21 16:33 hotplug
111 -r--r--r--  1 root root 4.0K Dec 21 16:33 offline
112 -r--r--r--  1 root root 4.0K Dec 21 16:33 online
113 -r--r--r--  1 root root 4.0K Dec 21 16:33 possible
114 -r--r--r--  1 root root 4.0K Dec 21 16:33 present
115
116The files *offline*, *online*, *possible*, *present* represent the CPU masks.
117Each CPU folder contains an *online* file which controls the logical on (1) and
118off (0) state. To logically shutdown CPU4::
119
120 $ echo 0 > /sys/devices/system/cpu/cpu4/online
121  smpboot: CPU 4 is now offline
122
123Once the CPU is shutdown, it will be removed from */proc/interrupts*,
124*/proc/cpuinfo* and should also not be shown visible by the *top* command. To
125bring CPU4 back online::
126
127 $ echo 1 > /sys/devices/system/cpu/cpu4/online
128 smpboot: Booting Node 0 Processor 4 APIC 0x1
129
130The CPU is usable again. This should work on all CPUs. CPU0 is often special
131and excluded from CPU hotplug. On X86 the kernel option
132*CONFIG_BOOTPARAM_HOTPLUG_CPU0* has to be enabled in order to be able to
133shutdown CPU0. Alternatively the kernel command option *cpu0_hotplug* can be
134used. Some known dependencies of CPU0:
135
136* Resume from hibernate/suspend. Hibernate/suspend will fail if CPU0 is offline.
137* PIC interrupts. CPU0 can't be removed if a PIC interrupt is detected.
138
139Please let Fenghua Yu <fenghua.yu@intel.com> know if you find any dependencies
140on CPU0.
141
142The CPU hotplug coordination
143============================
144
145The offline case
146----------------
147
148Once a CPU has been logically shutdown the teardown callbacks of registered
149hotplug states will be invoked, starting with ``CPUHP_ONLINE`` and terminating
150at state ``CPUHP_OFFLINE``. This includes:
151
152* If tasks are frozen due to a suspend operation then *cpuhp_tasks_frozen*
153  will be set to true.
154* All processes are migrated away from this outgoing CPU to new CPUs.
155  The new CPU is chosen from each process' current cpuset, which may be
156  a subset of all online CPUs.
157* All interrupts targeted to this CPU are migrated to a new CPU
158* timers are also migrated to a new CPU
159* Once all services are migrated, kernel calls an arch specific routine
160  ``__cpu_disable()`` to perform arch specific cleanup.
161
162
163The CPU hotplug API
164===================
165
166CPU hotplug state machine
167-------------------------
168
169CPU hotplug uses a trivial state machine with a linear state space from
170CPUHP_OFFLINE to CPUHP_ONLINE. Each state has a startup and a teardown
171callback.
172
173When a CPU is onlined, the startup callbacks are invoked sequentially until
174the state CPUHP_ONLINE is reached. They can also be invoked when the
175callbacks of a state are set up or an instance is added to a multi-instance
176state.
177
178When a CPU is offlined the teardown callbacks are invoked in the reverse
179order sequentially until the state CPUHP_OFFLINE is reached. They can also
180be invoked when the callbacks of a state are removed or an instance is
181removed from a multi-instance state.
182
183If a usage site requires only a callback in one direction of the hotplug
184operations (CPU online or CPU offline) then the other not-required callback
185can be set to NULL when the state is set up.
186
187The state space is divided into three sections:
188
189* The PREPARE section
190
191  The PREPARE section covers the state space from CPUHP_OFFLINE to
192  CPUHP_BRINGUP_CPU.
193
194  The startup callbacks in this section are invoked before the CPU is
195  started during a CPU online operation. The teardown callbacks are invoked
196  after the CPU has become dysfunctional during a CPU offline operation.
197
198  The callbacks are invoked on a control CPU as they can't obviously run on
199  the hotplugged CPU which is either not yet started or has become
200  dysfunctional already.
201
202  The startup callbacks are used to setup resources which are required to
203  bring a CPU successfully online. The teardown callbacks are used to free
204  resources or to move pending work to an online CPU after the hotplugged
205  CPU became dysfunctional.
206
207  The startup callbacks are allowed to fail. If a callback fails, the CPU
208  online operation is aborted and the CPU is brought down to the previous
209  state (usually CPUHP_OFFLINE) again.
210
211  The teardown callbacks in this section are not allowed to fail.
212
213* The STARTING section
214
215  The STARTING section covers the state space between CPUHP_BRINGUP_CPU + 1
216  and CPUHP_AP_ONLINE.
217
218  The startup callbacks in this section are invoked on the hotplugged CPU
219  with interrupts disabled during a CPU online operation in the early CPU
220  setup code. The teardown callbacks are invoked with interrupts disabled
221  on the hotplugged CPU during a CPU offline operation shortly before the
222  CPU is completely shut down.
223
224  The callbacks in this section are not allowed to fail.
225
226  The callbacks are used for low level hardware initialization/shutdown and
227  for core subsystems.
228
229* The ONLINE section
230
231  The ONLINE section covers the state space between CPUHP_AP_ONLINE + 1 and
232  CPUHP_ONLINE.
233
234  The startup callbacks in this section are invoked on the hotplugged CPU
235  during a CPU online operation. The teardown callbacks are invoked on the
236  hotplugged CPU during a CPU offline operation.
237
238  The callbacks are invoked in the context of the per CPU hotplug thread,
239  which is pinned on the hotplugged CPU. The callbacks are invoked with
240  interrupts and preemption enabled.
241
242  The callbacks are allowed to fail. When a callback fails the hotplug
243  operation is aborted and the CPU is brought back to the previous state.
244
245CPU online/offline operations
246-----------------------------
247
248A successful online operation looks like this::
249
250  [CPUHP_OFFLINE]
251  [CPUHP_OFFLINE + 1]->startup()       -> success
252  [CPUHP_OFFLINE + 2]->startup()       -> success
253  [CPUHP_OFFLINE + 3]                  -> skipped because startup == NULL
254  ...
255  [CPUHP_BRINGUP_CPU]->startup()       -> success
256  === End of PREPARE section
257  [CPUHP_BRINGUP_CPU + 1]->startup()   -> success
258  ...
259  [CPUHP_AP_ONLINE]->startup()         -> success
260  === End of STARTUP section
261  [CPUHP_AP_ONLINE + 1]->startup()     -> success
262  ...
263  [CPUHP_ONLINE - 1]->startup()        -> success
264  [CPUHP_ONLINE]
265
266A successful offline operation looks like this::
267
268  [CPUHP_ONLINE]
269  [CPUHP_ONLINE - 1]->teardown()       -> success
270  ...
271  [CPUHP_AP_ONLINE + 1]->teardown()    -> success
272  === Start of STARTUP section
273  [CPUHP_AP_ONLINE]->teardown()        -> success
274  ...
275  [CPUHP_BRINGUP_ONLINE - 1]->teardown()
276  ...
277  === Start of PREPARE section
278  [CPUHP_BRINGUP_CPU]->teardown()
279  [CPUHP_OFFLINE + 3]->teardown()
280  [CPUHP_OFFLINE + 2]                  -> skipped because teardown == NULL
281  [CPUHP_OFFLINE + 1]->teardown()
282  [CPUHP_OFFLINE]
283
284A failed online operation looks like this::
285
286  [CPUHP_OFFLINE]
287  [CPUHP_OFFLINE + 1]->startup()       -> success
288  [CPUHP_OFFLINE + 2]->startup()       -> success
289  [CPUHP_OFFLINE + 3]                  -> skipped because startup == NULL
290  ...
291  [CPUHP_BRINGUP_CPU]->startup()       -> success
292  === End of PREPARE section
293  [CPUHP_BRINGUP_CPU + 1]->startup()   -> success
294  ...
295  [CPUHP_AP_ONLINE]->startup()         -> success
296  === End of STARTUP section
297  [CPUHP_AP_ONLINE + 1]->startup()     -> success
298  ---
299  [CPUHP_AP_ONLINE + N]->startup()     -> fail
300  [CPUHP_AP_ONLINE + (N - 1)]->teardown()
301  ...
302  [CPUHP_AP_ONLINE + 1]->teardown()
303  === Start of STARTUP section
304  [CPUHP_AP_ONLINE]->teardown()
305  ...
306  [CPUHP_BRINGUP_ONLINE - 1]->teardown()
307  ...
308  === Start of PREPARE section
309  [CPUHP_BRINGUP_CPU]->teardown()
310  [CPUHP_OFFLINE + 3]->teardown()
311  [CPUHP_OFFLINE + 2]                  -> skipped because teardown == NULL
312  [CPUHP_OFFLINE + 1]->teardown()
313  [CPUHP_OFFLINE]
314
315A failed offline operation looks like this::
316
317  [CPUHP_ONLINE]
318  [CPUHP_ONLINE - 1]->teardown()       -> success
319  ...
320  [CPUHP_ONLINE - N]->teardown()       -> fail
321  [CPUHP_ONLINE - (N - 1)]->startup()
322  ...
323  [CPUHP_ONLINE - 1]->startup()
324  [CPUHP_ONLINE]
325
326Recursive failures cannot be handled sensibly. Look at the following
327example of a recursive fail due to a failed offline operation: ::
328
329  [CPUHP_ONLINE]
330  [CPUHP_ONLINE - 1]->teardown()       -> success
331  ...
332  [CPUHP_ONLINE - N]->teardown()       -> fail
333  [CPUHP_ONLINE - (N - 1)]->startup()  -> success
334  [CPUHP_ONLINE - (N - 2)]->startup()  -> fail
335
336The CPU hotplug state machine stops right here and does not try to go back
337down again because that would likely result in an endless loop::
338
339  [CPUHP_ONLINE - (N - 1)]->teardown() -> success
340  [CPUHP_ONLINE - N]->teardown()       -> fail
341  [CPUHP_ONLINE - (N - 1)]->startup()  -> success
342  [CPUHP_ONLINE - (N - 2)]->startup()  -> fail
343  [CPUHP_ONLINE - (N - 1)]->teardown() -> success
344  [CPUHP_ONLINE - N]->teardown()       -> fail
345
346Lather, rinse and repeat. In this case the CPU left in state::
347
348  [CPUHP_ONLINE - (N - 1)]
349
350which at least lets the system make progress and gives the user a chance to
351debug or even resolve the situation.
352
353Allocating a state
354------------------
355
356There are two ways to allocate a CPU hotplug state:
357
358* Static allocation
359
360  Static allocation has to be used when the subsystem or driver has
361  ordering requirements versus other CPU hotplug states. E.g. the PERF core
362  startup callback has to be invoked before the PERF driver startup
363  callbacks during a CPU online operation. During a CPU offline operation
364  the driver teardown callbacks have to be invoked before the core teardown
365  callback. The statically allocated states are described by constants in
366  the cpuhp_state enum which can be found in include/linux/cpuhotplug.h.
367
368  Insert the state into the enum at the proper place so the ordering
369  requirements are fulfilled. The state constant has to be used for state
370  setup and removal.
371
372  Static allocation is also required when the state callbacks are not set
373  up at runtime and are part of the initializer of the CPU hotplug state
374  array in kernel/cpu.c.
375
376* Dynamic allocation
377
378  When there are no ordering requirements for the state callbacks then
379  dynamic allocation is the preferred method. The state number is allocated
380  by the setup function and returned to the caller on success.
381
382  Only the PREPARE and ONLINE sections provide a dynamic allocation
383  range. The STARTING section does not as most of the callbacks in that
384  section have explicit ordering requirements.
385
386Setup of a CPU hotplug state
387----------------------------
388
389The core code provides the following functions to setup a state:
390
391* cpuhp_setup_state(state, name, startup, teardown)
392* cpuhp_setup_state_nocalls(state, name, startup, teardown)
393* cpuhp_setup_state_cpuslocked(state, name, startup, teardown)
394* cpuhp_setup_state_nocalls_cpuslocked(state, name, startup, teardown)
395
396For cases where a driver or a subsystem has multiple instances and the same
397CPU hotplug state callbacks need to be invoked for each instance, the CPU
398hotplug core provides multi-instance support. The advantage over driver
399specific instance lists is that the instance related functions are fully
400serialized against CPU hotplug operations and provide the automatic
401invocations of the state callbacks on add and removal. To set up such a
402multi-instance state the following function is available:
403
404* cpuhp_setup_state_multi(state, name, startup, teardown)
405
406The @state argument is either a statically allocated state or one of the
407constants for dynamically allocated states - CPUHP_PREPARE_DYN,
408CPUHP_ONLINE_DYN - depending on the state section (PREPARE, ONLINE) for
409which a dynamic state should be allocated.
410
411The @name argument is used for sysfs output and for instrumentation. The
412naming convention is "subsys:mode" or "subsys/driver:mode",
413e.g. "perf:mode" or "perf/x86:mode". The common mode names are:
414
415======== =======================================================
416prepare  For states in the PREPARE section
417
418dead     For states in the PREPARE section which do not provide
419         a startup callback
420
421starting For states in the STARTING section
422
423dying    For states in the STARTING section which do not provide
424         a startup callback
425
426online   For states in the ONLINE section
427
428offline  For states in the ONLINE section which do not provide
429         a startup callback
430======== =======================================================
431
432As the @name argument is only used for sysfs and instrumentation other mode
433descriptors can be used as well if they describe the nature of the state
434better than the common ones.
435
436Examples for @name arguments: "perf/online", "perf/x86:prepare",
437"RCU/tree:dying", "sched/waitempty"
438
439The @startup argument is a function pointer to the callback which should be
440invoked during a CPU online operation. If the usage site does not require a
441startup callback set the pointer to NULL.
442
443The @teardown argument is a function pointer to the callback which should
444be invoked during a CPU offline operation. If the usage site does not
445require a teardown callback set the pointer to NULL.
446
447The functions differ in the way how the installed callbacks are treated:
448
449  * cpuhp_setup_state_nocalls(), cpuhp_setup_state_nocalls_cpuslocked()
450    and cpuhp_setup_state_multi() only install the callbacks
451
452  * cpuhp_setup_state() and cpuhp_setup_state_cpuslocked() install the
453    callbacks and invoke the @startup callback (if not NULL) for all online
454    CPUs which have currently a state greater than the newly installed
455    state. Depending on the state section the callback is either invoked on
456    the current CPU (PREPARE section) or on each online CPU (ONLINE
457    section) in the context of the CPU's hotplug thread.
458
459    If a callback fails for CPU N then the teardown callback for CPU
460    0 .. N-1 is invoked to rollback the operation. The state setup fails,
461    the callbacks for the state are not installed and in case of dynamic
462    allocation the allocated state is freed.
463
464The state setup and the callback invocations are serialized against CPU
465hotplug operations. If the setup function has to be called from a CPU
466hotplug read locked region, then the _cpuslocked() variants have to be
467used. These functions cannot be used from within CPU hotplug callbacks.
468
469The function return values:
470  ======== ===================================================================
471  0        Statically allocated state was successfully set up
472
473  >0       Dynamically allocated state was successfully set up.
474
475           The returned number is the state number which was allocated. If
476           the state callbacks have to be removed later, e.g. module
477           removal, then this number has to be saved by the caller and used
478           as @state argument for the state remove function. For
479           multi-instance states the dynamically allocated state number is
480           also required as @state argument for the instance add/remove
481           operations.
482
483  <0	   Operation failed
484  ======== ===================================================================
485
486Removal of a CPU hotplug state
487------------------------------
488
489To remove a previously set up state, the following functions are provided:
490
491* cpuhp_remove_state(state)
492* cpuhp_remove_state_nocalls(state)
493* cpuhp_remove_state_nocalls_cpuslocked(state)
494* cpuhp_remove_multi_state(state)
495
496The @state argument is either a statically allocated state or the state
497number which was allocated in the dynamic range by cpuhp_setup_state*(). If
498the state is in the dynamic range, then the state number is freed and
499available for dynamic allocation again.
500
501The functions differ in the way how the installed callbacks are treated:
502
503  * cpuhp_remove_state_nocalls(), cpuhp_remove_state_nocalls_cpuslocked()
504    and cpuhp_remove_multi_state() only remove the callbacks.
505
506  * cpuhp_remove_state() removes the callbacks and invokes the teardown
507    callback (if not NULL) for all online CPUs which have currently a state
508    greater than the removed state. Depending on the state section the
509    callback is either invoked on the current CPU (PREPARE section) or on
510    each online CPU (ONLINE section) in the context of the CPU's hotplug
511    thread.
512
513    In order to complete the removal, the teardown callback should not fail.
514
515The state removal and the callback invocations are serialized against CPU
516hotplug operations. If the remove function has to be called from a CPU
517hotplug read locked region, then the _cpuslocked() variants have to be
518used. These functions cannot be used from within CPU hotplug callbacks.
519
520If a multi-instance state is removed then the caller has to remove all
521instances first.
522
523Multi-Instance state instance management
524----------------------------------------
525
526Once the multi-instance state is set up, instances can be added to the
527state:
528
529  * cpuhp_state_add_instance(state, node)
530  * cpuhp_state_add_instance_nocalls(state, node)
531
532The @state argument is either a statically allocated state or the state
533number which was allocated in the dynamic range by cpuhp_setup_state_multi().
534
535The @node argument is a pointer to an hlist_node which is embedded in the
536instance's data structure. The pointer is handed to the multi-instance
537state callbacks and can be used by the callback to retrieve the instance
538via container_of().
539
540The functions differ in the way how the installed callbacks are treated:
541
542  * cpuhp_state_add_instance_nocalls() and only adds the instance to the
543    multi-instance state's node list.
544
545  * cpuhp_state_add_instance() adds the instance and invokes the startup
546    callback (if not NULL) associated with @state for all online CPUs which
547    have currently a state greater than @state. The callback is only
548    invoked for the to be added instance. Depending on the state section
549    the callback is either invoked on the current CPU (PREPARE section) or
550    on each online CPU (ONLINE section) in the context of the CPU's hotplug
551    thread.
552
553    If a callback fails for CPU N then the teardown callback for CPU
554    0 .. N-1 is invoked to rollback the operation, the function fails and
555    the instance is not added to the node list of the multi-instance state.
556
557To remove an instance from the state's node list these functions are
558available:
559
560  * cpuhp_state_remove_instance(state, node)
561  * cpuhp_state_remove_instance_nocalls(state, node)
562
563The arguments are the same as for the the cpuhp_state_add_instance*()
564variants above.
565
566The functions differ in the way how the installed callbacks are treated:
567
568  * cpuhp_state_remove_instance_nocalls() only removes the instance from the
569    state's node list.
570
571  * cpuhp_state_remove_instance() removes the instance and invokes the
572    teardown callback (if not NULL) associated with @state for all online
573    CPUs which have currently a state greater than @state.  The callback is
574    only invoked for the to be removed instance.  Depending on the state
575    section the callback is either invoked on the current CPU (PREPARE
576    section) or on each online CPU (ONLINE section) in the context of the
577    CPU's hotplug thread.
578
579    In order to complete the removal, the teardown callback should not fail.
580
581The node list add/remove operations and the callback invocations are
582serialized against CPU hotplug operations. These functions cannot be used
583from within CPU hotplug callbacks and CPU hotplug read locked regions.
584
585Examples
586--------
587
588Setup and teardown a statically allocated state in the STARTING section for
589notifications on online and offline operations::
590
591   ret = cpuhp_setup_state(CPUHP_SUBSYS_STARTING, "subsys:starting", subsys_cpu_starting, subsys_cpu_dying);
592   if (ret < 0)
593        return ret;
594   ....
595   cpuhp_remove_state(CPUHP_SUBSYS_STARTING);
596
597Setup and teardown a dynamically allocated state in the ONLINE section
598for notifications on offline operations::
599
600   state = cpuhp_setup_state(CPUHP_ONLINE_DYN, "subsys:offline", NULL, subsys_cpu_offline);
601   if (state < 0)
602       return state;
603   ....
604   cpuhp_remove_state(state);
605
606Setup and teardown a dynamically allocated state in the ONLINE section
607for notifications on online operations without invoking the callbacks::
608
609   state = cpuhp_setup_state_nocalls(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, NULL);
610   if (state < 0)
611       return state;
612   ....
613   cpuhp_remove_state_nocalls(state);
614
615Setup, use and teardown a dynamically allocated multi-instance state in the
616ONLINE section for notifications on online and offline operation::
617
618   state = cpuhp_setup_state_multi(CPUHP_ONLINE_DYN, "subsys:online", subsys_cpu_online, subsys_cpu_offline);
619   if (state < 0)
620       return state;
621   ....
622   ret = cpuhp_state_add_instance(state, &inst1->node);
623   if (ret)
624        return ret;
625   ....
626   ret = cpuhp_state_add_instance(state, &inst2->node);
627   if (ret)
628        return ret;
629   ....
630   cpuhp_remove_instance(state, &inst1->node);
631   ....
632   cpuhp_remove_instance(state, &inst2->node);
633   ....
634   remove_multi_state(state);
635
636
637Testing of hotplug states
638=========================
639
640One way to verify whether a custom state is working as expected or not is to
641shutdown a CPU and then put it online again. It is also possible to put the CPU
642to certain state (for instance *CPUHP_AP_ONLINE*) and then go back to
643*CPUHP_ONLINE*. This would simulate an error one state after *CPUHP_AP_ONLINE*
644which would lead to rollback to the online state.
645
646All registered states are enumerated in ``/sys/devices/system/cpu/hotplug/states`` ::
647
648 $ tail /sys/devices/system/cpu/hotplug/states
649 138: mm/vmscan:online
650 139: mm/vmstat:online
651 140: lib/percpu_cnt:online
652 141: acpi/cpu-drv:online
653 142: base/cacheinfo:online
654 143: virtio/net:online
655 144: x86/mce:online
656 145: printk:online
657 168: sched:active
658 169: online
659
660To rollback CPU4 to ``lib/percpu_cnt:online`` and back online just issue::
661
662  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
663  169
664  $ echo 140 > /sys/devices/system/cpu/cpu4/hotplug/target
665  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
666  140
667
668It is important to note that the teardown callback of state 140 have been
669invoked. And now get back online::
670
671  $ echo 169 > /sys/devices/system/cpu/cpu4/hotplug/target
672  $ cat /sys/devices/system/cpu/cpu4/hotplug/state
673  169
674
675With trace events enabled, the individual steps are visible, too::
676
677  #  TASK-PID   CPU#    TIMESTAMP  FUNCTION
678  #     | |       |        |         |
679      bash-394  [001]  22.976: cpuhp_enter: cpu: 0004 target: 140 step: 169 (cpuhp_kick_ap_work)
680   cpuhp/4-31   [004]  22.977: cpuhp_enter: cpu: 0004 target: 140 step: 168 (sched_cpu_deactivate)
681   cpuhp/4-31   [004]  22.990: cpuhp_exit:  cpu: 0004  state: 168 step: 168 ret: 0
682   cpuhp/4-31   [004]  22.991: cpuhp_enter: cpu: 0004 target: 140 step: 144 (mce_cpu_pre_down)
683   cpuhp/4-31   [004]  22.992: cpuhp_exit:  cpu: 0004  state: 144 step: 144 ret: 0
684   cpuhp/4-31   [004]  22.993: cpuhp_multi_enter: cpu: 0004 target: 140 step: 143 (virtnet_cpu_down_prep)
685   cpuhp/4-31   [004]  22.994: cpuhp_exit:  cpu: 0004  state: 143 step: 143 ret: 0
686   cpuhp/4-31   [004]  22.995: cpuhp_enter: cpu: 0004 target: 140 step: 142 (cacheinfo_cpu_pre_down)
687   cpuhp/4-31   [004]  22.996: cpuhp_exit:  cpu: 0004  state: 142 step: 142 ret: 0
688      bash-394  [001]  22.997: cpuhp_exit:  cpu: 0004  state: 140 step: 169 ret: 0
689      bash-394  [005]  95.540: cpuhp_enter: cpu: 0004 target: 169 step: 140 (cpuhp_kick_ap_work)
690   cpuhp/4-31   [004]  95.541: cpuhp_enter: cpu: 0004 target: 169 step: 141 (acpi_soft_cpu_online)
691   cpuhp/4-31   [004]  95.542: cpuhp_exit:  cpu: 0004  state: 141 step: 141 ret: 0
692   cpuhp/4-31   [004]  95.543: cpuhp_enter: cpu: 0004 target: 169 step: 142 (cacheinfo_cpu_online)
693   cpuhp/4-31   [004]  95.544: cpuhp_exit:  cpu: 0004  state: 142 step: 142 ret: 0
694   cpuhp/4-31   [004]  95.545: cpuhp_multi_enter: cpu: 0004 target: 169 step: 143 (virtnet_cpu_online)
695   cpuhp/4-31   [004]  95.546: cpuhp_exit:  cpu: 0004  state: 143 step: 143 ret: 0
696   cpuhp/4-31   [004]  95.547: cpuhp_enter: cpu: 0004 target: 169 step: 144 (mce_cpu_online)
697   cpuhp/4-31   [004]  95.548: cpuhp_exit:  cpu: 0004  state: 144 step: 144 ret: 0
698   cpuhp/4-31   [004]  95.549: cpuhp_enter: cpu: 0004 target: 169 step: 145 (console_cpu_notify)
699   cpuhp/4-31   [004]  95.550: cpuhp_exit:  cpu: 0004  state: 145 step: 145 ret: 0
700   cpuhp/4-31   [004]  95.551: cpuhp_enter: cpu: 0004 target: 169 step: 168 (sched_cpu_activate)
701   cpuhp/4-31   [004]  95.552: cpuhp_exit:  cpu: 0004  state: 168 step: 168 ret: 0
702      bash-394  [005]  95.553: cpuhp_exit:  cpu: 0004  state: 169 step: 140 ret: 0
703
704As it an be seen, CPU4 went down until timestamp 22.996 and then back up until
70595.552. All invoked callbacks including their return codes are visible in the
706trace.
707
708Architecture's requirements
709===========================
710
711The following functions and configurations are required:
712
713``CONFIG_HOTPLUG_CPU``
714  This entry needs to be enabled in Kconfig
715
716``__cpu_up()``
717  Arch interface to bring up a CPU
718
719``__cpu_disable()``
720  Arch interface to shutdown a CPU, no more interrupts can be handled by the
721  kernel after the routine returns. This includes the shutdown of the timer.
722
723``__cpu_die()``
724  This actually supposed to ensure death of the CPU. Actually look at some
725  example code in other arch that implement CPU hotplug. The processor is taken
726  down from the ``idle()`` loop for that specific architecture. ``__cpu_die()``
727  typically waits for some per_cpu state to be set, to ensure the processor dead
728  routine is called to be sure positively.
729
730User Space Notification
731=======================
732
733After CPU successfully onlined or offline udev events are sent. A udev rule like::
734
735  SUBSYSTEM=="cpu", DRIVERS=="processor", DEVPATH=="/devices/system/cpu/*", RUN+="the_hotplug_receiver.sh"
736
737will receive all events. A script like::
738
739  #!/bin/sh
740
741  if [ "${ACTION}" = "offline" ]
742  then
743      echo "CPU ${DEVPATH##*/} offline"
744
745  elif [ "${ACTION}" = "online" ]
746  then
747      echo "CPU ${DEVPATH##*/} online"
748
749  fi
750
751can process the event further.
752
753Kernel Inline Documentations Reference
754======================================
755
756.. kernel-doc:: include/linux/cpuhotplug.h
757