1==================
2Memory Hot(Un)Plug
3==================
4
5This document describes generic Linux support for memory hot(un)plug with
6a focus on System RAM, including ZONE_MOVABLE support.
7
8.. contents:: :local:
9
10Introduction
11============
12
13Memory hot(un)plug allows for increasing and decreasing the size of physical
14memory available to a machine at runtime. In the simplest case, it consists of
15physically plugging or unplugging a DIMM at runtime, coordinated with the
16operating system.
17
18Memory hot(un)plug is used for various purposes:
19
20- The physical memory available to a machine can be adjusted at runtime, up- or
21  downgrading the memory capacity. This dynamic memory resizing, sometimes
22  referred to as "capacity on demand", is frequently used with virtual machines
23  and logical partitions.
24
25- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One
26  example is replacing failing memory modules.
27
28- Reducing energy consumption either by physically unplugging memory modules or
29  by logically unplugging (parts of) memory modules from Linux.
30
31Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also
32used to expose persistent memory, other performance-differentiated memory and
33reserved memory regions as ordinary system RAM to Linux.
34
35Linux only supports memory hot(un)plug on selected 64 bit architectures, such as
36x86_64, arm64, ppc64, s390x and ia64.
37
38Memory Hot(Un)Plug Granularity
39------------------------------
40
41Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the
42physical memory address space into chunks of the same size: memory sections. The
43size of a memory section is architecture dependent. For example, x86_64 uses
44128 MiB and ppc64 uses 16 MiB.
45
46Memory sections are combined into chunks referred to as "memory blocks". The
47size of a memory block is architecture dependent and corresponds to the smallest
48granularity that can be hot(un)plugged. The default size of a memory block is
49the same as memory section size, unless an architecture specifies otherwise.
50
51All memory blocks have the same size.
52
53Phases of Memory Hotplug
54------------------------
55
56Memory hotplug consists of two phases:
57
58(1) Adding the memory to Linux
59(2) Onlining memory blocks
60
61In the first phase, metadata, such as the memory map ("memmap") and page tables
62for the direct mapping, is allocated and initialized, and memory blocks are
63created; the latter also creates sysfs files for managing newly created memory
64blocks.
65
66In the second phase, added memory is exposed to the page allocator. After this
67phase, the memory is visible in memory statistics, such as free and total
68memory, of the system.
69
70Phases of Memory Hotunplug
71--------------------------
72
73Memory hotunplug consists of two phases:
74
75(1) Offlining memory blocks
76(2) Removing the memory from Linux
77
78In the fist phase, memory is "hidden" from the page allocator again, for
79example, by migrating busy memory to other memory locations and removing all
80relevant free pages from the page allocator After this phase, the memory is no
81longer visible in memory statistics of the system.
82
83In the second phase, the memory blocks are removed and metadata is freed.
84
85Memory Hotplug Notifications
86============================
87
88There are various ways how Linux is notified about memory hotplug events such
89that it can start adding hotplugged memory. This description is limited to
90systems that support ACPI; mechanisms specific to other firmware interfaces or
91virtual machines are not described.
92
93ACPI Notifications
94------------------
95
96Platforms that support ACPI, such as x86_64, can support memory hotplug
97notifications via ACPI.
98
99In general, a firmware supporting memory hotplug defines a memory class object
100HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI
101driver will hotplug the memory to Linux.
102
103If the firmware supports hotplug of NUMA nodes, it defines an object _HID
104"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all
105assigned memory devices are added to Linux by the ACPI driver.
106
107Similarly, Linux can be notified about requests to hotunplug a memory device or
108a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory
109blocks, and, if successful, hotunplug the memory from Linux.
110
111Manual Probing
112--------------
113
114On some architectures, the firmware may not be able to notify the operating
115system about a memory hotplug event. Instead, the memory has to be manually
116probed from user space.
117
118The probe interface is located at::
119
120	/sys/devices/system/memory/probe
121
122Only complete memory blocks can be probed. Individual memory blocks are probed
123by providing the physical start address of the memory block::
124
125	% echo addr > /sys/devices/system/memory/probe
126
127Which results in a memory block for the range [addr, addr + memory_block_size)
128being created.
129
130.. note::
131
132  Using the probe interface is discouraged as it is easy to crash the kernel,
133  because Linux cannot validate user input; this interface might be removed in
134  the future.
135
136Onlining and Offlining Memory Blocks
137====================================
138
139After a memory block has been created, Linux has to be instructed to actually
140make use of that memory: the memory block has to be "online".
141
142Before a memory block can be removed, Linux has to stop using any memory part of
143the memory block: the memory block has to be "offlined".
144
145The Linux kernel can be configured to automatically online added memory blocks
146and drivers automatically trigger offlining of memory blocks when trying
147hotunplug of memory. Memory blocks can only be removed once offlining succeeded
148and drivers may trigger offlining of memory blocks when attempting hotunplug of
149memory.
150
151Onlining Memory Blocks Manually
152-------------------------------
153
154If auto-onlining of memory blocks isn't enabled, user-space has to manually
155trigger onlining of memory blocks. Often, udev rules are used to automate this
156task in user space.
157
158Onlining of a memory block can be triggered via::
159
160	% echo online > /sys/devices/system/memory/memoryXXX/state
161
162Or alternatively::
163
164	% echo 1 > /sys/devices/system/memory/memoryXXX/online
165
166The kernel will select the target zone automatically, depending on the
167configured ``online_policy``.
168
169One can explicitly request to associate an offline memory block with
170ZONE_MOVABLE by::
171
172	% echo online_movable > /sys/devices/system/memory/memoryXXX/state
173
174Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by::
175
176	% echo online_kernel > /sys/devices/system/memory/memoryXXX/state
177
178In any case, if onlining succeeds, the state of the memory block is changed to
179be "online". If it fails, the state of the memory block will remain unchanged
180and the above commands will fail.
181
182Onlining Memory Blocks Automatically
183------------------------------------
184
185The kernel can be configured to try auto-onlining of newly added memory blocks.
186If this feature is disabled, the memory blocks will stay offline until
187explicitly onlined from user space.
188
189The configured auto-online behavior can be observed via::
190
191	% cat /sys/devices/system/memory/auto_online_blocks
192
193Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
194``online_movable`` to that file, like::
195
196	% echo online > /sys/devices/system/memory/auto_online_blocks
197
198Similarly to manual onlining, with ``online`` the kernel will select the
199target zone automatically, depending on the configured ``online_policy``.
200
201Modifying the auto-online behavior will only affect all subsequently added
202memory blocks only.
203
204.. note::
205
206  In corner cases, auto-onlining can fail. The kernel won't retry. Note that
207  auto-onlining is not expected to fail in default configurations.
208
209.. note::
210
211  DLPAR on ppc64 ignores the ``offline`` setting and will still online added
212  memory blocks; if onlining fails, memory blocks are removed again.
213
214Offlining Memory Blocks
215-----------------------
216
217In the current implementation, Linux's memory offlining will try migrating all
218movable pages off the affected memory block. As most kernel allocations, such as
219page tables, are unmovable, page migration can fail and, therefore, inhibit
220memory offlining from succeeding.
221
222Having the memory provided by memory block managed by ZONE_MOVABLE significantly
223increases memory offlining reliability; still, memory offlining can fail in
224some corner cases.
225
226Further, memory offlining might retry for a long time (or even forever), until
227aborted by the user.
228
229Offlining of a memory block can be triggered via::
230
231	% echo offline > /sys/devices/system/memory/memoryXXX/state
232
233Or alternatively::
234
235	% echo 0 > /sys/devices/system/memory/memoryXXX/online
236
237If offlining succeeds, the state of the memory block is changed to be "offline".
238If it fails, the state of the memory block will remain unchanged and the above
239commands will fail, for example, via::
240
241	bash: echo: write error: Device or resource busy
242
243or via::
244
245	bash: echo: write error: Invalid argument
246
247Observing the State of Memory Blocks
248------------------------------------
249
250The state (online/offline/going-offline) of a memory block can be observed
251either via::
252
253	% cat /sys/device/system/memory/memoryXXX/state
254
255Or alternatively (1/0) via::
256
257	% cat /sys/device/system/memory/memoryXXX/online
258
259For an online memory block, the managing zone can be observed via::
260
261	% cat /sys/device/system/memory/memoryXXX/valid_zones
262
263Configuring Memory Hot(Un)Plug
264==============================
265
266There are various ways how system administrators can configure memory
267hot(un)plug and interact with memory blocks, especially, to online them.
268
269Memory Hot(Un)Plug Configuration via Sysfs
270------------------------------------------
271
272Some memory hot(un)plug properties can be configured or inspected via sysfs in::
273
274	/sys/devices/system/memory/
275
276The following files are currently defined:
277
278====================== =========================================================
279``auto_online_blocks`` read-write: set or get the default state of new memory
280		       blocks; configure auto-onlining.
281
282		       The default value depends on the
283		       CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration
284		       option.
285
286		       See the ``state`` property of memory blocks for details.
287``block_size_bytes``   read-only: the size in bytes of a memory block.
288``probe``	       write-only: add (probe) selected memory blocks manually
289		       from user space by supplying the physical start address.
290
291		       Availability depends on the CONFIG_ARCH_MEMORY_PROBE
292		       kernel configuration option.
293``uevent``	       read-write: generic udev file for device subsystems.
294``crash_hotplug``      read-only: when changes to the system memory map
295		       occur due to hot un/plug of memory, this file contains
296		       '1' if the kernel updates the kdump capture kernel memory
297		       map itself (via elfcorehdr), or '0' if userspace must update
298		       the kdump capture kernel memory map.
299
300		       Availability depends on the CONFIG_MEMORY_HOTPLUG kernel
301		       configuration option.
302====================== =========================================================
303
304.. note::
305
306  When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two
307  additional files ``hard_offline_page`` and ``soft_offline_page`` are available
308  to trigger hwpoisoning of pages, for example, for testing purposes. Note that
309  this functionality is not really related to memory hot(un)plug or actual
310  offlining of memory blocks.
311
312Memory Block Configuration via Sysfs
313------------------------------------
314
315Each memory block is represented as a memory block device that can be
316onlined or offlined. All memory blocks have their device information located in
317sysfs. Each present memory block is listed under
318``/sys/devices/system/memory`` as::
319
320	/sys/devices/system/memory/memoryXXX
321
322where XXX is the memory block id; the number of digits is variable.
323
324A present memory block indicates that some memory in the range is present;
325however, a memory block might span memory holes. A memory block spanning memory
326holes cannot be offlined.
327
328For example, assume 1 GiB memory block size. A device for a memory starting at
3290x100000000 is ``/sys/device/system/memory/memory4``::
330
331	(0x100000000 / 1Gib = 4)
332
333This device covers address range [0x100000000 ... 0x140000000)
334
335The following files are currently defined:
336
337=================== ============================================================
338``online``	    read-write: simplified interface to trigger onlining /
339		    offlining and to observe the state of a memory block.
340		    When onlining, the zone is selected automatically.
341``phys_device``	    read-only: legacy interface only ever used on s390x to
342		    expose the covered storage increment.
343``phys_index``	    read-only: the memory block id (XXX).
344``removable``	    read-only: legacy interface that indicated whether a memory
345		    block was likely to be offlineable or not. Nowadays, the
346		    kernel return ``1`` if and only if it supports memory
347		    offlining.
348``state``	    read-write: advanced interface to trigger onlining /
349		    offlining and to observe the state of a memory block.
350
351		    When writing, ``online``, ``offline``, ``online_kernel`` and
352		    ``online_movable`` are supported.
353
354		    ``online_movable`` specifies onlining to ZONE_MOVABLE.
355		    ``online_kernel`` specifies onlining to the default kernel
356		    zone for the memory block, such as ZONE_NORMAL.
357                    ``online`` let's the kernel select the zone automatically.
358
359		    When reading, ``online``, ``offline`` and ``going-offline``
360		    may be returned.
361``uevent``	    read-write: generic uevent file for devices.
362``valid_zones``     read-only: when a block is online, shows the zone it
363		    belongs to; when a block is offline, shows what zone will
364		    manage it when the block will be onlined.
365
366		    For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,
367		    ``Movable`` and ``none`` may be returned. ``none`` indicates
368		    that memory provided by a memory block is managed by
369		    multiple zones or spans multiple nodes; such memory blocks
370		    cannot be offlined. ``Movable`` indicates ZONE_MOVABLE.
371		    Other values indicate a kernel zone.
372
373		    For offline memory blocks, the first column shows the
374		    zone the kernel would select when onlining the memory block
375		    right now without further specifying a zone.
376
377		    Availability depends on the CONFIG_MEMORY_HOTREMOVE
378		    kernel configuration option.
379=================== ============================================================
380
381.. note::
382
383  If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/
384  directories can also be accessed via symbolic links located in the
385  ``/sys/devices/system/node/node*`` directories.
386
387  For example::
388
389	/sys/devices/system/node/node0/memory9 -> ../../memory/memory9
390
391  A backlink will also be created::
392
393	/sys/devices/system/memory/memory9/node0 -> ../../node/node0
394
395Command Line Parameters
396-----------------------
397
398Some command line parameters affect memory hot(un)plug handling. The following
399command line parameters are relevant:
400
401======================== =======================================================
402``memhp_default_state``	 configure auto-onlining by essentially setting
403                         ``/sys/devices/system/memory/auto_online_blocks``.
404``movable_node``	 configure automatic zone selection in the kernel when
405			 using the ``contig-zones`` online policy. When
406			 set, the kernel will default to ZONE_MOVABLE when
407			 onlining a memory block, unless other zones can be kept
408			 contiguous.
409======================== =======================================================
410
411See Documentation/admin-guide/kernel-parameters.txt for a more generic
412description of these command line parameters.
413
414Module Parameters
415------------------
416
417Instead of additional command line parameters or sysfs files, the
418``memory_hotplug`` subsystem now provides a dedicated namespace for module
419parameters. Module parameters can be set via the command line by predicating
420them with ``memory_hotplug.`` such as::
421
422	memory_hotplug.memmap_on_memory=1
423
424and they can be observed (and some even modified at runtime) via::
425
426	/sys/module/memory_hotplug/parameters/
427
428The following module parameters are currently defined:
429
430================================ ===============================================
431``memmap_on_memory``		 read-write: Allocate memory for the memmap from
432				 the added memory block itself. Even if enabled,
433				 actual support depends on various other system
434				 properties and should only be regarded as a
435				 hint whether the behavior would be desired.
436
437				 While allocating the memmap from the memory
438				 block itself makes memory hotplug less likely
439				 to fail and keeps the memmap on the same NUMA
440				 node in any case, it can fragment physical
441				 memory in a way that huge pages in bigger
442				 granularity cannot be formed on hotplugged
443				 memory.
444
445				 With value "force" it could result in memory
446				 wastage due to memmap size limitations. For
447				 example, if the memmap for a memory block
448				 requires 1 MiB, but the pageblock size is 2
449				 MiB, 1 MiB of hotplugged memory will be wasted.
450				 Note that there are still cases where the
451				 feature cannot be enforced: for example, if the
452				 memmap is smaller than a single page, or if the
453				 architecture does not support the forced mode
454				 in all configurations.
455
456``online_policy``		 read-write: Set the basic policy used for
457				 automatic zone selection when onlining memory
458				 blocks without specifying a target zone.
459				 ``contig-zones`` has been the kernel default
460				 before this parameter was added. After an
461				 online policy was configured and memory was
462				 online, the policy should not be changed
463				 anymore.
464
465				 When set to ``contig-zones``, the kernel will
466				 try keeping zones contiguous. If a memory block
467				 intersects multiple zones or no zone, the
468				 behavior depends on the ``movable_node`` kernel
469				 command line parameter: default to ZONE_MOVABLE
470				 if set, default to the applicable kernel zone
471				 (usually ZONE_NORMAL) if not set.
472
473				 When set to ``auto-movable``, the kernel will
474				 try onlining memory blocks to ZONE_MOVABLE if
475				 possible according to the configuration and
476				 memory device details. With this policy, one
477				 can avoid zone imbalances when eventually
478				 hotplugging a lot of memory later and still
479				 wanting to be able to hotunplug as much as
480				 possible reliably, very desirable in
481				 virtualized environments. This policy ignores
482				 the ``movable_node`` kernel command line
483				 parameter and isn't really applicable in
484				 environments that require it (e.g., bare metal
485				 with hotunpluggable nodes) where hotplugged
486				 memory might be exposed via the
487				 firmware-provided memory map early during boot
488				 to the system instead of getting detected,
489				 added and onlined  later during boot (such as
490				 done by virtio-mem or by some hypervisors
491				 implementing emulated DIMMs). As one example, a
492				 hotplugged DIMM will be onlined either
493				 completely to ZONE_MOVABLE or completely to
494				 ZONE_NORMAL, not a mixture.
495				 As another example, as many memory blocks
496				 belonging to a virtio-mem device will be
497				 onlined to ZONE_MOVABLE as possible,
498				 special-casing units of memory blocks that can
499				 only get hotunplugged together. *This policy
500				 does not protect from setups that are
501				 problematic with ZONE_MOVABLE and does not
502				 change the zone of memory blocks dynamically
503				 after they were onlined.*
504``auto_movable_ratio``		 read-write: Set the maximum MOVABLE:KERNEL
505				 memory ratio in % for the ``auto-movable``
506				 online policy. Whether the ratio applies only
507				 for the system across all NUMA nodes or also
508				 per NUMA nodes depends on the
509				 ``auto_movable_numa_aware`` configuration.
510
511				 All accounting is based on present memory pages
512				 in the zones combined with accounting per
513				 memory device. Memory dedicated to the CMA
514				 allocator is accounted as MOVABLE, although
515				 residing on one of the kernel zones. The
516				 possible ratio depends on the actual workload.
517				 The kernel default is "301" %, for example,
518				 allowing for hotplugging 24 GiB to a 8 GiB VM
519				 and automatically onlining all hotplugged
520				 memory to ZONE_MOVABLE in many setups. The
521				 additional 1% deals with some pages being not
522				 present, for example, because of some firmware
523				 allocations.
524
525				 Note that ZONE_NORMAL memory provided by one
526				 memory device does not allow for more
527				 ZONE_MOVABLE memory for a different memory
528				 device. As one example, onlining memory of a
529				 hotplugged DIMM to ZONE_NORMAL will not allow
530				 for another hotplugged DIMM to get onlined to
531				 ZONE_MOVABLE automatically. In contrast, memory
532				 hotplugged by a virtio-mem device that got
533				 onlined to ZONE_NORMAL will allow for more
534				 ZONE_MOVABLE memory within *the same*
535				 virtio-mem device.
536``auto_movable_numa_aware``	 read-write: Configure whether the
537				 ``auto_movable_ratio`` in the ``auto-movable``
538				 online policy also applies per NUMA
539				 node in addition to the whole system across all
540				 NUMA nodes. The kernel default is "Y".
541
542				 Disabling NUMA awareness can be helpful when
543				 dealing with NUMA nodes that should be
544				 completely hotunpluggable, onlining the memory
545				 completely to ZONE_MOVABLE automatically if
546				 possible.
547
548				 Parameter availability depends on CONFIG_NUMA.
549================================ ===============================================
550
551ZONE_MOVABLE
552============
553
554ZONE_MOVABLE is an important mechanism for more reliable memory offlining.
555Further, having system RAM managed by ZONE_MOVABLE instead of one of the
556kernel zones can increase the number of possible transparent huge pages and
557dynamically allocated huge pages.
558
559Most kernel allocations are unmovable. Important examples include the memory
560map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations
561can only be served from the kernel zones.
562
563Most user space pages, such as anonymous memory, and page cache pages are
564movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones.
565
566Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable
567allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is
568absolutely no guarantee whether a memory block can be offlined successfully.
569
570Zone Imbalances
571---------------
572
573Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance,
574which can harm the system or degrade performance. As one example, the kernel
575might crash because it runs out of free memory for unmovable allocations,
576although there is still plenty of free memory left in ZONE_MOVABLE.
577
578Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1
579are definitely impossible due to the overhead for the memory map.
580
581Actual safe zone ratios depend on the workload. Extreme cases, like excessive
582long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all.
583
584.. note::
585
586  CMA memory part of a kernel zone essentially behaves like memory in
587  ZONE_MOVABLE and similar considerations apply, especially when combining
588  CMA with ZONE_MOVABLE.
589
590ZONE_MOVABLE Sizing Considerations
591----------------------------------
592
593We usually expect that a large portion of available system RAM will actually
594be consumed by user space, either directly or indirectly via the page cache. In
595the normal case, ZONE_MOVABLE can be used when allocating such pages just fine.
596
597With that in mind, it makes sense that we can have a big portion of system RAM
598managed by ZONE_MOVABLE. However, there are some things to consider when using
599ZONE_MOVABLE, especially when fine-tuning zone ratios:
600
601- Having a lot of offline memory blocks. Even offline memory blocks consume
602  memory for metadata and page tables in the direct map; having a lot of offline
603  memory blocks is not a typical case, though.
604
605- Memory ballooning without balloon compaction is incompatible with
606  ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
607  pseries CMM, fully support balloon compaction.
608
609  Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be
610  disabled. In that case, balloon inflation will only perform unmovable
611  allocations and silently create a zone imbalance, usually triggered by
612  inflation requests from the hypervisor.
613
614- Gigantic pages are unmovable, resulting in user space consuming a
615  lot of unmovable memory.
616
617- Huge pages are unmovable when an architectures does not support huge
618  page migration, resulting in a similar issue as with gigantic pages.
619
620- Page tables are unmovable. Excessive swapping, mapping extremely large
621  files or ZONE_DEVICE memory can be problematic, although only really relevant
622  in corner cases. When we manage a lot of user space memory that has been
623  swapped out or is served from a file/persistent memory/... we still need a lot
624  of page tables to manage that memory once user space accessed that memory.
625
626- In certain DAX configurations the memory map for the device memory will be
627  allocated from the kernel zones.
628
629- KASAN can have a significant memory overhead, for example, consuming 1/8th of
630  the total system memory size as (unmovable) tracking metadata.
631
632- Long-term pinning of pages. Techniques that rely on long-term pinnings
633  (especially, RDMA and vfio/mdev) are fundamentally problematic with
634  ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside
635  on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they
636  have to be migrated off that zone while pinning. Pinning a page can fail
637  even if there is plenty of free memory in ZONE_MOVABLE.
638
639  In addition, using ZONE_MOVABLE might make page pinning more expensive,
640  because of the page migration overhead.
641
642By default, all the memory configured at boot time is managed by the kernel
643zones and ZONE_MOVABLE is not used.
644
645To enable ZONE_MOVABLE to include the memory present at boot and to control the
646ratio between movable and kernel zones there are two command line options:
647``kernelcore=`` and ``movablecore=``. See
648Documentation/admin-guide/kernel-parameters.rst for their description.
649
650Memory Offlining and ZONE_MOVABLE
651---------------------------------
652
653Even with ZONE_MOVABLE, there are some corner cases where offlining a memory
654block might fail:
655
656- Memory blocks with memory holes; this applies to memory blocks present during
657  boot and can apply to memory blocks hotplugged via the XEN balloon and the
658  Hyper-V balloon.
659
660- Mixed NUMA nodes and mixed zones within a single memory block prevent memory
661  offlining; this applies to memory blocks present during boot only.
662
663- Special memory blocks prevented by the system from getting offlined. Examples
664  include any memory available during boot on arm64 or memory blocks spanning
665  the crashkernel area on s390x; this usually applies to memory blocks present
666  during boot only.
667
668- Memory blocks overlapping with CMA areas cannot be offlined, this applies to
669  memory blocks present during boot only.
670
671- Concurrent activity that operates on the same physical memory area, such as
672  allocating gigantic pages, can result in temporary offlining failures.
673
674- Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
675  Optimization (HVO) is enabled.
676
677  Offlining code may be able to migrate huge page contents, but may not be able
678  to dissolve the source huge page because it fails allocating (unmovable) pages
679  for the vmemmap, because the system might not have free memory in the kernel
680  zones left.
681
682  Users that depend on memory offlining to succeed for movable zones should
683  carefully consider whether the memory savings gained from this feature are
684  worth the risk of possibly not being able to offline memory in certain
685  situations.
686
687Further, when running into out of memory situations while migrating pages, or
688when still encountering permanently unmovable pages within ZONE_MOVABLE
689(-> BUG), memory offlining will keep retrying until it eventually succeeds.
690
691When offlining is triggered from user space, the offlining context can be
692terminated by sending a signal. A timeout based offlining can easily be
693implemented via::
694
695	% timeout $TIMEOUT offline_block | failure_handling
696