Lines Matching refs:memory
5 This document describes generic Linux support for memory hot(un)plug with
14 memory available to a machine at runtime. In the simplest case, it consists of
20 - The physical memory available to a machine can be adjusted at runtime, up- or
21 downgrading the memory capacity. This dynamic memory resizing, sometimes
26 example is replacing failing memory modules.
28 - Reducing energy consumption either by physically unplugging memory modules or
29 by logically unplugging (parts of) memory modules from Linux.
31 Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also
32 used to expose persistent memory, other performance-differentiated memory and
33 reserved memory regions as ordinary system RAM to Linux.
35 Linux only supports memory hot(un)plug on selected 64 bit architectures, such as
41 Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the
42 physical memory address space into chunks of the same size: memory sections. The
43 size of a memory section is architecture dependent. For example, x86_64 uses
46 Memory sections are combined into chunks referred to as "memory blocks". The
47 size of a memory block is architecture dependent and corresponds to the smallest
48 granularity that can be hot(un)plugged. The default size of a memory block is
49 the same as memory section size, unless an architecture specifies otherwise.
51 All memory blocks have the same size.
58 (1) Adding the memory to Linux
59 (2) Onlining memory blocks
61 In the first phase, metadata, such as the memory map ("memmap") and page tables
62 for the direct mapping, is allocated and initialized, and memory blocks are
63 created; the latter also creates sysfs files for managing newly created memory
66 In the second phase, added memory is exposed to the page allocator. After this
67 phase, the memory is visible in memory statistics, such as free and total
68 memory, of the system.
75 (1) Offlining memory blocks
76 (2) Removing the memory from Linux
78 In the fist phase, memory is "hidden" from the page allocator again, for
79 example, by migrating busy memory to other memory locations and removing all
80 relevant free pages from the page allocator After this phase, the memory is no
81 longer visible in memory statistics of the system.
83 In the second phase, the memory blocks are removed and metadata is freed.
88 There are various ways how Linux is notified about memory hotplug events such
89 that it can start adding hotplugged memory. This description is limited to
96 Platforms that support ACPI, such as x86_64, can support memory hotplug
99 In general, a firmware supporting memory hotplug defines a memory class object
100 HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI
101 driver will hotplug the memory to Linux.
105 assigned memory devices are added to Linux by the ACPI driver.
107 Similarly, Linux can be notified about requests to hotunplug a memory device or
108 a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory
109 blocks, and, if successful, hotunplug the memory from Linux.
115 system about a memory hotplug event. Instead, the memory has to be manually
120 /sys/devices/system/memory/probe
122 Only complete memory blocks can be probed. Individual memory blocks are probed
123 by providing the physical start address of the memory block::
125 % echo addr > /sys/devices/system/memory/probe
127 Which results in a memory block for the range [addr, addr + memory_block_size)
139 After a memory block has been created, Linux has to be instructed to actually
140 make use of that memory: the memory block has to be "online".
142 Before a memory block can be removed, Linux has to stop using any memory part of
143 the memory block: the memory block has to be "offlined".
145 The Linux kernel can be configured to automatically online added memory blocks
146 and drivers automatically trigger offlining of memory blocks when trying
147 hotunplug of memory. Memory blocks can only be removed once offlining succeeded
148 and drivers may trigger offlining of memory blocks when attempting hotunplug of
149 memory.
154 If auto-onlining of memory blocks isn't enabled, user-space has to manually
155 trigger onlining of memory blocks. Often, udev rules are used to automate this
158 Onlining of a memory block can be triggered via::
160 % echo online > /sys/devices/system/memory/memoryXXX/state
164 % echo 1 > /sys/devices/system/memory/memoryXXX/online
169 One can explicitly request to associate an offline memory block with
172 % echo online_movable > /sys/devices/system/memory/memoryXXX/state
176 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state
178 In any case, if onlining succeeds, the state of the memory block is changed to
179 be "online". If it fails, the state of the memory block will remain unchanged
185 The kernel can be configured to try auto-onlining of newly added memory blocks.
186 If this feature is disabled, the memory blocks will stay offline until
191 % cat /sys/devices/system/memory/auto_online_blocks
196 % echo online > /sys/devices/system/memory/auto_online_blocks
202 memory blocks only.
212 memory blocks; if onlining fails, memory blocks are removed again.
217 In the current implementation, Linux's memory offlining will try migrating all
218 movable pages off the affected memory block. As most kernel allocations, such as
220 memory offlining from succeeding.
222 Having the memory provided by memory block managed by ZONE_MOVABLE significantly
223 increases memory offlining reliability; still, memory offlining can fail in
226 Further, memory offlining might retry for a long time (or even forever), until
229 Offlining of a memory block can be triggered via::
231 % echo offline > /sys/devices/system/memory/memoryXXX/state
235 % echo 0 > /sys/devices/system/memory/memoryXXX/online
237 If offlining succeeds, the state of the memory block is changed to be "offline".
238 If it fails, the state of the memory block will remain unchanged and the above
250 The state (online/offline/going-offline) of a memory block can be observed
253 % cat /sys/device/system/memory/memoryXXX/state
257 % cat /sys/device/system/memory/memoryXXX/online
259 For an online memory block, the managing zone can be observed via::
261 % cat /sys/device/system/memory/memoryXXX/valid_zones
266 There are various ways how system administrators can configure memory
267 hot(un)plug and interact with memory blocks, especially, to online them.
272 Some memory hot(un)plug properties can be configured or inspected via sysfs in::
274 /sys/devices/system/memory/
279 ``auto_online_blocks`` read-write: set or get the default state of new memory
286 See the ``state`` property of memory blocks for details.
287 ``block_size_bytes`` read-only: the size in bytes of a memory block.
288 ``probe`` write-only: add (probe) selected memory blocks manually
294 ``crash_hotplug`` read-only: when changes to the system memory map
295 occur due to hot un/plug of memory, this file contains
296 '1' if the kernel updates the kdump capture kernel memory
298 the kdump capture kernel memory map.
309 this functionality is not really related to memory hot(un)plug or actual
310 offlining of memory blocks.
315 Each memory block is represented as a memory block device that can be
316 onlined or offlined. All memory blocks have their device information located in
317 sysfs. Each present memory block is listed under
318 ``/sys/devices/system/memory`` as::
320 /sys/devices/system/memory/memoryXXX
322 where XXX is the memory block id; the number of digits is variable.
324 A present memory block indicates that some memory in the range is present;
325 however, a memory block might span memory holes. A memory block spanning memory
328 For example, assume 1 GiB memory block size. A device for a memory starting at
329 0x100000000 is ``/sys/device/system/memory/memory4``::
339 offlining and to observe the state of a memory block.
343 ``phys_index`` read-only: the memory block id (XXX).
344 ``removable`` read-only: legacy interface that indicated whether a memory
346 kernel return ``1`` if and only if it supports memory
349 offlining and to observe the state of a memory block.
356 zone for the memory block, such as ZONE_NORMAL.
366 For online memory blocks, ``DMA``, ``DMA32``, ``Normal``,
368 that memory provided by a memory block is managed by
369 multiple zones or spans multiple nodes; such memory blocks
373 For offline memory blocks, the first column shows the
374 zone the kernel would select when onlining the memory block
389 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9
393 /sys/devices/system/memory/memory9/node0 -> ../../node/node0
398 Some command line parameters affect memory hot(un)plug handling. The following
403 ``/sys/devices/system/memory/auto_online_blocks``.
407 onlining a memory block, unless other zones can be kept
431 ``memmap_on_memory`` read-write: Allocate memory for the memmap from
432 the added memory block itself. Even if enabled,
437 While allocating the memmap from the memory
438 block itself makes memory hotplug less likely
441 memory in a way that huge pages in bigger
443 memory.
445 With value "force" it could result in memory
447 example, if the memmap for a memory block
449 MiB, 1 MiB of hotplugged memory will be wasted.
457 automatic zone selection when onlining memory
461 online policy was configured and memory was
466 try keeping zones contiguous. If a memory block
474 try onlining memory blocks to ZONE_MOVABLE if
476 memory device details. With this policy, one
478 hotplugging a lot of memory later and still
486 memory might be exposed via the
487 firmware-provided memory map early during boot
495 As another example, as many memory blocks
498 special-casing units of memory blocks that can
502 change the zone of memory blocks dynamically
505 memory ratio in % for the ``auto-movable``
511 All accounting is based on present memory pages
513 memory device. Memory dedicated to the CMA
520 memory to ZONE_MOVABLE in many setups. The
525 Note that ZONE_NORMAL memory provided by one
526 memory device does not allow for more
527 ZONE_MOVABLE memory for a different memory
528 device. As one example, onlining memory of a
531 ZONE_MOVABLE automatically. In contrast, memory
534 ZONE_MOVABLE memory within *the same*
544 completely hotunpluggable, onlining the memory
554 ZONE_MOVABLE is an important mechanism for more reliable memory offlining.
559 Most kernel allocations are unmovable. Important examples include the memory
560 map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations
563 Most user space pages, such as anonymous memory, and page cache pages are
568 absolutely no guarantee whether a memory block can be offlined successfully.
575 might crash because it runs out of free memory for unmovable allocations,
576 although there is still plenty of free memory left in ZONE_MOVABLE.
579 are definitely impossible due to the overhead for the memory map.
586 CMA memory part of a kernel zone essentially behaves like memory in
601 - Having a lot of offline memory blocks. Even offline memory blocks consume
602 memory for metadata and page tables in the direct map; having a lot of offline
603 memory blocks is not a typical case, though.
615 lot of unmovable memory.
621 files or ZONE_DEVICE memory can be problematic, although only really relevant
622 in corner cases. When we manage a lot of user space memory that has been
623 swapped out or is served from a file/persistent memory/... we still need a lot
624 of page tables to manage that memory once user space accessed that memory.
626 - In certain DAX configurations the memory map for the device memory will be
629 - KASAN can have a significant memory overhead, for example, consuming 1/8th of
630 the total system memory size as (unmovable) tracking metadata.
634 ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside
637 even if there is plenty of free memory in ZONE_MOVABLE.
642 By default, all the memory configured at boot time is managed by the kernel
645 To enable ZONE_MOVABLE to include the memory present at boot and to control the
653 Even with ZONE_MOVABLE, there are some corner cases where offlining a memory
656 - Memory blocks with memory holes; this applies to memory blocks present during
657 boot and can apply to memory blocks hotplugged via the XEN balloon and the
660 - Mixed NUMA nodes and mixed zones within a single memory block prevent memory
661 offlining; this applies to memory blocks present during boot only.
663 - Special memory blocks prevented by the system from getting offlined. Examples
664 include any memory available during boot on arm64 or memory blocks spanning
665 the crashkernel area on s390x; this usually applies to memory blocks present
669 memory blocks present during boot only.
671 - Concurrent activity that operates on the same physical memory area, such as
674 - Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
679 for the vmemmap, because the system might not have free memory in the kernel
682 Users that depend on memory offlining to succeed for movable zones should
683 carefully consider whether the memory savings gained from this feature are
684 worth the risk of possibly not being able to offline memory in certain
687 Further, when running into out of memory situations while migrating pages, or
689 (-> BUG), memory offlining will keep retrying until it eventually succeeds.