1.. _admin_guide_memory_hotplug: 2 3============== 4Memory Hotplug 5============== 6 7:Created: Jul 28 2007 8:Updated: Add some details about locking internals: Aug 20 2018 9 10This document is about memory hotplug including how-to-use and current status. 11Because Memory Hotplug is still under development, contents of this text will 12be changed often. 13 14.. contents:: :local: 15 16.. note:: 17 18 (1) x86_64's has special implementation for memory hotplug. 19 This text does not describe it. 20 (2) This text assumes that sysfs is mounted at ``/sys``. 21 22 23Introduction 24============ 25 26Purpose of memory hotplug 27------------------------- 28 29Memory Hotplug allows users to increase/decrease the amount of memory. 30Generally, there are two purposes. 31 32(A) For changing the amount of memory. 33 This is to allow a feature like capacity on demand. 34(B) For installing/removing DIMMs or NUMA-nodes physically. 35 This is to exchange DIMMs/NUMA-nodes, reduce power consumption, etc. 36 37(A) is required by highly virtualized environments and (B) is required by 38hardware which supports memory power management. 39 40Linux memory hotplug is designed for both purpose. 41 42Phases of memory hotplug 43------------------------ 44 45There are 2 phases in Memory Hotplug: 46 47 1) Physical Memory Hotplug phase 48 2) Logical Memory Hotplug phase. 49 50The First phase is to communicate hardware/firmware and make/erase 51environment for hotplugged memory. Basically, this phase is necessary 52for the purpose (B), but this is good phase for communication between 53highly virtualized environments too. 54 55When memory is hotplugged, the kernel recognizes new memory, makes new memory 56management tables, and makes sysfs files for new memory's operation. 57 58If firmware supports notification of connection of new memory to OS, 59this phase is triggered automatically. ACPI can notify this event. If not, 60"probe" operation by system administration is used instead. 61(see :ref:`memory_hotplug_physical_mem`). 62 63Logical Memory Hotplug phase is to change memory state into 64available/unavailable for users. Amount of memory from user's view is 65changed by this phase. The kernel makes all memory in it as free pages 66when a memory range is available. 67 68In this document, this phase is described as online/offline. 69 70Logical Memory Hotplug phase is triggered by write of sysfs file by system 71administrator. For the hot-add case, it must be executed after Physical Hotplug 72phase by hand. 73(However, if you writes udev's hotplug scripts for memory hotplug, these 74phases can be execute in seamless way.) 75 76Unit of Memory online/offline operation 77--------------------------------------- 78 79Memory hotplug uses SPARSEMEM memory model which allows memory to be divided 80into chunks of the same size. These chunks are called "sections". The size of 81a memory section is architecture dependent. For example, power uses 16MiB, ia64 82uses 1GiB. 83 84Memory sections are combined into chunks referred to as "memory blocks". The 85size of a memory block is architecture dependent and represents the logical 86unit upon which memory online/offline operations are to be performed. The 87default size of a memory block is the same as memory section size unless an 88architecture specifies otherwise. (see :ref:`memory_hotplug_sysfs_files`.) 89 90To determine the size (in bytes) of a memory block please read this file:: 91 92 /sys/devices/system/memory/block_size_bytes 93 94Kernel Configuration 95==================== 96 97To use memory hotplug feature, kernel must be compiled with following 98config options. 99 100- For all memory hotplug: 101 - Memory model -> Sparse Memory (``CONFIG_SPARSEMEM``) 102 - Allow for memory hot-add (``CONFIG_MEMORY_HOTPLUG``) 103 104- To enable memory removal, the following are also necessary: 105 - Allow for memory hot remove (``CONFIG_MEMORY_HOTREMOVE``) 106 - Page Migration (``CONFIG_MIGRATION``) 107 108- For ACPI memory hotplug, the following are also necessary: 109 - Memory hotplug (under ACPI Support menu) (``CONFIG_ACPI_HOTPLUG_MEMORY``) 110 - This option can be kernel module. 111 112- As a related configuration, if your box has a feature of NUMA-node hotplug 113 via ACPI, then this option is necessary too. 114 115 - ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu) 116 (``CONFIG_ACPI_CONTAINER``). 117 118 This option can be kernel module too. 119 120 121.. _memory_hotplug_sysfs_files: 122 123sysfs files for memory hotplug 124============================== 125 126All memory blocks have their device information in sysfs. Each memory block 127is described under ``/sys/devices/system/memory`` as:: 128 129 /sys/devices/system/memory/memoryXXX 130 131where XXX is the memory block id. 132 133For the memory block covered by the sysfs directory. It is expected that all 134memory sections in this range are present and no memory holes exist in the 135range. Currently there is no way to determine if there is a memory hole, but 136the existence of one should not affect the hotplug capabilities of the memory 137block. 138 139For example, assume 1GiB memory block size. A device for a memory starting at 1400x100000000 is ``/sys/device/system/memory/memory4``:: 141 142 (0x100000000 / 1Gib = 4) 143 144This device covers address range [0x100000000 ... 0x140000000) 145 146Under each memory block, you can see 5 files: 147 148- ``/sys/devices/system/memory/memoryXXX/phys_index`` 149- ``/sys/devices/system/memory/memoryXXX/phys_device`` 150- ``/sys/devices/system/memory/memoryXXX/state`` 151- ``/sys/devices/system/memory/memoryXXX/removable`` 152- ``/sys/devices/system/memory/memoryXXX/valid_zones`` 153 154=================== ============================================================ 155``phys_index`` read-only and contains memory block id, same as XXX. 156``state`` read-write 157 158 - at read: contains online/offline state of memory. 159 - at write: user can specify "online_kernel", 160 161 "online_movable", "online", "offline" command 162 which will be performed on all sections in the block. 163``phys_device`` read-only: legacy interface only ever used on s390x to 164 expose the covered storage increment. 165``removable`` read-only: legacy interface that indicated whether a memory 166 block was likely to be offlineable or not. Newer kernel 167 versions return "1" if and only if the kernel supports 168 memory offlining. 169``valid_zones`` read-only: designed to show by which zone memory provided by 170 a memory block is managed, and to show by which zone memory 171 provided by an offline memory block could be managed when 172 onlining. 173 174 The first column shows it`s default zone. 175 176 "memory6/valid_zones: Normal Movable" shows this memoryblock 177 can be onlined to ZONE_NORMAL by default and to ZONE_MOVABLE 178 by online_movable. 179 180 "memory7/valid_zones: Movable Normal" shows this memoryblock 181 can be onlined to ZONE_MOVABLE by default and to ZONE_NORMAL 182 by online_kernel. 183=================== ============================================================ 184 185.. note:: 186 187 These directories/files appear after physical memory hotplug phase. 188 189If CONFIG_NUMA is enabled the memoryXXX/ directories can also be accessed 190via symbolic links located in the ``/sys/devices/system/node/node*`` directories. 191 192For example:: 193 194 /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 195 196A backlink will also be created:: 197 198 /sys/devices/system/memory/memory9/node0 -> ../../node/node0 199 200.. _memory_hotplug_physical_mem: 201 202Physical memory hot-add phase 203============================= 204 205Hardware(Firmware) Support 206-------------------------- 207 208On x86_64/ia64 platform, memory hotplug by ACPI is supported. 209 210In general, the firmware (ACPI) which supports memory hotplug defines 211memory class object of _HID "PNP0C80". When a notify is asserted to PNP0C80, 212Linux's ACPI handler does hot-add memory to the system and calls a hotplug udev 213script. This will be done automatically. 214 215But scripts for memory hotplug are not contained in generic udev package(now). 216You may have to write it by yourself or online/offline memory by hand. 217Please see :ref:`memory_hotplug_how_to_online_memory` and 218:ref:`memory_hotplug_how_to_offline_memory`. 219 220If firmware supports NUMA-node hotplug, and defines an object _HID "ACPI0004", 221"PNP0A05", or "PNP0A06", notification is asserted to it, and ACPI handler 222calls hotplug code for all of objects which are defined in it. 223If memory device is found, memory hotplug code will be called. 224 225Notify memory hot-add event by hand 226----------------------------------- 227 228On some architectures, the firmware may not notify the kernel of a memory 229hotplug event. Therefore, the memory "probe" interface is supported to 230explicitly notify the kernel. This interface depends on 231CONFIG_ARCH_MEMORY_PROBE and can be configured on powerpc, sh, and x86 232if hotplug is supported, although for x86 this should be handled by ACPI 233notification. 234 235Probe interface is located at:: 236 237 /sys/devices/system/memory/probe 238 239You can tell the physical address of new memory to the kernel by:: 240 241 % echo start_address_of_new_memory > /sys/devices/system/memory/probe 242 243Then, [start_address_of_new_memory, start_address_of_new_memory + 244memory_block_size] memory range is hot-added. In this case, hotplug script is 245not called (in current implementation). You'll have to online memory by 246yourself. Please see :ref:`memory_hotplug_how_to_online_memory`. 247 248Logical Memory hot-add phase 249============================ 250 251State of memory 252--------------- 253 254To see (online/offline) state of a memory block, read 'state' file:: 255 256 % cat /sys/device/system/memory/memoryXXX/state 257 258 259- If the memory block is online, you'll read "online". 260- If the memory block is offline, you'll read "offline". 261 262 263.. _memory_hotplug_how_to_online_memory: 264 265How to online memory 266-------------------- 267 268When the memory is hot-added, the kernel decides whether or not to "online" 269it according to the policy which can be read from "auto_online_blocks" file:: 270 271 % cat /sys/devices/system/memory/auto_online_blocks 272 273The default depends on the CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config 274option. If it is disabled the default is "offline" which means the newly added 275memory is not in a ready-to-use state and you have to "online" the newly added 276memory blocks manually. Automatic onlining can be requested by writing "online" 277to "auto_online_blocks" file:: 278 279 % echo online > /sys/devices/system/memory/auto_online_blocks 280 281This sets a global policy and impacts all memory blocks that will subsequently 282be hotplugged. Currently offline blocks keep their state. It is possible, under 283certain circumstances, that some memory blocks will be added but will fail to 284online. User space tools can check their "state" files 285(``/sys/devices/system/memory/memoryXXX/state``) and try to online them manually. 286 287If the automatic onlining wasn't requested, failed, or some memory block was 288offlined it is possible to change the individual block's state by writing to the 289"state" file:: 290 291 % echo online > /sys/devices/system/memory/memoryXXX/state 292 293This onlining will not change the ZONE type of the target memory block, 294If the memory block doesn't belong to any zone an appropriate kernel zone 295(usually ZONE_NORMAL) will be used unless movable_node kernel command line 296option is specified when ZONE_MOVABLE will be used. 297 298You can explicitly request to associate it with ZONE_MOVABLE by:: 299 300 % echo online_movable > /sys/devices/system/memory/memoryXXX/state 301 302.. note:: current limit: this memory block must be adjacent to ZONE_MOVABLE 303 304Or you can explicitly request a kernel zone (usually ZONE_NORMAL) by:: 305 306 % echo online_kernel > /sys/devices/system/memory/memoryXXX/state 307 308.. note:: current limit: this memory block must be adjacent to ZONE_NORMAL 309 310An explicit zone onlining can fail (e.g. when the range is already within 311and existing and incompatible zone already). 312 313After this, memory block XXX's state will be 'online' and the amount of 314available memory will be increased. 315 316This may be changed in future. 317 318Logical memory remove 319===================== 320 321Memory offline and ZONE_MOVABLE 322------------------------------- 323 324Memory offlining is more complicated than memory online. Because memory offline 325has to make the whole memory block be unused, memory offline can fail if 326the memory block includes memory which cannot be freed. 327 328In general, memory offline can use 2 techniques. 329 330(1) reclaim and free all memory in the memory block. 331(2) migrate all pages in the memory block. 332 333In the current implementation, Linux's memory offline uses method (2), freeing 334all pages in the memory block by page migration. But not all pages are 335migratable. Under current Linux, migratable pages are anonymous pages and 336page caches. For offlining a memory block by migration, the kernel has to 337guarantee that the memory block contains only migratable pages. 338 339Now, a boot option for making a memory block which consists of migratable pages 340is supported. By specifying "kernelcore=" or "movablecore=" boot option, you can 341create ZONE_MOVABLE...a zone which is just used for movable pages. 342(See also Documentation/admin-guide/kernel-parameters.rst) 343 344Assume the system has "TOTAL" amount of memory at boot time, this boot option 345creates ZONE_MOVABLE as following. 346 3471) When kernelcore=YYYY boot option is used, 348 Size of memory not for movable pages (not for offline) is YYYY. 349 Size of memory for movable pages (for offline) is TOTAL-YYYY. 350 3512) When movablecore=ZZZZ boot option is used, 352 Size of memory not for movable pages (not for offline) is TOTAL - ZZZZ. 353 Size of memory for movable pages (for offline) is ZZZZ. 354 355.. note:: 356 357 Unfortunately, there is no information to show which memory block belongs 358 to ZONE_MOVABLE. This is TBD. 359 360.. note:: 361 Techniques that rely on long-term pinnings of memory (especially, RDMA and 362 vfio) are fundamentally problematic with ZONE_MOVABLE and, therefore, memory 363 hot remove. Pinned pages cannot reside on ZONE_MOVABLE, to guarantee that 364 memory can still get hot removed - be aware that pinning can fail even if 365 there is plenty of free memory in ZONE_MOVABLE. In addition, using 366 ZONE_MOVABLE might make page pinning more expensive, because pages have to be 367 migrated off that zone first. 368 369.. _memory_hotplug_how_to_offline_memory: 370 371How to offline memory 372--------------------- 373 374You can offline a memory block by using the same sysfs interface that was used 375in memory onlining:: 376 377 % echo offline > /sys/devices/system/memory/memoryXXX/state 378 379If offline succeeds, the state of the memory block is changed to be "offline". 380If it fails, some error core (like -EBUSY) will be returned by the kernel. 381Even if a memory block does not belong to ZONE_MOVABLE, you can try to offline 382it. If it doesn't contain 'unmovable' memory, you'll get success. 383 384A memory block under ZONE_MOVABLE is considered to be able to be offlined 385easily. But under some busy state, it may return -EBUSY. Even if a memory 386block cannot be offlined due to -EBUSY, you can retry offlining it and may be 387able to offline it (or not). (For example, a page is referred to by some kernel 388internal call and released soon.) 389 390Consideration: 391 Memory hotplug's design direction is to make the possibility of memory 392 offlining higher and to guarantee unplugging memory under any situation. But 393 it needs more work. Returning -EBUSY under some situation may be good because 394 the user can decide to retry more or not by himself. Currently, memory 395 offlining code does some amount of retry with 120 seconds timeout. 396 397Physical memory remove 398====================== 399 400Need more implementation yet.... 401 - Notification completion of remove works by OS to firmware. 402 - Guard from remove if not yet. 403 404 405Locking Internals 406================= 407 408When adding/removing memory that uses memory block devices (i.e. ordinary RAM), 409the device_hotplug_lock should be held to: 410 411- synchronize against online/offline requests (e.g. via sysfs). This way, memory 412 block devices can only be accessed (.online/.state attributes) by user 413 space once memory has been fully added. And when removing memory, we 414 know nobody is in critical sections. 415- synchronize against CPU hotplug and similar (e.g. relevant for ACPI and PPC) 416 417Especially, there is a possible lock inversion that is avoided using 418device_hotplug_lock when adding memory and user space tries to online that 419memory faster than expected: 420 421- device_online() will first take the device_lock(), followed by 422 mem_hotplug_lock 423- add_memory_resource() will first take the mem_hotplug_lock, followed by 424 the device_lock() (while creating the devices, during bus_add_device()). 425 426As the device is visible to user space before taking the device_lock(), this 427can result in a lock inversion. 428 429onlining/offlining of memory should be done via device_online()/ 430device_offline() - to make sure it is properly synchronized to actions 431via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) 432 433When adding/removing/onlining/offlining memory or adding/removing 434heterogeneous/device memory, we should always hold the mem_hotplug_lock in 435write mode to serialise memory hotplug (e.g. access to global/zone 436variables). 437 438In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read 439mode allows for a quite efficient get_online_mems/put_online_mems 440implementation, so code accessing memory can protect from that memory 441vanishing. 442 443 444Future Work 445=========== 446 447 - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like 448 sysctl or new control file. 449 - showing memory block and physical device relationship. 450 - test and make it better memory offlining. 451 - support HugeTLB page migration and offlining. 452 - memmap removing at memory offline. 453 - physical remove memory. 454