1ee65728eSMike Rapoport==================== 2ee65728eSMike RapoportHigh Memory Handling 3ee65728eSMike Rapoport==================== 4ee65728eSMike Rapoport 5ee65728eSMike RapoportBy: Peter Zijlstra <a.p.zijlstra@chello.nl> 6ee65728eSMike Rapoport 7ee65728eSMike Rapoport.. contents:: :local: 8ee65728eSMike Rapoport 9ee65728eSMike RapoportWhat Is High Memory? 10ee65728eSMike Rapoport==================== 11ee65728eSMike Rapoport 12ee65728eSMike RapoportHigh memory (highmem) is used when the size of physical memory approaches or 13ee65728eSMike Rapoportexceeds the maximum size of virtual memory. At that point it becomes 14ee65728eSMike Rapoportimpossible for the kernel to keep all of the available physical memory mapped 15ee65728eSMike Rapoportat all times. This means the kernel needs to start using temporary mappings of 16ee65728eSMike Rapoportthe pieces of physical memory that it wants to access. 17ee65728eSMike Rapoport 18ee65728eSMike RapoportThe part of (physical) memory not covered by a permanent mapping is what we 19ee65728eSMike Rapoportrefer to as 'highmem'. There are various architecture dependent constraints on 20ee65728eSMike Rapoportwhere exactly that border lies. 21ee65728eSMike Rapoport 22ee65728eSMike RapoportIn the i386 arch, for example, we choose to map the kernel into every process's 23ee65728eSMike RapoportVM space so that we don't have to pay the full TLB invalidation costs for 24ee65728eSMike Rapoportkernel entry/exit. This means the available virtual memory space (4GiB on 25ee65728eSMike Rapoporti386) has to be divided between user and kernel space. 26ee65728eSMike Rapoport 27ee65728eSMike RapoportThe traditional split for architectures using this approach is 3:1, 3GiB for 28ee65728eSMike Rapoportuserspace and the top 1GiB for kernel space:: 29ee65728eSMike Rapoport 30ee65728eSMike Rapoport +--------+ 0xffffffff 31ee65728eSMike Rapoport | Kernel | 32ee65728eSMike Rapoport +--------+ 0xc0000000 33ee65728eSMike Rapoport | | 34ee65728eSMike Rapoport | User | 35ee65728eSMike Rapoport | | 36ee65728eSMike Rapoport +--------+ 0x00000000 37ee65728eSMike Rapoport 38ee65728eSMike RapoportThis means that the kernel can at most map 1GiB of physical memory at any one 39ee65728eSMike Rapoporttime, but because we need virtual address space for other things - including 40ee65728eSMike Rapoporttemporary maps to access the rest of the physical memory - the actual direct 41ee65728eSMike Rapoportmap will typically be less (usually around ~896MiB). 42ee65728eSMike Rapoport 43ee65728eSMike RapoportOther architectures that have mm context tagged TLBs can have separate kernel 44ee65728eSMike Rapoportand user maps. Some hardware (like some ARMs), however, have limited virtual 45ee65728eSMike Rapoportspace when they use mm context tags. 46ee65728eSMike Rapoport 47ee65728eSMike Rapoport 48ee65728eSMike RapoportTemporary Virtual Mappings 49ee65728eSMike Rapoport========================== 50ee65728eSMike Rapoport 51ee65728eSMike RapoportThe kernel contains several ways of creating temporary mappings. The following 52ee65728eSMike Rapoportlist shows them in order of preference of use. 53ee65728eSMike Rapoport 54ee65728eSMike Rapoport* kmap_local_page(), kmap_local_folio() - These functions are used to create 55ee65728eSMike Rapoport short term mappings. They can be invoked from any context (including 56ee65728eSMike Rapoport interrupts) but the mappings can only be used in the context which acquired 57ee65728eSMike Rapoport them. The only differences between them consist in the first taking a pointer 5892b64bd0SFabio M. De Francesco to a struct page and the second taking a pointer to struct folio and the byte 5992b64bd0SFabio M. De Francesco offset within the folio which identifies the page. 60ee65728eSMike Rapoport 61ee65728eSMike Rapoport These functions should always be used, whereas kmap_atomic() and kmap() have 62a9e9c939SFabio M. De Francesco been deprecated. 63a9e9c939SFabio M. De Francesco 64a9e9c939SFabio M. De Francesco These mappings are thread-local and CPU-local, meaning that the mapping 65a9e9c939SFabio M. De Francesco can only be accessed from within this thread and the thread is bound to the 66ee65728eSMike Rapoport CPU while the mapping is active. Although preemption is never disabled by 67ee65728eSMike Rapoport this function, the CPU can not be unplugged from the system via 68ee65728eSMike Rapoport CPU-hotplug until the mapping is disposed. 69ee65728eSMike Rapoport 70a9e9c939SFabio M. De Francesco It's valid to take pagefaults in a local kmap region, unless the context 71a9e9c939SFabio M. De Francesco in which the local mapping is acquired does not allow it for other reasons. 72a9e9c939SFabio M. De Francesco 73a9e9c939SFabio M. De Francesco As said, pagefaults and preemption are never disabled. There is no need to 74a9e9c939SFabio M. De Francesco disable preemption because, when context switches to a different task, the 75ee65728eSMike Rapoport maps of the outgoing task are saved and those of the incoming one are 76ee65728eSMike Rapoport restored. 77ee65728eSMike Rapoport 78516ea046SFabio M. De Francesco kmap_local_page(), as well as kmap_local_folio() always returns valid virtual 79516ea046SFabio M. De Francesco kernel addresses and it is assumed that kunmap_local() will never fail. 80516ea046SFabio M. De Francesco 81516ea046SFabio M. De Francesco On CONFIG_HIGHMEM=n kernels and for low memory pages they return the 82516ea046SFabio M. De Francesco virtual address of the direct mapping. Only real highmem pages are 83516ea046SFabio M. De Francesco temporarily mapped. Therefore, users may call a plain page_address() 84d0634a62SDeming Wang for pages which are known to not come from ZONE_HIGHMEM. However, it is 856b3afe2eSFabio M. De Francesco always safe to use kmap_local_{page,folio}() / kunmap_local(). 866b3afe2eSFabio M. De Francesco 876b3afe2eSFabio M. De Francesco While they are significantly faster than kmap(), for the highmem case they 886b3afe2eSFabio M. De Francesco come with restrictions about the pointers validity. Contrary to kmap() 896b3afe2eSFabio M. De Francesco mappings, the local mappings are only valid in the context of the caller 906b3afe2eSFabio M. De Francesco and cannot be handed to other contexts. This implies that users must 9184b86f60SFabio M. De Francesco be absolutely sure to keep the use of the return address local to the 9284b86f60SFabio M. De Francesco thread which mapped it. 9384b86f60SFabio M. De Francesco 9484b86f60SFabio M. De Francesco Most code can be designed to use thread local mappings. User should 9584b86f60SFabio M. De Francesco therefore try to design their code to avoid the use of kmap() by mapping 96ee65728eSMike Rapoport pages in the same thread the address will be used and prefer 97ee65728eSMike Rapoport kmap_local_page() or kmap_local_folio(). 98ee65728eSMike Rapoport 99ee65728eSMike Rapoport Nesting kmap_local_page() and kmap_atomic() mappings is allowed to a certain 100ee65728eSMike Rapoport extent (up to KMAP_TYPE_NR) but their invocations have to be strictly ordered 101ee65728eSMike Rapoport because the map implementation is stack based. See kmap_local_page() kdocs 10292b64bd0SFabio M. De Francesco (included in the "Functions" section) for details on how to manage nested 10392b64bd0SFabio M. De Francesco mappings. 10492b64bd0SFabio M. De Francesco 10592b64bd0SFabio M. De Francesco* kmap_atomic(). This function has been deprecated; use kmap_local_page(). 10692b64bd0SFabio M. De Francesco 10792b64bd0SFabio M. De Francesco NOTE: Conversions to kmap_local_page() must take care to follow the mapping 10892b64bd0SFabio M. De Francesco restrictions imposed on kmap_local_page(). Furthermore, the code between 10992b64bd0SFabio M. De Francesco calls to kmap_atomic() and kunmap_atomic() may implicitly depend on the side 11092b64bd0SFabio M. De Francesco effects of atomic mappings, i.e. disabling page faults or preemption, or both. 11192b64bd0SFabio M. De Francesco In that case, explicit calls to pagefault_disable() or preempt_disable() or 11292b64bd0SFabio M. De Francesco both must be made in conjunction with the use of kmap_local_page(). 11392b64bd0SFabio M. De Francesco 11492b64bd0SFabio M. De Francesco [Legacy documentation] 11592b64bd0SFabio M. De Francesco 11692b64bd0SFabio M. De Francesco This permits a very short duration mapping of a single page. Since the 117ee65728eSMike Rapoport mapping is restricted to the CPU that issued it, it performs well, but 118ee65728eSMike Rapoport the issuing task is therefore required to stay on that CPU until it has 119ee65728eSMike Rapoport finished, lest some other task displace its mappings. 120ee65728eSMike Rapoport 121ee65728eSMike Rapoport kmap_atomic() may also be used by interrupt contexts, since it does not 122ee65728eSMike Rapoport sleep and the callers too may not sleep until after kunmap_atomic() is 123ee65728eSMike Rapoport called. 124ee65728eSMike Rapoport 125ee65728eSMike Rapoport Each call of kmap_atomic() in the kernel creates a non-preemptible section 126ee65728eSMike Rapoport and disable pagefaults. This could be a source of unwanted latency. Therefore 127ee65728eSMike Rapoport users should prefer kmap_local_page() instead of kmap_atomic(). 12892b64bd0SFabio M. De Francesco 12992b64bd0SFabio M. De Francesco It is assumed that k[un]map_atomic() won't fail. 13092b64bd0SFabio M. De Francesco 13192b64bd0SFabio M. De Francesco* kmap(). This function has been deprecated; use kmap_local_page(). 13292b64bd0SFabio M. De Francesco 13392b64bd0SFabio M. De Francesco NOTE: Conversions to kmap_local_page() must take care to follow the mapping 13492b64bd0SFabio M. De Francesco restrictions imposed on kmap_local_page(). In particular, it is necessary to 13592b64bd0SFabio M. De Francesco make sure that the kernel virtual memory pointer is only valid in the thread 13692b64bd0SFabio M. De Francesco that obtained it. 13792b64bd0SFabio M. De Francesco 13892b64bd0SFabio M. De Francesco [Legacy documentation] 13992b64bd0SFabio M. De Francesco 14092b64bd0SFabio M. De Francesco This should be used to make short duration mapping of a single page with no 14192b64bd0SFabio M. De Francesco restrictions on preemption or migration. It comes with an overhead as mapping 142ee65728eSMike Rapoport space is restricted and protected by a global lock for synchronization. When 143ee65728eSMike Rapoport mapping is no longer needed, the address that the page was mapped to must be 144ee65728eSMike Rapoport released with kunmap(). 145ee65728eSMike Rapoport 146ee65728eSMike Rapoport Mapping changes must be propagated across all the CPUs. kmap() also 147ee65728eSMike Rapoport requires global TLB invalidation when the kmap's pool wraps and it might 148ee65728eSMike Rapoport block when the mapping space is fully utilized until a slot becomes 149ee65728eSMike Rapoport available. Therefore, kmap() is only callable from preemptible context. 150ee65728eSMike Rapoport 151ee65728eSMike Rapoport All the above work is necessary if a mapping must last for a relatively 152ee65728eSMike Rapoport long time but the bulk of high-memory mappings in the kernel are 153ee65728eSMike Rapoport short-lived and only used in one place. This means that the cost of 154ee65728eSMike Rapoport kmap() is mostly wasted in such cases. kmap() was not intended for long 155ee65728eSMike Rapoport term mappings but it has morphed in that direction and its use is 156ee65728eSMike Rapoport strongly discouraged in newer code and the set of the preceding functions 157ee65728eSMike Rapoport should be preferred. 158ee65728eSMike Rapoport 159ee65728eSMike Rapoport On 64-bit systems, calls to kmap_local_page(), kmap_atomic() and kmap() have 160ee65728eSMike Rapoport no real work to do because a 64-bit address space is more than sufficient to 161ee65728eSMike Rapoport address all the physical memory whose pages are permanently mapped. 162ee65728eSMike Rapoport 163ee65728eSMike Rapoport* vmap(). This can be used to make a long duration mapping of multiple 164ee65728eSMike Rapoport physical pages into a contiguous virtual space. It needs global 165ee65728eSMike Rapoport synchronization to unmap. 166ee65728eSMike Rapoport 167ee65728eSMike Rapoport 168ee65728eSMike RapoportCost of Temporary Mappings 169ee65728eSMike Rapoport========================== 170ee65728eSMike Rapoport 171ee65728eSMike RapoportThe cost of creating temporary mappings can be quite high. The arch has to 172ee65728eSMike Rapoportmanipulate the kernel's page tables, the data TLB and/or the MMU's registers. 173ee65728eSMike Rapoport 174ee65728eSMike RapoportIf CONFIG_HIGHMEM is not set, then the kernel will try and create a mapping 175ee65728eSMike Rapoportsimply with a bit of arithmetic that will convert the page struct address into 176ee65728eSMike Rapoporta pointer to the page contents rather than juggling mappings about. In such a 177ee65728eSMike Rapoportcase, the unmap operation may be a null operation. 178ee65728eSMike Rapoport 179ee65728eSMike RapoportIf CONFIG_MMU is not set, then there can be no temporary mappings and no 180ee65728eSMike Rapoporthighmem. In such a case, the arithmetic approach will also be used. 181ee65728eSMike Rapoport 182ee65728eSMike Rapoport 183ee65728eSMike Rapoporti386 PAE 184ee65728eSMike Rapoport======== 185ee65728eSMike Rapoport 186ee65728eSMike RapoportThe i386 arch, under some circumstances, will permit you to stick up to 64GiB 187ee65728eSMike Rapoportof RAM into your 32-bit machine. This has a number of consequences: 188ee65728eSMike Rapoport 189ee65728eSMike Rapoport* Linux needs a page-frame structure for each page in the system and the 190ee65728eSMike Rapoport pageframes need to live in the permanent mapping, which means: 191ee65728eSMike Rapoport 192ee65728eSMike Rapoport* you can have 896M/sizeof(struct page) page-frames at most; with struct 193ee65728eSMike Rapoport page being 32-bytes that would end up being something in the order of 112G 194ee65728eSMike Rapoport worth of pages; the kernel, however, needs to store more than just 195ee65728eSMike Rapoport page-frames in that memory... 196ee65728eSMike Rapoport 197ee65728eSMike Rapoport* PAE makes your page tables larger - which slows the system down as more 198ee65728eSMike Rapoport data has to be accessed to traverse in TLB fills and the like. One 199ee65728eSMike Rapoport advantage is that PAE has more PTE bits and can provide advanced features 200ee65728eSMike Rapoport like NX and PAT. 201ee65728eSMike Rapoport 202ee65728eSMike RapoportThe general recommendation is that you don't use more than 8GiB on a 32-bit 203ee65728eSMike Rapoportmachine - although more might work for you and your workload, you're pretty 204ee65728eSMike Rapoportmuch on your own - don't expect kernel developers to really care much if things 205ee65728eSMike Rapoportcome apart. 206ee65728eSMike Rapoport 207ee65728eSMike Rapoport 208ee65728eSMike RapoportFunctions 209*61ff748bSMatthew Wilcox (Oracle)========= 210ee65728eSMike Rapoport 211.. kernel-doc:: include/linux/highmem.h 212.. kernel-doc:: mm/highmem.c 213.. kernel-doc:: include/linux/highmem-internal.h 214