1.. _admin_guide_transhuge:
2
3============================
4Transparent Hugepage Support
5============================
6
7Objective
8=========
9
10Performance critical computing applications dealing with large memory
11working sets are already running on top of libhugetlbfs and in turn
12hugetlbfs. Transparent HugePage Support (THP) is an alternative mean of
13using huge pages for the backing of virtual memory with huge pages
14that supports the automatic promotion and demotion of page sizes and
15without the shortcomings of hugetlbfs.
16
17Currently THP only works for anonymous memory mappings and tmpfs/shmem.
18But in the future it can expand to other filesystems.
19
20.. note::
21   in the examples below we presume that the basic page size is 4K and
22   the huge page size is 2M, although the actual numbers may vary
23   depending on the CPU architecture.
24
25The reason applications are running faster is because of two
26factors. The first factor is almost completely irrelevant and it's not
27of significant interest because it'll also have the downside of
28requiring larger clear-page copy-page in page faults which is a
29potentially negative effect. The first factor consists in taking a
30single page fault for each 2M virtual region touched by userland (so
31reducing the enter/exit kernel frequency by a 512 times factor). This
32only matters the first time the memory is accessed for the lifetime of
33a memory mapping. The second long lasting and much more important
34factor will affect all subsequent accesses to the memory for the whole
35runtime of the application. The second factor consist of two
36components:
37
381) the TLB miss will run faster (especially with virtualization using
39   nested pagetables but almost always also on bare metal without
40   virtualization)
41
422) a single TLB entry will be mapping a much larger amount of virtual
43   memory in turn reducing the number of TLB misses. With
44   virtualization and nested pagetables the TLB can be mapped of
45   larger size only if both KVM and the Linux guest are using
46   hugepages but a significant speedup already happens if only one of
47   the two is using hugepages just because of the fact the TLB miss is
48   going to run faster.
49
50THP can be enabled system wide or restricted to certain tasks or even
51memory ranges inside task's address space. Unless THP is completely
52disabled, there is ``khugepaged`` daemon that scans memory and
53collapses sequences of basic pages into huge pages.
54
55The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
56interface and using madvise(2) and prctl(2) system calls.
57
58Transparent Hugepage Support maximizes the usefulness of free memory
59if compared to the reservation approach of hugetlbfs by allowing all
60unused memory to be used as cache or other movable (or even unmovable
61entities). It doesn't require reservation to prevent hugepage
62allocation failures to be noticeable from userland. It allows paging
63and all other advanced VM features to be available on the
64hugepages. It requires no modifications for applications to take
65advantage of it.
66
67Applications however can be further optimized to take advantage of
68this feature, like for example they've been optimized before to avoid
69a flood of mmap system calls for every malloc(4k). Optimizing userland
70is by far not mandatory and khugepaged already can take care of long
71lived page allocations even for hugepage unaware applications that
72deals with large amounts of memory.
73
74In certain cases when hugepages are enabled system wide, application
75may end up allocating more memory resources. An application may mmap a
76large region but only touch 1 byte of it, in that case a 2M page might
77be allocated instead of a 4k page for no good. This is why it's
78possible to disable hugepages system-wide and to only have them inside
79MADV_HUGEPAGE madvise regions.
80
81Embedded systems should enable hugepages only inside madvise regions
82to eliminate any risk of wasting any precious byte of memory and to
83only run faster.
84
85Applications that gets a lot of benefit from hugepages and that don't
86risk to lose memory by using hugepages, should use
87madvise(MADV_HUGEPAGE) on their critical mmapped regions.
88
89.. _thp_sysfs:
90
91sysfs
92=====
93
94Global THP controls
95-------------------
96
97Transparent Hugepage Support for anonymous memory can be entirely disabled
98(mostly for debugging purposes) or only enabled inside MADV_HUGEPAGE
99regions (to avoid the risk of consuming more memory resources) or enabled
100system wide. This can be achieved with one of::
101
102	echo always >/sys/kernel/mm/transparent_hugepage/enabled
103	echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
104	echo never >/sys/kernel/mm/transparent_hugepage/enabled
105
106It's also possible to limit defrag efforts in the VM to generate
107anonymous hugepages in case they're not immediately free to madvise
108regions or to never try to defrag memory and simply fallback to regular
109pages unless hugepages are immediately available. Clearly if we spend CPU
110time to defrag memory, we would expect to gain even more by the fact we
111use hugepages later instead of regular pages. This isn't always
112guaranteed, but it may be more likely in case the allocation is for a
113MADV_HUGEPAGE region.
114
115::
116
117	echo always >/sys/kernel/mm/transparent_hugepage/defrag
118	echo defer >/sys/kernel/mm/transparent_hugepage/defrag
119	echo defer+madvise >/sys/kernel/mm/transparent_hugepage/defrag
120	echo madvise >/sys/kernel/mm/transparent_hugepage/defrag
121	echo never >/sys/kernel/mm/transparent_hugepage/defrag
122
123always
124	means that an application requesting THP will stall on
125	allocation failure and directly reclaim pages and compact
126	memory in an effort to allocate a THP immediately. This may be
127	desirable for virtual machines that benefit heavily from THP
128	use and are willing to delay the VM start to utilise them.
129
130defer
131	means that an application will wake kswapd in the background
132	to reclaim pages and wake kcompactd to compact memory so that
133	THP is available in the near future. It's the responsibility
134	of khugepaged to then install the THP pages later.
135
136defer+madvise
137	will enter direct reclaim and compaction like ``always``, but
138	only for regions that have used madvise(MADV_HUGEPAGE); all
139	other regions will wake kswapd in the background to reclaim
140	pages and wake kcompactd to compact memory so that THP is
141	available in the near future.
142
143madvise
144	will enter direct reclaim like ``always`` but only for regions
145	that are have used madvise(MADV_HUGEPAGE). This is the default
146	behaviour.
147
148never
149	should be self-explanatory.
150
151By default kernel tries to use huge zero page on read page fault to
152anonymous mapping. It's possible to disable huge zero page by writing 0
153or enable it back by writing 1::
154
155	echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page
156	echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
157
158Some userspace (such as a test program, or an optimized memory allocation
159library) may want to know the size (in bytes) of a transparent hugepage::
160
161	cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
162
163khugepaged will be automatically started when
164transparent_hugepage/enabled is set to "always" or "madvise, and it'll
165be automatically shutdown if it's set to "never".
166
167Khugepaged controls
168-------------------
169
170khugepaged runs usually at low frequency so while one may not want to
171invoke defrag algorithms synchronously during the page faults, it
172should be worth invoking defrag at least in khugepaged. However it's
173also possible to disable defrag in khugepaged by writing 0 or enable
174defrag in khugepaged by writing 1::
175
176	echo 0 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
177	echo 1 >/sys/kernel/mm/transparent_hugepage/khugepaged/defrag
178
179You can also control how many pages khugepaged should scan at each
180pass::
181
182	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan
183
184and how many milliseconds to wait in khugepaged between each pass (you
185can set this to 0 to run khugepaged at 100% utilization of one core)::
186
187	/sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs
188
189and how many milliseconds to wait in khugepaged if there's an hugepage
190allocation failure to throttle the next allocation attempt::
191
192	/sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs
193
194The khugepaged progress can be seen in the number of pages collapsed::
195
196	/sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed
197
198for each pass::
199
200	/sys/kernel/mm/transparent_hugepage/khugepaged/full_scans
201
202``max_ptes_none`` specifies how many extra small pages (that are
203not already mapped) can be allocated when collapsing a group
204of small pages into one large page::
205
206	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
207
208A higher value leads to use additional memory for programs.
209A lower value leads to gain less thp performance. Value of
210max_ptes_none can waste cpu time very little, you can
211ignore it.
212
213``max_ptes_swap`` specifies how many pages can be brought in from
214swap when collapsing a group of pages into a transparent huge page::
215
216	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_swap
217
218A higher value can cause excessive swap IO and waste
219memory. A lower value can prevent THPs from being
220collapsed, resulting fewer pages being collapsed into
221THPs, and lower memory access performance.
222
223``max_ptes_shared`` specifies how many pages can be shared across multiple
224processes. Exceeding the number would block the collapse::
225
226	/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared
227
228A higher value may increase memory footprint for some workloads.
229
230Boot parameter
231==============
232
233You can change the sysfs boot time defaults of Transparent Hugepage
234Support by passing the parameter ``transparent_hugepage=always`` or
235``transparent_hugepage=madvise`` or ``transparent_hugepage=never``
236to the kernel command line.
237
238Hugepages in tmpfs/shmem
239========================
240
241You can control hugepage allocation policy in tmpfs with mount option
242``huge=``. It can have following values:
243
244always
245    Attempt to allocate huge pages every time we need a new page;
246
247never
248    Do not allocate huge pages;
249
250within_size
251    Only allocate huge page if it will be fully within i_size.
252    Also respect fadvise()/madvise() hints;
253
254advise
255    Only allocate huge pages if requested with fadvise()/madvise();
256
257The default policy is ``never``.
258
259``mount -o remount,huge= /mountpoint`` works fine after mount: remounting
260``huge=never`` will not attempt to break up huge pages at all, just stop more
261from being allocated.
262
263There's also sysfs knob to control hugepage allocation policy for internal
264shmem mount: /sys/kernel/mm/transparent_hugepage/shmem_enabled. The mount
265is used for SysV SHM, memfds, shared anonymous mmaps (of /dev/zero or
266MAP_ANONYMOUS), GPU drivers' DRM objects, Ashmem.
267
268In addition to policies listed above, shmem_enabled allows two further
269values:
270
271deny
272    For use in emergencies, to force the huge option off from
273    all mounts;
274force
275    Force the huge option on for all - very useful for testing;
276
277Need of application restart
278===========================
279
280The transparent_hugepage/enabled values and tmpfs mount option only affect
281future behavior. So to make them effective you need to restart any
282application that could have been using hugepages. This also applies to the
283regions registered in khugepaged.
284
285Monitoring usage
286================
287
288The number of anonymous transparent huge pages currently used by the
289system is available by reading the AnonHugePages field in ``/proc/meminfo``.
290To identify what applications are using anonymous transparent huge pages,
291it is necessary to read ``/proc/PID/smaps`` and count the AnonHugePages fields
292for each mapping.
293
294The number of file transparent huge pages mapped to userspace is available
295by reading ShmemPmdMapped and ShmemHugePages fields in ``/proc/meminfo``.
296To identify what applications are mapping file transparent huge pages, it
297is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields
298for each mapping.
299
300Note that reading the smaps file is expensive and reading it
301frequently will incur overhead.
302
303There are a number of counters in ``/proc/vmstat`` that may be used to
304monitor how successfully the system is providing huge pages for use.
305
306thp_fault_alloc
307	is incremented every time a huge page is successfully
308	allocated to handle a page fault.
309
310thp_collapse_alloc
311	is incremented by khugepaged when it has found
312	a range of pages to collapse into one huge page and has
313	successfully allocated a new huge page to store the data.
314
315thp_fault_fallback
316	is incremented if a page fault fails to allocate
317	a huge page and instead falls back to using small pages.
318
319thp_fault_fallback_charge
320	is incremented if a page fault fails to charge a huge page and
321	instead falls back to using small pages even though the
322	allocation was successful.
323
324thp_collapse_alloc_failed
325	is incremented if khugepaged found a range
326	of pages that should be collapsed into one huge page but failed
327	the allocation.
328
329thp_file_alloc
330	is incremented every time a file huge page is successfully
331	allocated.
332
333thp_file_fallback
334	is incremented if a file huge page is attempted to be allocated
335	but fails and instead falls back to using small pages.
336
337thp_file_fallback_charge
338	is incremented if a file huge page cannot be charged and instead
339	falls back to using small pages even though the allocation was
340	successful.
341
342thp_file_mapped
343	is incremented every time a file huge page is mapped into
344	user address space.
345
346thp_split_page
347	is incremented every time a huge page is split into base
348	pages. This can happen for a variety of reasons but a common
349	reason is that a huge page is old and is being reclaimed.
350	This action implies splitting all PMD the page mapped with.
351
352thp_split_page_failed
353	is incremented if kernel fails to split huge
354	page. This can happen if the page was pinned by somebody.
355
356thp_deferred_split_page
357	is incremented when a huge page is put onto split
358	queue. This happens when a huge page is partially unmapped and
359	splitting it would free up some memory. Pages on split queue are
360	going to be split under memory pressure.
361
362thp_split_pmd
363	is incremented every time a PMD split into table of PTEs.
364	This can happen, for instance, when application calls mprotect() or
365	munmap() on part of huge page. It doesn't split huge page, only
366	page table entry.
367
368thp_zero_page_alloc
369	is incremented every time a huge zero page is
370	successfully allocated. It includes allocations which where
371	dropped due race with other allocation. Note, it doesn't count
372	every map of the huge zero page, only its allocation.
373
374thp_zero_page_alloc_failed
375	is incremented if kernel fails to allocate
376	huge zero page and falls back to using small pages.
377
378thp_swpout
379	is incremented every time a huge page is swapout in one
380	piece without splitting.
381
382thp_swpout_fallback
383	is incremented if a huge page has to be split before swapout.
384	Usually because failed to allocate some continuous swap space
385	for the huge page.
386
387As the system ages, allocating huge pages may be expensive as the
388system uses memory compaction to copy data around memory to free a
389huge page for use. There are some counters in ``/proc/vmstat`` to help
390monitor this overhead.
391
392compact_stall
393	is incremented every time a process stalls to run
394	memory compaction so that a huge page is free for use.
395
396compact_success
397	is incremented if the system compacted memory and
398	freed a huge page for use.
399
400compact_fail
401	is incremented if the system tries to compact memory
402	but failed.
403
404It is possible to establish how long the stalls were using the function
405tracer to record how long was spent in __alloc_pages_nodemask and
406using the mm_page_alloc tracepoint to identify which allocations were
407for huge pages.
408
409Optimizing the applications
410===========================
411
412To be guaranteed that the kernel will map a 2M page immediately in any
413memory region, the mmap region has to be hugepage naturally
414aligned. posix_memalign() can provide that guarantee.
415
416Hugetlbfs
417=========
418
419You can use hugetlbfs on a kernel that has transparent hugepage
420support enabled just fine as always. No difference can be noted in
421hugetlbfs other than there will be less overall fragmentation. All
422usual features belonging to hugetlbfs are preserved and
423unaffected. libhugetlbfs will also work fine as usual.
424