1.. SPDX-License-Identifier: GPL-2.0
2
3=======================
4DAMON-based Reclamation
5=======================
6
7DAMON-based Reclamation (DAMON_RECLAIM) is a static kernel module that aimed to
8be used for proactive and lightweight reclamation under light memory pressure.
9It doesn't aim to replace the LRU-list based page_granularity reclamation, but
10to be selectively used for different level of memory pressure and requirements.
11
12Where Proactive Reclamation is Required?
13========================================
14
15On general memory over-committed systems, proactively reclaiming cold pages
16helps saving memory and reducing latency spikes that incurred by the direct
17reclaim of the process or CPU consumption of kswapd, while incurring only
18minimal performance degradation [1]_ [2]_ .
19
20Free Pages Reporting [3]_ based memory over-commit virtualization systems are
21good example of the cases.  In such systems, the guest VMs reports their free
22memory to host, and the host reallocates the reported memory to other guests.
23As a result, the memory of the systems are fully utilized.  However, the
24guests could be not so memory-frugal, mainly because some kernel subsystems and
25user-space applications are designed to use as much memory as available.  Then,
26guests could report only small amount of memory as free to host, results in
27memory utilization drop of the systems.  Running the proactive reclamation in
28guests could mitigate this problem.
29
30How It Works?
31=============
32
33DAMON_RECLAIM finds memory regions that didn't accessed for specific time
34duration and page out.  To avoid it consuming too much CPU for the paging out
35operation, a speed limit can be configured.  Under the speed limit, it pages
36out memory regions that didn't accessed longer time first.  System
37administrators can also configure under what situation this scheme should
38automatically activated and deactivated with three memory pressure watermarks.
39
40Interface: Module Parameters
41============================
42
43To use this feature, you should first ensure your system is running on a kernel
44that is built with ``CONFIG_DAMON_RECLAIM=y``.
45
46To let sysadmins enable or disable it and tune for the given system,
47DAMON_RECLAIM utilizes module parameters.  That is, you can put
48``damon_reclaim.<parameter>=<value>`` on the kernel boot command line or write
49proper values to ``/sys/modules/damon_reclaim/parameters/<parameter>`` files.
50
51Below are the description of each parameter.
52
53enabled
54-------
55
56Enable or disable DAMON_RECLAIM.
57
58You can enable DAMON_RCLAIM by setting the value of this parameter as ``Y``.
59Setting it as ``N`` disables DAMON_RECLAIM.  Note that DAMON_RECLAIM could do
60no real monitoring and reclamation due to the watermarks-based activation
61condition.  Refer to below descriptions for the watermarks parameter for this.
62
63commit_inputs
64-------------
65
66Make DAMON_RECLAIM reads the input parameters again, except ``enabled``.
67
68Input parameters that updated while DAMON_RECLAIM is running are not applied
69by default.  Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
70of parametrs except ``enabled`` again.  Once the re-reading is done, this
71parameter is set as ``N``.  If invalid parameters are found while the
72re-reading, DAMON_RECLAIM will be disabled.
73
74min_age
75-------
76
77Time threshold for cold memory regions identification in microseconds.
78
79If a memory region is not accessed for this or longer time, DAMON_RECLAIM
80identifies the region as cold, and reclaims it.
81
82120 seconds by default.
83
84quota_ms
85--------
86
87Limit of time for the reclamation in milliseconds.
88
89DAMON_RECLAIM tries to use only up to this time within a time window
90(quota_reset_interval_ms) for trying reclamation of cold pages.  This can be
91used for limiting CPU consumption of DAMON_RECLAIM.  If the value is zero, the
92limit is disabled.
93
9410 ms by default.
95
96quota_sz
97--------
98
99Limit of size of memory for the reclamation in bytes.
100
101DAMON_RECLAIM charges amount of memory which it tried to reclaim within a time
102window (quota_reset_interval_ms) and makes no more than this limit is tried.
103This can be used for limiting consumption of CPU and IO.  If this value is
104zero, the limit is disabled.
105
106128 MiB by default.
107
108quota_reset_interval_ms
109-----------------------
110
111The time/size quota charge reset interval in milliseconds.
112
113The charget reset interval for the quota of time (quota_ms) and size
114(quota_sz).  That is, DAMON_RECLAIM does not try reclamation for more than
115quota_ms milliseconds or quota_sz bytes within quota_reset_interval_ms
116milliseconds.
117
1181 second by default.
119
120wmarks_interval
121---------------
122
123Minimal time to wait before checking the watermarks, when DAMON_RECLAIM is
124enabled but inactive due to its watermarks rule.
125
126wmarks_high
127-----------
128
129Free memory rate (per thousand) for the high watermark.
130
131If free memory of the system in bytes per thousand bytes is higher than this,
132DAMON_RECLAIM becomes inactive, so it does nothing but only periodically checks
133the watermarks.
134
135wmarks_mid
136----------
137
138Free memory rate (per thousand) for the middle watermark.
139
140If free memory of the system in bytes per thousand bytes is between this and
141the low watermark, DAMON_RECLAIM becomes active, so starts the monitoring and
142the reclaiming.
143
144wmarks_low
145----------
146
147Free memory rate (per thousand) for the low watermark.
148
149If free memory of the system in bytes per thousand bytes is lower than this,
150DAMON_RECLAIM becomes inactive, so it does nothing but periodically checks the
151watermarks.  In the case, the system falls back to the LRU-list based page
152granularity reclamation logic.
153
154sample_interval
155---------------
156
157Sampling interval for the monitoring in microseconds.
158
159The sampling interval of DAMON for the cold memory monitoring.  Please refer to
160the DAMON documentation (:doc:`usage`) for more detail.
161
162aggr_interval
163-------------
164
165Aggregation interval for the monitoring in microseconds.
166
167The aggregation interval of DAMON for the cold memory monitoring.  Please
168refer to the DAMON documentation (:doc:`usage`) for more detail.
169
170min_nr_regions
171--------------
172
173Minimum number of monitoring regions.
174
175The minimal number of monitoring regions of DAMON for the cold memory
176monitoring.  This can be used to set lower-bound of the monitoring quality.
177But, setting this too high could result in increased monitoring overhead.
178Please refer to the DAMON documentation (:doc:`usage`) for more detail.
179
180max_nr_regions
181--------------
182
183Maximum number of monitoring regions.
184
185The maximum number of monitoring regions of DAMON for the cold memory
186monitoring.  This can be used to set upper-bound of the monitoring overhead.
187However, setting this too low could result in bad monitoring quality.  Please
188refer to the DAMON documentation (:doc:`usage`) for more detail.
189
190monitor_region_start
191--------------------
192
193Start of target memory region in physical address.
194
195The start physical address of memory region that DAMON_RECLAIM will do work
196against.  That is, DAMON_RECLAIM will find cold memory regions in this region
197and reclaims.  By default, biggest System RAM is used as the region.
198
199monitor_region_end
200------------------
201
202End of target memory region in physical address.
203
204The end physical address of memory region that DAMON_RECLAIM will do work
205against.  That is, DAMON_RECLAIM will find cold memory regions in this region
206and reclaims.  By default, biggest System RAM is used as the region.
207
208kdamond_pid
209-----------
210
211PID of the DAMON thread.
212
213If DAMON_RECLAIM is enabled, this becomes the PID of the worker thread.  Else,
214-1.
215
216nr_reclaim_tried_regions
217------------------------
218
219Number of memory regions that tried to be reclaimed by DAMON_RECLAIM.
220
221bytes_reclaim_tried_regions
222---------------------------
223
224Total bytes of memory regions that tried to be reclaimed by DAMON_RECLAIM.
225
226nr_reclaimed_regions
227--------------------
228
229Number of memory regions that successfully be reclaimed by DAMON_RECLAIM.
230
231bytes_reclaimed_regions
232-----------------------
233
234Total bytes of memory regions that successfully be reclaimed by DAMON_RECLAIM.
235
236nr_quota_exceeds
237----------------
238
239Number of times that the time/space quota limits have exceeded.
240
241Example
242=======
243
244Below runtime example commands make DAMON_RECLAIM to find memory regions that
245not accessed for 30 seconds or more and pages out.  The reclamation is limited
246to be done only up to 1 GiB per second to avoid DAMON_RECLAIM consuming too
247much CPU time for the paging out operation.  It also asks DAMON_RECLAIM to do
248nothing if the system's free memory rate is more than 50%, but start the real
249works if it becomes lower than 40%.  If DAMON_RECLAIM doesn't make progress and
250therefore the free memory rate becomes lower than 20%, it asks DAMON_RECLAIM to
251do nothing again, so that we can fall back to the LRU-list based page
252granularity reclamation. ::
253
254    # cd /sys/modules/damon_reclaim/parameters
255    # echo 30000000 > min_age
256    # echo $((1 * 1024 * 1024 * 1024)) > quota_sz
257    # echo 1000 > quota_reset_interval_ms
258    # echo 500 > wmarks_high
259    # echo 400 > wmarks_mid
260    # echo 200 > wmarks_low
261    # echo Y > enabled
262
263.. [1] https://research.google/pubs/pub48551/
264.. [2] https://lwn.net/Articles/787611/
265.. [3] https://www.kernel.org/doc/html/latest/mm/free_page_reporting.html
266