1.. SPDX-License-Identifier: GPL-2.0
2
3=============
4Page Pool API
5=============
6
7The page_pool allocator is optimized for the XDP mode that uses one frame
8per-page, but it can fallback on the regular page allocator APIs.
9
10Basic use involves replacing alloc_pages() calls with the
11page_pool_alloc_pages() call.  Drivers should use page_pool_dev_alloc_pages()
12replacing dev_alloc_pages().
13
14API keeps track of in-flight pages, in order to let API user know
15when it is safe to free a page_pool object.  Thus, API users
16must run page_pool_release_page() when a page is leaving the page_pool or
17call page_pool_put_page() where appropriate in order to maintain correct
18accounting.
19
20API user must call page_pool_put_page() once on a page, as it
21will either recycle the page, or in case of refcnt > 1, it will
22release the DMA mapping and in-flight state accounting.
23
24Architecture overview
25=====================
26
27.. code-block:: none
28
29    +------------------+
30    |       Driver     |
31    +------------------+
32            ^
33            |
34            |
35            |
36            v
37    +--------------------------------------------+
38    |                request memory              |
39    +--------------------------------------------+
40        ^                                  ^
41        |                                  |
42        | Pool empty                       | Pool has entries
43        |                                  |
44        v                                  v
45    +-----------------------+     +------------------------+
46    | alloc (and map) pages |     |  get page from cache   |
47    +-----------------------+     +------------------------+
48                                    ^                    ^
49                                    |                    |
50                                    | cache available    | No entries, refill
51                                    |                    | from ptr-ring
52                                    |                    |
53                                    v                    v
54                          +-----------------+     +------------------+
55                          |   Fast cache    |     |  ptr-ring cache  |
56                          +-----------------+     +------------------+
57
58API interface
59=============
60The number of pools created **must** match the number of hardware queues
61unless hardware restrictions make that impossible. This would otherwise beat the
62purpose of page pool, which is allocate pages fast from cache without locking.
63This lockless guarantee naturally comes from running under a NAPI softirq.
64The protection doesn't strictly have to be NAPI, any guarantee that allocating
65a page will cause no race conditions is enough.
66
67* page_pool_create(): Create a pool.
68    * flags:      PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV
69    * order:      2^order pages on allocation
70    * pool_size:  size of the ptr_ring
71    * nid:        preferred NUMA node for allocation
72    * dev:        struct device. Used on DMA operations
73    * dma_dir:    DMA direction
74    * max_len:    max DMA sync memory size
75    * offset:     DMA address offset
76
77* page_pool_put_page(): The outcome of this depends on the page refcnt. If the
78  driver bumps the refcnt > 1 this will unmap the page. If the page refcnt is 1
79  the allocator owns the page and will try to recycle it in one of the pool
80  caches. If PP_FLAG_DMA_SYNC_DEV is set, the page will be synced for_device
81  using dma_sync_single_range_for_device().
82
83* page_pool_put_full_page(): Similar to page_pool_put_page(), but will DMA sync
84  for the entire memory area configured in area pool->max_len.
85
86* page_pool_recycle_direct(): Similar to page_pool_put_full_page() but caller
87  must guarantee safe context (e.g NAPI), since it will recycle the page
88  directly into the pool fast cache.
89
90* page_pool_release_page(): Unmap the page (if mapped) and account for it on
91  in-flight counters.
92
93* page_pool_dev_alloc_pages(): Get a page from the page allocator or page_pool
94  caches.
95
96* page_pool_get_dma_addr(): Retrieve the stored DMA address.
97
98* page_pool_get_dma_dir(): Retrieve the stored DMA direction.
99
100* page_pool_put_page_bulk(): Tries to refill a number of pages into the
101  ptr_ring cache holding ptr_ring producer lock. If the ptr_ring is full,
102  page_pool_put_page_bulk() will release leftover pages to the page allocator.
103  page_pool_put_page_bulk() is suitable to be run inside the driver NAPI tx
104  completion loop for the XDP_REDIRECT use case.
105  Please note the caller must not use data area after running
106  page_pool_put_page_bulk(), as this function overwrites it.
107
108* page_pool_get_stats(): Retrieve statistics about the page_pool. This API
109  is only available if the kernel has been configured with
110  ``CONFIG_PAGE_POOL_STATS=y``. A pointer to a caller allocated ``struct
111  page_pool_stats`` structure is passed to this API which is filled in. The
112  caller can then report those stats to the user (perhaps via ethtool,
113  debugfs, etc.). See below for an example usage of this API.
114
115Stats API and structures
116------------------------
117If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
118``page_pool_get_stats()`` and structures described below are available. It
119takes a  pointer to a ``struct page_pool`` and a pointer to a ``struct
120page_pool_stats`` allocated by the caller.
121
122The API will fill in the provided ``struct page_pool_stats`` with
123statistics about the page_pool.
124
125The stats structure has the following fields::
126
127    struct page_pool_stats {
128        struct page_pool_alloc_stats alloc_stats;
129        struct page_pool_recycle_stats recycle_stats;
130    };
131
132
133The ``struct page_pool_alloc_stats`` has the following fields:
134  * ``fast``: successful fast path allocations
135  * ``slow``: slow path order-0 allocations
136  * ``slow_high_order``: slow path high order allocations
137  * ``empty``: ptr ring is empty, so a slow path allocation was forced.
138  * ``refill``: an allocation which triggered a refill of the cache
139  * ``waive``: pages obtained from the ptr ring that cannot be added to
140    the cache due to a NUMA mismatch.
141
142The ``struct page_pool_recycle_stats`` has the following fields:
143  * ``cached``: recycling placed page in the page pool cache
144  * ``cache_full``: page pool cache was full
145  * ``ring``: page placed into the ptr ring
146  * ``ring_full``: page released from page pool because the ptr ring was full
147  * ``released_refcnt``: page released (and not recycled) because refcnt > 1
148
149Coding examples
150===============
151
152Registration
153------------
154
155.. code-block:: c
156
157    /* Page pool registration */
158    struct page_pool_params pp_params = { 0 };
159    struct xdp_rxq_info xdp_rxq;
160    int err;
161
162    pp_params.order = 0;
163    /* internal DMA mapping in page_pool */
164    pp_params.flags = PP_FLAG_DMA_MAP;
165    pp_params.pool_size = DESC_NUM;
166    pp_params.nid = NUMA_NO_NODE;
167    pp_params.dev = priv->dev;
168    pp_params.napi = napi; /* only if locking is tied to NAPI */
169    pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
170    page_pool = page_pool_create(&pp_params);
171
172    err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0);
173    if (err)
174        goto err_out;
175
176    err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool);
177    if (err)
178        goto err_out;
179
180NAPI poller
181-----------
182
183
184.. code-block:: c
185
186    /* NAPI Rx poller */
187    enum dma_data_direction dma_dir;
188
189    dma_dir = page_pool_get_dma_dir(dring->page_pool);
190    while (done < budget) {
191        if (some error)
192            page_pool_recycle_direct(page_pool, page);
193        if (packet_is_xdp) {
194            if XDP_DROP:
195                page_pool_recycle_direct(page_pool, page);
196        } else (packet_is_skb) {
197            page_pool_release_page(page_pool, page);
198            new_page = page_pool_dev_alloc_pages(page_pool);
199        }
200    }
201
202Stats
203-----
204
205.. code-block:: c
206
207	#ifdef CONFIG_PAGE_POOL_STATS
208	/* retrieve stats */
209	struct page_pool_stats stats = { 0 };
210	if (page_pool_get_stats(page_pool, &stats)) {
211		/* perhaps the driver reports statistics with ethool */
212		ethtool_print_allocation_stats(&stats.alloc_stats);
213		ethtool_print_recycle_stats(&stats.recycle_stats);
214	}
215	#endif
216
217Driver unload
218-------------
219
220.. code-block:: c
221
222    /* Driver unload */
223    page_pool_put_full_page(page_pool, page, false);
224    xdp_rxq_info_unreg(&xdp_rxq);
225