1.. SPDX-License-Identifier: GPL-2.0
2
3======
4AF_XDP
5======
6
7Overview
8========
9
10AF_XDP is an address family that is optimized for high performance
11packet processing.
12
13This document assumes that the reader is familiar with BPF and XDP. If
14not, the Cilium project has an excellent reference guide at
15http://cilium.readthedocs.io/en/latest/bpf/.
16
17Using the XDP_REDIRECT action from an XDP program, the program can
18redirect ingress frames to other XDP enabled netdevs, using the
19bpf_redirect_map() function. AF_XDP sockets enable the possibility for
20XDP programs to redirect frames to a memory buffer in a user-space
21application.
22
23An AF_XDP socket (XSK) is created with the normal socket()
24syscall. Associated with each XSK are two rings: the RX ring and the
25TX ring. A socket can receive packets on the RX ring and it can send
26packets on the TX ring. These rings are registered and sized with the
27setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory
28to have at least one of these rings for each socket. An RX or TX
29descriptor ring points to a data buffer in a memory area called a
30UMEM. RX and TX can share the same UMEM so that a packet does not have
31to be copied between RX and TX. Moreover, if a packet needs to be kept
32for a while due to a possible retransmit, the descriptor that points
33to that packet can be changed to point to another and reused right
34away. This again avoids copying data.
35
36The UMEM consists of a number of equally sized chunks. A descriptor in
37one of the rings references a frame by referencing its addr. The addr
38is simply an offset within the entire UMEM region. The user space
39allocates memory for this UMEM using whatever means it feels is most
40appropriate (malloc, mmap, huge pages, etc). This memory area is then
41registered with the kernel using the new setsockopt XDP_UMEM_REG. The
42UMEM also has two rings: the FILL ring and the COMPLETION ring. The
43fill ring is used by the application to send down addr for the kernel
44to fill in with RX packet data. References to these frames will then
45appear in the RX ring once each packet has been received. The
46completion ring, on the other hand, contains frame addr that the
47kernel has transmitted completely and can now be used again by user
48space, for either TX or RX. Thus, the frame addrs appearing in the
49completion ring are addrs that were previously transmitted using the
50TX ring. In summary, the RX and FILL rings are used for the RX path
51and the TX and COMPLETION rings are used for the TX path.
52
53The socket is then finally bound with a bind() call to a device and a
54specific queue id on that device, and it is not until bind is
55completed that traffic starts to flow.
56
57The UMEM can be shared between processes, if desired. If a process
58wants to do this, it simply skips the registration of the UMEM and its
59corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind
60call and submits the XSK of the process it would like to share UMEM
61with as well as its own newly created XSK socket. The new process will
62then receive frame addr references in its own RX ring that point to
63this shared UMEM. Note that since the ring structures are
64single-consumer / single-producer (for performance reasons), the new
65process has to create its own socket with associated RX and TX rings,
66since it cannot share this with the other process. This is also the
67reason that there is only one set of FILL and COMPLETION rings per
68UMEM. It is the responsibility of a single process to handle the UMEM.
69
70How is then packets distributed from an XDP program to the XSKs? There
71is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The
72user-space application can place an XSK at an arbitrary place in this
73map. The XDP program can then redirect a packet to a specific index in
74this map and at this point XDP validates that the XSK in that map was
75indeed bound to that device and ring number. If not, the packet is
76dropped. If the map is empty at that index, the packet is also
77dropped. This also means that it is currently mandatory to have an XDP
78program loaded (and one XSK in the XSKMAP) to be able to get any
79traffic to user space through the XSK.
80
81AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the
82driver does not have support for XDP, or XDP_SKB is explicitly chosen
83when loading the XDP program, XDP_SKB mode is employed that uses SKBs
84together with the generic XDP support and copies out the data to user
85space. A fallback mode that works for any network device. On the other
86hand, if the driver has support for XDP, it will be used by the AF_XDP
87code to provide better performance, but there is still a copy of the
88data into user space.
89
90Concepts
91========
92
93In order to use an AF_XDP socket, a number of associated objects need
94to be setup.
95
96Jonathan Corbet has also written an excellent article on LWN,
97"Accelerating networking with AF_XDP". It can be found at
98https://lwn.net/Articles/750845/.
99
100UMEM
101----
102
103UMEM is a region of virtual contiguous memory, divided into
104equal-sized frames. An UMEM is associated to a netdev and a specific
105queue id of that netdev. It is created and configured (chunk size,
106headroom, start address and size) by using the XDP_UMEM_REG setsockopt
107system call. A UMEM is bound to a netdev and queue id, via the bind()
108system call.
109
110An AF_XDP is socket linked to a single UMEM, but one UMEM can have
111multiple AF_XDP sockets. To share an UMEM created via one socket A,
112the next socket B can do this by setting the XDP_SHARED_UMEM flag in
113struct sockaddr_xdp member sxdp_flags, and passing the file descriptor
114of A to struct sockaddr_xdp member sxdp_shared_umem_fd.
115
116The UMEM has two single-producer/single-consumer rings, that are used
117to transfer ownership of UMEM frames between the kernel and the
118user-space application.
119
120Rings
121-----
122
123There are a four different kind of rings: Fill, Completion, RX and
124TX. All rings are single-producer/single-consumer, so the user-space
125application need explicit synchronization of multiple
126processes/threads are reading/writing to them.
127
128The UMEM uses two rings: Fill and Completion. Each socket associated
129with the UMEM must have an RX queue, TX queue or both. Say, that there
130is a setup with four sockets (all doing TX and RX). Then there will be
131one Fill ring, one Completion ring, four TX rings and four RX rings.
132
133The rings are head(producer)/tail(consumer) based rings. A producer
134writes the data ring at the index pointed out by struct xdp_ring
135producer member, and increasing the producer index. A consumer reads
136the data ring at the index pointed out by struct xdp_ring consumer
137member, and increasing the consumer index.
138
139The rings are configured and created via the _RING setsockopt system
140calls and mmapped to user-space using the appropriate offset to mmap()
141(XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and
142XDP_UMEM_PGOFF_COMPLETION_RING).
143
144The size of the rings need to be of size power of two.
145
146UMEM Fill Ring
147~~~~~~~~~~~~~~
148
149The Fill ring is used to transfer ownership of UMEM frames from
150user-space to kernel-space. The UMEM addrs are passed in the ring. As
151an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has
15216 chunks and can pass addrs between 0 and 64k.
153
154Frames passed to the kernel are used for the ingress path (RX rings).
155
156The user application produces UMEM addrs to this ring. Note that the
157kernel will mask the incoming addr. E.g. for a chunk size of 2k, the
158log2(2048) LSB of the addr will be masked off, meaning that 2048, 2050
159and 3000 refers to the same chunk.
160
161
162UMEM Completion Ring
163~~~~~~~~~~~~~~~~~~~~
164
165The Completion Ring is used transfer ownership of UMEM frames from
166kernel-space to user-space. Just like the Fill ring, UMEM indicies are
167used.
168
169Frames passed from the kernel to user-space are frames that has been
170sent (TX ring) and can be used by user-space again.
171
172The user application consumes UMEM addrs from this ring.
173
174
175RX Ring
176~~~~~~~
177
178The RX ring is the receiving side of a socket. Each entry in the ring
179is a struct xdp_desc descriptor. The descriptor contains UMEM offset
180(addr) and the length of the data (len).
181
182If no frames have been passed to kernel via the Fill ring, no
183descriptors will (or can) appear on the RX ring.
184
185The user application consumes struct xdp_desc descriptors from this
186ring.
187
188TX Ring
189~~~~~~~
190
191The TX ring is used to send frames. The struct xdp_desc descriptor is
192filled (index, length and offset) and passed into the ring.
193
194To start the transfer a sendmsg() system call is required. This might
195be relaxed in the future.
196
197The user application produces struct xdp_desc descriptors to this
198ring.
199
200XSKMAP / BPF_MAP_TYPE_XSKMAP
201----------------------------
202
203On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that
204is used in conjunction with bpf_redirect_map() to pass the ingress
205frame to a socket.
206
207The user application inserts the socket into the map, via the bpf()
208system call.
209
210Note that if an XDP program tries to redirect to a socket that does
211not match the queue configuration and netdev, the frame will be
212dropped. E.g. an AF_XDP socket is bound to netdev eth0 and
213queue 17. Only the XDP program executing for eth0 and queue 17 will
214successfully pass data to the socket. Please refer to the sample
215application (samples/bpf/) in for an example.
216
217Usage
218=====
219
220In order to use AF_XDP sockets there are two parts needed. The
221user-space application and the XDP program. For a complete setup and
222usage example, please refer to the sample application. The user-space
223side is xdpsock_user.c and the XDP side xdpsock_kern.c.
224
225Naive ring dequeue and enqueue could look like this::
226
227    // struct xdp_rxtx_ring {
228    // 	__u32 *producer;
229    // 	__u32 *consumer;
230    // 	struct xdp_desc *desc;
231    // };
232
233    // struct xdp_umem_ring {
234    // 	__u32 *producer;
235    // 	__u32 *consumer;
236    // 	__u64 *desc;
237    // };
238
239    // typedef struct xdp_rxtx_ring RING;
240    // typedef struct xdp_umem_ring RING;
241
242    // typedef struct xdp_desc RING_TYPE;
243    // typedef __u64 RING_TYPE;
244
245    int dequeue_one(RING *ring, RING_TYPE *item)
246    {
247        __u32 entries = *ring->producer - *ring->consumer;
248
249        if (entries == 0)
250            return -1;
251
252        // read-barrier!
253
254        *item = ring->desc[*ring->consumer & (RING_SIZE - 1)];
255        (*ring->consumer)++;
256        return 0;
257    }
258
259    int enqueue_one(RING *ring, const RING_TYPE *item)
260    {
261        u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer);
262
263        if (free_entries == 0)
264            return -1;
265
266        ring->desc[*ring->producer & (RING_SIZE - 1)] = *item;
267
268        // write-barrier!
269
270        (*ring->producer)++;
271        return 0;
272    }
273
274
275For a more optimized version, please refer to the sample application.
276
277Sample application
278==================
279
280There is a xdpsock benchmarking/test application included that
281demonstrates how to use AF_XDP sockets with both private and shared
282UMEMs. Say that you would like your UDP traffic from port 4242 to end
283up in queue 16, that we will enable AF_XDP on. Here, we use ethtool
284for this::
285
286      ethtool -N p3p2 rx-flow-hash udp4 fn
287      ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \
288          action 16
289
290Running the rxdrop benchmark in XDP_DRV mode can then be done
291using::
292
293      samples/bpf/xdpsock -i p3p2 -q 16 -r -N
294
295For XDP_SKB mode, use the switch "-S" instead of "-N" and all options
296can be displayed with "-h", as usual.
297
298Credits
299=======
300
301- Björn Töpel (AF_XDP core)
302- Magnus Karlsson (AF_XDP core)
303- Alexander Duyck
304- Alexei Starovoitov
305- Daniel Borkmann
306- Jesper Dangaard Brouer
307- John Fastabend
308- Jonathan Corbet (LWN coverage)
309- Michael S. Tsirkin
310- Qi Z Zhang
311- Willem de Bruijn
312
313