1.. SPDX-License-Identifier: GPL-2.0
2
3====================================
4HOWTO for the linux packet generator
5====================================
6
7Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
8or as a module.  A module is preferred; modprobe pktgen if needed.  Once
9running, pktgen creates a thread for each CPU with affinity to that CPU.
10Monitoring and controlling is done via /proc.  It is easiest to select a
11suitable sample script and configure that.
12
13On a dual CPU::
14
15    ps aux | grep pkt
16    root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [kpktgend_0]
17    root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [kpktgend_1]
18
19
20For monitoring and control pktgen creates::
21
22	/proc/net/pktgen/pgctrl
23	/proc/net/pktgen/kpktgend_X
24	/proc/net/pktgen/ethX
25
26
27Tuning NIC for max performance
28==============================
29
30The default NIC settings are (likely) not tuned for pktgen's artificial
31overload type of benchmarking, as this could hurt the normal use-case.
32
33Specifically increasing the TX ring buffer in the NIC::
34
35 # ethtool -G ethX tx 1024
36
37A larger TX ring can improve pktgen's performance, while it can hurt
38in the general case, 1) because the TX ring buffer might get larger
39than the CPU's L1/L2 cache, 2) because it allows more queueing in the
40NIC HW layer (which is bad for bufferbloat).
41
42One should hesitate to conclude that packets/descriptors in the HW
43TX ring cause delay.  Drivers usually delay cleaning up the
44ring-buffers for various performance reasons, and packets stalling
45the TX ring might just be waiting for cleanup.
46
47This cleanup issue is specifically the case for the driver ixgbe
48(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
49and the cleanup interval is affected by the ethtool --coalesce setting
50of parameter "rx-usecs".
51
52For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6)::
53
54 # ethtool -C ethX rx-usecs 30
55
56
57Kernel threads
58==============
59Pktgen creates a thread for each CPU with affinity to that CPU.
60Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
61
62Example: /proc/net/pktgen/kpktgend_0::
63
64 Running:
65 Stopped: eth4@0
66 Result: OK: add_device=eth4@0
67
68Most important are the devices assigned to the thread.
69
70The two basic thread commands are:
71
72 * add_device DEVICE@NAME -- adds a single device
73 * rem_device_all         -- remove all associated devices
74
75When adding a device to a thread, a corresponding procfile is created
76which is used for configuring this device. Thus, device names need to
77be unique.
78
79To support adding the same device to multiple threads, which is useful
80with multi queue NICs, the device naming scheme is extended with "@":
81device@something
82
83The part after "@" can be anything, but it is custom to use the thread
84number.
85
86Viewing devices
87===============
88
89The Params section holds configured information.  The Current section
90holds running statistics.  The Result is printed after a run or after
91interruption.  Example::
92
93    /proc/net/pktgen/eth4@0
94
95    Params: count 100000  min_pkt_size: 60  max_pkt_size: 60
96	frags: 0  delay: 0  clone_skb: 64  ifname: eth4@0
97	flows: 0 flowlen: 0
98	queue_map_min: 0  queue_map_max: 0
99	dst_min: 192.168.81.2  dst_max:
100	src_min:   src_max:
101	src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
102	udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
103	src_mac_count: 0  dst_mac_count: 0
104	Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_CPU
105    Current:
106	pkts-sofar: 100000  errors: 0
107	started: 623913381008us  stopped: 623913396439us idle: 25us
108	seq_num: 100001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
109	cur_saddr: 192.168.8.3  cur_daddr: 192.168.81.2
110	cur_udp_dst: 9  cur_udp_src: 42
111	cur_queue_map: 0
112	flows: 0
113    Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
114    6480562pps 3110Mb/sec (3110669760bps) errors: 0
115
116
117Configuring devices
118===================
119This is done via the /proc interface, and most easily done via pgset
120as defined in the sample scripts.
121You need to specify PGDEV environment variable to use functions from sample
122scripts, i.e.::
123
124    export PGDEV=/proc/net/pktgen/eth4@0
125    source samples/pktgen/functions.sh
126
127Examples::
128
129 pg_ctrl start           starts injection.
130 pg_ctrl stop            aborts injection. Also, ^C aborts generator.
131
132 pgset "clone_skb 1"     sets the number of copies of the same packet
133 pgset "clone_skb 0"     use single SKB for all transmits
134 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
135			 packet and update HW tx queue tail pointer once.
136			 "burst 1" is the default
137 pgset "pkt_size 9014"   sets packet size to 9014
138 pgset "frags 5"         packet will consist of 5 fragments
139 pgset "count 200000"    sets number of packets to send, set to zero
140			 for continuous sends until explicitly stopped.
141
142 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds
143
144 pgset "dst 10.0.0.1"    sets IP destination address
145			 (BEWARE! This generator is very aggressive!)
146
147 pgset "dst_min 10.0.0.1"            Same as dst
148 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
149 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
150 pgset "src_max 10.0.0.254"          Set the maximum source IP.
151 pgset "dst6 fec0::1"     IPV6 destination address
152 pgset "src6 fec0::2"     IPV6 source address
153 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
154 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address
155
156 pgset "queue_map_min 0" Sets the min value of tx queue interval
157 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
158			 To select queue 1 of a given device,
159			 use queue_map_min=1 and queue_map_max=1
160
161 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
162			 The 'minimum' MAC is what you set with srcmac.
163
164 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
165			 The 'minimum' MAC is what you set with dstmac.
166
167 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
168			 are: IPSRC_RND # IP source is random (between min/max)
169			      IPDST_RND # IP destination is random
170			      UDPSRC_RND, UDPDST_RND,
171			      MACSRC_RND, MACDST_RND
172			      TXSIZE_RND, IPV6,
173			      MPLS_RND, VID_RND, SVID_RND
174			      FLOW_SEQ,
175			      QUEUE_MAP_RND # queue map random
176			      QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
177			      UDPCSUM,
178			      IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
179			      NODE_ALLOC # node specific memory allocation
180			      NO_TIMESTAMP # disable timestamping
181 pgset 'flag ![name]'    Clear a flag to determine behaviour.
182			 Note that you might need to use single quote in
183			 interactive mode, so that your shell wouldn't expand
184			 the specified flag as a history command.
185
186 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
187
188 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
189			 cycle through the port range.
190
191 pgset "udp_src_max 9"   set UDP source port max.
192 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
193			 cycle through the port range.
194 pgset "udp_dst_max 9"   set UDP destination port max.
195
196 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
197					 outer label=16,middle label=32,
198					 inner label=0 (IPv4 NULL)) Note that
199					 there must be no spaces between the
200					 arguments. Leading zeros are required.
201					 Do not set the bottom of stack bit,
202					 that's done automatically. If you do
203					 set the bottom of stack bit, that
204					 indicates that you want to randomly
205					 generate that address and the flag
206					 MPLS_RND will be turned on. You
207					 can have any mix of random and fixed
208					 labels in the label stack.
209
210 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)
211
212 pgset "vlan_id 77"       set VLAN ID 0-4095
213 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
214 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)
215
216 pgset "svlan_id 22"      set SVLAN ID 0-4095
217 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
218 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)
219
220 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
221 pgset "svlan 9999"       > 4095 remove svlan tag
222
223
224 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
225 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
226
227 pgset "rate 300M"        set rate to 300 Mb/s
228 pgset "ratep 1000000"    set rate to 1Mpps
229
230 pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
231				  Works with "burst" but not with "clone_skb".
232				  Default xmit_mode is "start_xmit".
233
234Sample scripts
235==============
236
237A collection of tutorial scripts and helpers for pktgen is in the
238samples/pktgen directory. The helper parameters.sh file support easy
239and consistent parameter parsing across the sample scripts.
240
241Usage example and help::
242
243 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
244
245Usage:::
246
247  ./pktgen_sample01_simple.sh [-vx] -i ethX
248
249  -i : ($DEV)       output interface/device (required)
250  -s : ($PKT_SIZE)  packet size
251  -d : ($DEST_IP)   destination IP
252  -m : ($DST_MAC)   destination MAC-addr
253  -t : ($THREADS)   threads to start
254  -c : ($SKB_CLONE) SKB clones send before alloc new SKB
255  -b : ($BURST)     HW level bursting of SKBs
256  -v : ($VERBOSE)   verbose
257  -x : ($DEBUG)     debug
258
259The global variables being set are also listed.  E.g. the required
260interface/device parameter "-i" sets variable $DEV.  Copy the
261pktgen_sampleXX scripts and modify them to fit your own needs.
262
263The old scripts::
264
265    pktgen.conf-1-2                  # 1 CPU 2 dev
266    pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS
267    pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
268    pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
269    pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.
270
271
272Interrupt affinity
273===================
274Note that when adding devices to a specific CPU it is a good idea to
275also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
276to the same CPU.  This reduces cache bouncing when freeing skbs.
277
278Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
279to the running threads CPU (directly from smp_processor_id()).
280
281Enable IPsec
282============
283Default IPsec transformation with ESP encapsulation plus transport mode
284can be enabled by simply setting::
285
286    pgset "flag IPSEC"
287    pgset "flows 1"
288
289To avoid breaking existing testbed scripts for using AH type and tunnel mode,
290you can use "pgset spi SPI_VALUE" to specify which transformation mode
291to employ.
292
293
294Current commands and configuration options
295==========================================
296
297**Pgcontrol commands**::
298
299    start
300    stop
301    reset
302
303**Thread commands**::
304
305    add_device
306    rem_device_all
307
308
309**Device commands**::
310
311    count
312    clone_skb
313    burst
314    debug
315
316    frags
317    delay
318
319    src_mac_count
320    dst_mac_count
321
322    pkt_size
323    min_pkt_size
324    max_pkt_size
325
326    queue_map_min
327    queue_map_max
328    skb_priority
329
330    tos           (ipv4)
331    traffic_class (ipv6)
332
333    mpls
334
335    udp_src_min
336    udp_src_max
337
338    udp_dst_min
339    udp_dst_max
340
341    node
342
343    flag
344    IPSRC_RND
345    IPDST_RND
346    UDPSRC_RND
347    UDPDST_RND
348    MACSRC_RND
349    MACDST_RND
350    TXSIZE_RND
351    IPV6
352    MPLS_RND
353    VID_RND
354    SVID_RND
355    FLOW_SEQ
356    QUEUE_MAP_RND
357    QUEUE_MAP_CPU
358    UDPCSUM
359    IPSEC
360    NODE_ALLOC
361    NO_TIMESTAMP
362
363    spi (ipsec)
364
365    dst_min
366    dst_max
367
368    src_min
369    src_max
370
371    dst_mac
372    src_mac
373
374    clear_counters
375
376    src6
377    dst6
378    dst6_max
379    dst6_min
380
381    flows
382    flowlen
383
384    rate
385    ratep
386
387    xmit_mode <start_xmit|netif_receive>
388
389    vlan_cfi
390    vlan_id
391    vlan_p
392
393    svlan_cfi
394    svlan_id
395    svlan_p
396
397
398References:
399
400- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
401- tp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
402
403Paper from Linux-Kongress in Erlangen 2004.
404- ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
405
406Thanks to:
407
408Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
409Stephen Hemminger, Andi Kleen, Dave Miller and many others.
410
411
412Good luck with the linux net-development.
413