xref: /openbmc/linux/Documentation/networking/device_drivers/ethernet/ti/cpsw.rst (revision 4b4193256c8d3bc3a5397b5cd9494c2ad386317d)
1*132db935SJakub Kicinski.. SPDX-License-Identifier: GPL-2.0
2*132db935SJakub Kicinski
3*132db935SJakub Kicinski======================================
4*132db935SJakub KicinskiTexas Instruments CPSW ethernet driver
5*132db935SJakub Kicinski======================================
6*132db935SJakub Kicinski
7*132db935SJakub KicinskiMultiqueue & CBS & MQPRIO
8*132db935SJakub Kicinski=========================
9*132db935SJakub Kicinski
10*132db935SJakub Kicinski
11*132db935SJakub KicinskiThe cpsw has 3 CBS shapers for each external ports. This document
12*132db935SJakub Kicinskidescribes MQPRIO and CBS Qdisc offload configuration for cpsw driver
13*132db935SJakub Kicinskibased on examples. It potentially can be used in audio video bridging
14*132db935SJakub Kicinski(AVB) and time sensitive networking (TSN).
15*132db935SJakub Kicinski
16*132db935SJakub KicinskiThe following examples were tested on AM572x EVM and BBB boards.
17*132db935SJakub Kicinski
18*132db935SJakub KicinskiTest setup
19*132db935SJakub Kicinski==========
20*132db935SJakub Kicinski
21*132db935SJakub KicinskiUnder consideration two examples with AM572x EVM running cpsw driver
22*132db935SJakub Kicinskiin dual_emac mode.
23*132db935SJakub Kicinski
24*132db935SJakub KicinskiSeveral prerequisites:
25*132db935SJakub Kicinski
26*132db935SJakub Kicinski- TX queues must be rated starting from txq0 that has highest priority
27*132db935SJakub Kicinski- Traffic classes are used starting from 0, that has highest priority
28*132db935SJakub Kicinski- CBS shapers should be used with rated queues
29*132db935SJakub Kicinski- The bandwidth for CBS shapers has to be set a little bit more then
30*132db935SJakub Kicinski  potential incoming rate, thus, rate of all incoming tx queues has
31*132db935SJakub Kicinski  to be a little less
32*132db935SJakub Kicinski- Real rates can differ, due to discreetness
33*132db935SJakub Kicinski- Map skb-priority to txq is not enough, also skb-priority to l2 prio
34*132db935SJakub Kicinski  map has to be created with ip or vconfig tool
35*132db935SJakub Kicinski- Any l2/socket prio (0 - 7) for classes can be used, but for
36*132db935SJakub Kicinski  simplicity default values are used: 3 and 2
37*132db935SJakub Kicinski- only 2 classes tested: A and B, but checked and can work with more,
38*132db935SJakub Kicinski  maximum allowed 4, but only for 3 rate can be set.
39*132db935SJakub Kicinski
40*132db935SJakub KicinskiTest setup for examples
41*132db935SJakub Kicinski=======================
42*132db935SJakub Kicinski
43*132db935SJakub Kicinski::
44*132db935SJakub Kicinski
45*132db935SJakub Kicinski					+-------------------------------+
46*132db935SJakub Kicinski					|--+                            |
47*132db935SJakub Kicinski					|  |      Workstation0          |
48*132db935SJakub Kicinski					|E |  MAC 18:03:73:66:87:42     |
49*132db935SJakub Kicinski    +-----------------------------+  +--|t |                            |
50*132db935SJakub Kicinski    |                    | 1  | E |  |  |h |./tsn_listener -d \         |
51*132db935SJakub Kicinski    |  Target board:     | 0  | t |--+  |0 | 18:03:73:66:87:42 -i eth0 \|
52*132db935SJakub Kicinski    |  AM572x EVM        | 0  | h |     |  | -s 1500                    |
53*132db935SJakub Kicinski    |                    | 0  | 0 |     |--+                            |
54*132db935SJakub Kicinski    |  Only 2 classes:   |Mb  +---|     +-------------------------------+
55*132db935SJakub Kicinski    |  class A, class B  |        |
56*132db935SJakub Kicinski    |                    |    +---|     +-------------------------------+
57*132db935SJakub Kicinski    |                    | 1  | E |     |--+                            |
58*132db935SJakub Kicinski    |                    | 0  | t |     |  |      Workstation1          |
59*132db935SJakub Kicinski    |                    | 0  | h |--+  |E |  MAC 20:cf:30:85:7d:fd     |
60*132db935SJakub Kicinski    |                    |Mb  | 1 |  +--|t |                            |
61*132db935SJakub Kicinski    +-----------------------------+     |h |./tsn_listener -d \         |
62*132db935SJakub Kicinski					|0 | 20:cf:30:85:7d:fd -i eth0 \|
63*132db935SJakub Kicinski					|  | -s 1500                    |
64*132db935SJakub Kicinski					|--+                            |
65*132db935SJakub Kicinski					+-------------------------------+
66*132db935SJakub Kicinski
67*132db935SJakub Kicinski
68*132db935SJakub KicinskiExample 1: One port tx AVB configuration scheme for target board
69*132db935SJakub Kicinski----------------------------------------------------------------
70*132db935SJakub Kicinski
71*132db935SJakub Kicinski(prints and scheme for AM572x evm, applicable for single port boards)
72*132db935SJakub Kicinski
73*132db935SJakub Kicinski- tc - traffic class
74*132db935SJakub Kicinski- txq - transmit queue
75*132db935SJakub Kicinski- p - priority
76*132db935SJakub Kicinski- f - fifo (cpsw fifo)
77*132db935SJakub Kicinski- S - shaper configured
78*132db935SJakub Kicinski
79*132db935SJakub Kicinski::
80*132db935SJakub Kicinski
81*132db935SJakub Kicinski    +------------------------------------------------------------------+ u
82*132db935SJakub Kicinski    | +---------------+  +---------------+  +------+ +------+          | s
83*132db935SJakub Kicinski    | |               |  |               |  |      | |      |          | e
84*132db935SJakub Kicinski    | | App 1         |  | App 2         |  | Apps | | Apps |          | r
85*132db935SJakub Kicinski    | | Class A       |  | Class B       |  | Rest | | Rest |          |
86*132db935SJakub Kicinski    | | Eth0          |  | Eth0          |  | Eth0 | | Eth1 |          | s
87*132db935SJakub Kicinski    | | VLAN100       |  | VLAN100       |  |   |  | |   |  |          | p
88*132db935SJakub Kicinski    | | 40 Mb/s       |  | 20 Mb/s       |  |   |  | |   |  |          | a
89*132db935SJakub Kicinski    | | SO_PRIORITY=3 |  | SO_PRIORITY=2 |  |   |  | |   |  |          | c
90*132db935SJakub Kicinski    | |   |           |  |   |           |  |   |  | |   |  |          | e
91*132db935SJakub Kicinski    | +---|-----------+  +---|-----------+  +---|--+ +---|--+          |
92*132db935SJakub Kicinski    +-----|------------------|------------------|--------|-------------+
93*132db935SJakub Kicinski	+-+     +------------+                  |        |
94*132db935SJakub Kicinski	|       |             +-----------------+     +--+
95*132db935SJakub Kicinski	|       |             |                       |
96*132db935SJakub Kicinski    +---|-------|-------------|-----------------------|----------------+
97*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+                   +----+             |
98*132db935SJakub Kicinski    | | p3 | | p2 | | p1 | | p0 |                   | p0 |             | k
99*132db935SJakub Kicinski    | \    / \    / \    / \    /                   \    /             | e
100*132db935SJakub Kicinski    |  \  /   \  /   \  /   \  /                     \  /              | r
101*132db935SJakub Kicinski    |   \/     \/     \/     \/                       \/               | n
102*132db935SJakub Kicinski    |    |     |             |                        |                | e
103*132db935SJakub Kicinski    |    |     |       +-----+                        |                | l
104*132db935SJakub Kicinski    |    |     |       |                              |                |
105*132db935SJakub Kicinski    | +----+ +----+ +----+                          +----+             | s
106*132db935SJakub Kicinski    | |tc0 | |tc1 | |tc2 |                          |tc0 |             | p
107*132db935SJakub Kicinski    | \    / \    / \    /                          \    /             | a
108*132db935SJakub Kicinski    |  \  /   \  /   \  /                            \  /              | c
109*132db935SJakub Kicinski    |   \/     \/     \/                              \/               | e
110*132db935SJakub Kicinski    |   |      |       +-----+                        |                |
111*132db935SJakub Kicinski    |   |      |       |     |                        |                |
112*132db935SJakub Kicinski    |   |      |       |     |                        |                |
113*132db935SJakub Kicinski    |   |      |       |     |                        |                |
114*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+                   +----+             |
115*132db935SJakub Kicinski    | |txq0| |txq1| |txq2| |txq3|                   |txq4|             |
116*132db935SJakub Kicinski    | \    / \    / \    / \    /                   \    /             |
117*132db935SJakub Kicinski    |  \  /   \  /   \  /   \  /                     \  /              |
118*132db935SJakub Kicinski    |   \/     \/     \/     \/                       \/               |
119*132db935SJakub Kicinski    | +-|------|------|------|--+                  +--|--------------+ |
120*132db935SJakub Kicinski    | | |      |      |      |  | Eth0.100         |  |     Eth1     | |
121*132db935SJakub Kicinski    +---|------|------|------|------------------------|----------------+
122*132db935SJakub Kicinski	|      |      |      |                        |
123*132db935SJakub Kicinski	p      p      p      p                        |
124*132db935SJakub Kicinski	3      2      0-1, 4-7  <- L2 priority        |
125*132db935SJakub Kicinski	|      |      |      |                        |
126*132db935SJakub Kicinski	|      |      |      |                        |
127*132db935SJakub Kicinski    +---|------|------|------|------------------------|----------------+
128*132db935SJakub Kicinski    |   |      |      |      |             |----------+                |
129*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+       +----+                         |
130*132db935SJakub Kicinski    | |dma7| |dma6| |dma5| |dma4|       |dma3|                         |
131*132db935SJakub Kicinski    | \    / \    / \    / \    /       \    /                         | c
132*132db935SJakub Kicinski    |  \S /   \S /   \  /   \  /         \  /                          | p
133*132db935SJakub Kicinski    |   \/     \/     \/     \/           \/                           | s
134*132db935SJakub Kicinski    |   |      |      | +-----            |                            | w
135*132db935SJakub Kicinski    |   |      |      | |                 |                            |
136*132db935SJakub Kicinski    |   |      |      | |                 |                            | d
137*132db935SJakub Kicinski    | +----+ +----+ +----+p            p+----+                         | r
138*132db935SJakub Kicinski    | |    | |    | |    |o            o|    |                         | i
139*132db935SJakub Kicinski    | | f3 | | f2 | | f0 |r            r| f0 |                         | v
140*132db935SJakub Kicinski    | |tc0 | |tc1 | |tc2 |t            t|tc0 |                         | e
141*132db935SJakub Kicinski    | \CBS / \CBS / \CBS /1            2\CBS /                         | r
142*132db935SJakub Kicinski    |  \S /   \S /   \  /                \  /                          |
143*132db935SJakub Kicinski    |   \/     \/     \/                  \/                           |
144*132db935SJakub Kicinski    +------------------------------------------------------------------+
145*132db935SJakub Kicinski
146*132db935SJakub Kicinski
147*132db935SJakub Kicinski1) ::
148*132db935SJakub Kicinski
149*132db935SJakub Kicinski
150*132db935SJakub Kicinski	// Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1
151*132db935SJakub Kicinski	$ ethtool -L eth0 rx 1 tx 5
152*132db935SJakub Kicinski	rx unmodified, ignoring
153*132db935SJakub Kicinski
154*132db935SJakub Kicinski2) ::
155*132db935SJakub Kicinski
156*132db935SJakub Kicinski	// Check if num of queues is set correctly:
157*132db935SJakub Kicinski	$ ethtool -l eth0
158*132db935SJakub Kicinski	Channel parameters for eth0:
159*132db935SJakub Kicinski	Pre-set maximums:
160*132db935SJakub Kicinski	RX:             8
161*132db935SJakub Kicinski	TX:             8
162*132db935SJakub Kicinski	Other:          0
163*132db935SJakub Kicinski	Combined:       0
164*132db935SJakub Kicinski	Current hardware settings:
165*132db935SJakub Kicinski	RX:             1
166*132db935SJakub Kicinski	TX:             5
167*132db935SJakub Kicinski	Other:          0
168*132db935SJakub Kicinski	Combined:       0
169*132db935SJakub Kicinski
170*132db935SJakub Kicinski3) ::
171*132db935SJakub Kicinski
172*132db935SJakub Kicinski	// TX queues must be rated starting from 0, so set bws for tx0 and tx1
173*132db935SJakub Kicinski	// Set rates 40 and 20 Mb/s appropriately.
174*132db935SJakub Kicinski	// Pay attention, real speed can differ a bit due to discreetness.
175*132db935SJakub Kicinski	// Leave last 2 tx queues not rated.
176*132db935SJakub Kicinski	$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
177*132db935SJakub Kicinski	$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
178*132db935SJakub Kicinski
179*132db935SJakub Kicinski4) ::
180*132db935SJakub Kicinski
181*132db935SJakub Kicinski	// Check maximum rate of tx (cpdma) queues:
182*132db935SJakub Kicinski	$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
183*132db935SJakub Kicinski	40
184*132db935SJakub Kicinski	20
185*132db935SJakub Kicinski	0
186*132db935SJakub Kicinski	0
187*132db935SJakub Kicinski	0
188*132db935SJakub Kicinski
189*132db935SJakub Kicinski5) ::
190*132db935SJakub Kicinski
191*132db935SJakub Kicinski	// Map skb->priority to traffic class:
192*132db935SJakub Kicinski	// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
193*132db935SJakub Kicinski	// Map traffic class to transmit queue:
194*132db935SJakub Kicinski	// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3)
195*132db935SJakub Kicinski	$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
196*132db935SJakub Kicinski	map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1
197*132db935SJakub Kicinski
198*132db935SJakub Kicinski5a) ::
199*132db935SJakub Kicinski
200*132db935SJakub Kicinski	// As two interface sharing same set of tx queues, assign all traffic
201*132db935SJakub Kicinski	// coming to interface Eth1 to separate queue in order to not mix it
202*132db935SJakub Kicinski	// with traffic from interface Eth0, so use separate txq to send
203*132db935SJakub Kicinski	// packets to Eth1, so all prio -> tc0 and tc0 -> txq4
204*132db935SJakub Kicinski	// Here hw 0, so here still default configuration for eth1 in hw
205*132db935SJakub Kicinski	$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \
206*132db935SJakub Kicinski	map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0
207*132db935SJakub Kicinski
208*132db935SJakub Kicinski6) ::
209*132db935SJakub Kicinski
210*132db935SJakub Kicinski	// Check classes settings
211*132db935SJakub Kicinski	$ tc -g class show dev eth0
212*132db935SJakub Kicinski	+---(100:ffe2) mqprio
213*132db935SJakub Kicinski	|    +---(100:3) mqprio
214*132db935SJakub Kicinski	|    +---(100:4) mqprio
215*132db935SJakub Kicinski	|
216*132db935SJakub Kicinski	+---(100:ffe1) mqprio
217*132db935SJakub Kicinski	|    +---(100:2) mqprio
218*132db935SJakub Kicinski	|
219*132db935SJakub Kicinski	+---(100:ffe0) mqprio
220*132db935SJakub Kicinski	    +---(100:1) mqprio
221*132db935SJakub Kicinski
222*132db935SJakub Kicinski	$ tc -g class show dev eth1
223*132db935SJakub Kicinski	+---(100:ffe0) mqprio
224*132db935SJakub Kicinski	    +---(100:5) mqprio
225*132db935SJakub Kicinski
226*132db935SJakub Kicinski7) ::
227*132db935SJakub Kicinski
228*132db935SJakub Kicinski	// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc
229*132db935SJakub Kicinski	// Set it +1 Mb for reserve (important!)
230*132db935SJakub Kicinski	// here only idle slope is important, others arg are ignored
231*132db935SJakub Kicinski	// Pay attention, real speed can differ a bit due to discreetness
232*132db935SJakub Kicinski	$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \
233*132db935SJakub Kicinski	hicredit 62 sendslope -959000 idleslope 41000 offload 1
234*132db935SJakub Kicinski	net eth0: set FIFO3 bw = 50
235*132db935SJakub Kicinski
236*132db935SJakub Kicinski8) ::
237*132db935SJakub Kicinski
238*132db935SJakub Kicinski	// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc:
239*132db935SJakub Kicinski	// Set it +1 Mb for reserve (important!)
240*132db935SJakub Kicinski	$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \
241*132db935SJakub Kicinski	hicredit 65 sendslope -979000 idleslope 21000 offload 1
242*132db935SJakub Kicinski	net eth0: set FIFO2 bw = 30
243*132db935SJakub Kicinski
244*132db935SJakub Kicinski9) ::
245*132db935SJakub Kicinski
246*132db935SJakub Kicinski	// Create vlan 100 to map sk->priority to vlan qos
247*132db935SJakub Kicinski	$ ip link add link eth0 name eth0.100 type vlan id 100
248*132db935SJakub Kicinski	8021q: 802.1Q VLAN Support v1.8
249*132db935SJakub Kicinski	8021q: adding VLAN 0 to HW filter on device eth0
250*132db935SJakub Kicinski	8021q: adding VLAN 0 to HW filter on device eth1
251*132db935SJakub Kicinski	net eth0: Adding vlanid 100 to vlan filter
252*132db935SJakub Kicinski
253*132db935SJakub Kicinski10) ::
254*132db935SJakub Kicinski
255*132db935SJakub Kicinski	// Map skb->priority to L2 prio, 1 to 1
256*132db935SJakub Kicinski	$ ip link set eth0.100 type vlan \
257*132db935SJakub Kicinski	egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
258*132db935SJakub Kicinski
259*132db935SJakub Kicinski11) ::
260*132db935SJakub Kicinski
261*132db935SJakub Kicinski	// Check egress map for vlan 100
262*132db935SJakub Kicinski	$ cat /proc/net/vlan/eth0.100
263*132db935SJakub Kicinski	[...]
264*132db935SJakub Kicinski	INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
265*132db935SJakub Kicinski	EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
266*132db935SJakub Kicinski
267*132db935SJakub Kicinski12) ::
268*132db935SJakub Kicinski
269*132db935SJakub Kicinski	// Run your appropriate tools with socket option "SO_PRIORITY"
270*132db935SJakub Kicinski	// to 3 for class A and/or to 2 for class B
271*132db935SJakub Kicinski	// (I took at https://www.spinics.net/lists/netdev/msg460869.html)
272*132db935SJakub Kicinski	./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
273*132db935SJakub Kicinski	./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
274*132db935SJakub Kicinski
275*132db935SJakub Kicinski13) ::
276*132db935SJakub Kicinski
277*132db935SJakub Kicinski	// run your listener on workstation (should be in same vlan)
278*132db935SJakub Kicinski	// (I took at https://www.spinics.net/lists/netdev/msg460869.html)
279*132db935SJakub Kicinski	./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
280*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
281*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
282*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
283*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
284*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
285*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
286*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
287*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
288*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
289*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
290*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
291*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
292*132db935SJakub Kicinski	Receiving data rate: 39000 kbps
293*132db935SJakub Kicinski
294*132db935SJakub Kicinski14) ::
295*132db935SJakub Kicinski
296*132db935SJakub Kicinski	// Restore default configuration if needed
297*132db935SJakub Kicinski	$ ip link del eth0.100
298*132db935SJakub Kicinski	$ tc qdisc del dev eth1 root
299*132db935SJakub Kicinski	$ tc qdisc del dev eth0 root
300*132db935SJakub Kicinski	net eth0: Prev FIFO2 is shaped
301*132db935SJakub Kicinski	net eth0: set FIFO3 bw = 0
302*132db935SJakub Kicinski	net eth0: set FIFO2 bw = 0
303*132db935SJakub Kicinski	$ ethtool -L eth0 rx 1 tx 1
304*132db935SJakub Kicinski
305*132db935SJakub KicinskiExample 2: Two port tx AVB configuration scheme for target board
306*132db935SJakub Kicinski----------------------------------------------------------------
307*132db935SJakub Kicinski
308*132db935SJakub Kicinski(prints and scheme for AM572x evm, for dual emac boards only)
309*132db935SJakub Kicinski
310*132db935SJakub Kicinski::
311*132db935SJakub Kicinski
312*132db935SJakub Kicinski    +------------------------------------------------------------------+ u
313*132db935SJakub Kicinski    | +----------+  +----------+  +------+  +----------+  +----------+ | s
314*132db935SJakub Kicinski    | |          |  |          |  |      |  |          |  |          | | e
315*132db935SJakub Kicinski    | | App 1    |  | App 2    |  | Apps |  | App 3    |  | App 4    | | r
316*132db935SJakub Kicinski    | | Class A  |  | Class B  |  | Rest |  | Class B  |  | Class A  | |
317*132db935SJakub Kicinski    | | Eth0     |  | Eth0     |  |   |  |  | Eth1     |  | Eth1     | | s
318*132db935SJakub Kicinski    | | VLAN100  |  | VLAN100  |  |   |  |  | VLAN100  |  | VLAN100  | | p
319*132db935SJakub Kicinski    | | 40 Mb/s  |  | 20 Mb/s  |  |   |  |  | 10 Mb/s  |  | 30 Mb/s  | | a
320*132db935SJakub Kicinski    | | SO_PRI=3 |  | SO_PRI=2 |  |   |  |  | SO_PRI=3 |  | SO_PRI=2 | | c
321*132db935SJakub Kicinski    | |   |      |  |   |      |  |   |  |  |   |      |  |   |      | | e
322*132db935SJakub Kicinski    | +---|------+  +---|------+  +---|--+  +---|------+  +---|------+ |
323*132db935SJakub Kicinski    +-----|-------------|-------------|---------|-------------|--------+
324*132db935SJakub Kicinski	+-+     +-------+             |         +----------+  +----+
325*132db935SJakub Kicinski	|       |             +-------+------+             |       |
326*132db935SJakub Kicinski	|       |             |              |             |       |
327*132db935SJakub Kicinski    +---|-------|-------------|--------------|-------------|-------|---+
328*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+          +----+ +----+ +----+ +----+ |
329*132db935SJakub Kicinski    | | p3 | | p2 | | p1 | | p0 |          | p0 | | p1 | | p2 | | p3 | | k
330*132db935SJakub Kicinski    | \    / \    / \    / \    /          \    / \    / \    / \    / | e
331*132db935SJakub Kicinski    |  \  /   \  /   \  /   \  /            \  /   \  /   \  /   \  /  | r
332*132db935SJakub Kicinski    |   \/     \/     \/     \/              \/     \/     \/     \/   | n
333*132db935SJakub Kicinski    |   |      |             |                |             |      |   | e
334*132db935SJakub Kicinski    |   |      |        +----+                +----+        |      |   | l
335*132db935SJakub Kicinski    |   |      |        |                          |        |      |   |
336*132db935SJakub Kicinski    | +----+ +----+ +----+                        +----+ +----+ +----+ | s
337*132db935SJakub Kicinski    | |tc0 | |tc1 | |tc2 |                        |tc2 | |tc1 | |tc0 | | p
338*132db935SJakub Kicinski    | \    / \    / \    /                        \    / \    / \    / | a
339*132db935SJakub Kicinski    |  \  /   \  /   \  /                          \  /   \  /   \  /  | c
340*132db935SJakub Kicinski    |   \/     \/     \/                            \/     \/     \/   | e
341*132db935SJakub Kicinski    |   |      |       +-----+                +-----+      |       |   |
342*132db935SJakub Kicinski    |   |      |       |     |                |     |      |       |   |
343*132db935SJakub Kicinski    |   |      |       |     |                |     |      |       |   |
344*132db935SJakub Kicinski    |   |      |       |     |    E      E    |     |      |       |   |
345*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+ t      t +----+ +----+ +----+ +----+ |
346*132db935SJakub Kicinski    | |txq0| |txq1| |txq4| |txq5| h      h |txq6| |txq7| |txq3| |txq2| |
347*132db935SJakub Kicinski    | \    / \    / \    / \    / 0      1 \    / \    / \    / \    / |
348*132db935SJakub Kicinski    |  \  /   \  /   \  /   \  /  .      .  \  /   \  /   \  /   \  /  |
349*132db935SJakub Kicinski    |   \/     \/     \/     \/   1      1   \/     \/     \/     \/   |
350*132db935SJakub Kicinski    | +-|------|------|------|--+ 0      0 +-|------|------|------|--+ |
351*132db935SJakub Kicinski    | | |      |      |      |  | 0      0 | |      |      |      |  | |
352*132db935SJakub Kicinski    +---|------|------|------|---------------|------|------|------|----+
353*132db935SJakub Kicinski	|      |      |      |               |      |      |      |
354*132db935SJakub Kicinski	p      p      p      p               p      p      p      p
355*132db935SJakub Kicinski	3      2      0-1, 4-7   <-L2 pri->  0-1, 4-7      2      3
356*132db935SJakub Kicinski	|      |      |      |               |      |      |      |
357*132db935SJakub Kicinski	|      |      |      |               |      |      |      |
358*132db935SJakub Kicinski    +---|------|------|------|---------------|------|------|------|----+
359*132db935SJakub Kicinski    |   |      |      |      |               |      |      |      |    |
360*132db935SJakub Kicinski    | +----+ +----+ +----+ +----+          +----+ +----+ +----+ +----+ |
361*132db935SJakub Kicinski    | |dma7| |dma6| |dma3| |dma2|          |dma1| |dma0| |dma4| |dma5| |
362*132db935SJakub Kicinski    | \    / \    / \    / \    /          \    / \    / \    / \    / | c
363*132db935SJakub Kicinski    |  \S /   \S /   \  /   \  /            \  /   \  /   \S /   \S /  | p
364*132db935SJakub Kicinski    |   \/     \/     \/     \/              \/     \/     \/     \/   | s
365*132db935SJakub Kicinski    |   |      |      | +-----                |      |      |      |   | w
366*132db935SJakub Kicinski    |   |      |      | |                     +----+ |      |      |   |
367*132db935SJakub Kicinski    |   |      |      | |                          | |      |      |   | d
368*132db935SJakub Kicinski    | +----+ +----+ +----+p                      p+----+ +----+ +----+ | r
369*132db935SJakub Kicinski    | |    | |    | |    |o                      o|    | |    | |    | | i
370*132db935SJakub Kicinski    | | f3 | | f2 | | f0 |r        CPSW          r| f3 | | f2 | | f0 | | v
371*132db935SJakub Kicinski    | |tc0 | |tc1 | |tc2 |t                      t|tc0 | |tc1 | |tc2 | | e
372*132db935SJakub Kicinski    | \CBS / \CBS / \CBS /1                      2\CBS / \CBS / \CBS / | r
373*132db935SJakub Kicinski    |  \S /   \S /   \  /                          \S /   \S /   \  /  |
374*132db935SJakub Kicinski    |   \/     \/     \/                            \/     \/     \/   |
375*132db935SJakub Kicinski    +------------------------------------------------------------------+
376*132db935SJakub Kicinski    ========================================Eth==========================>
377*132db935SJakub Kicinski
378*132db935SJakub Kicinski1) ::
379*132db935SJakub Kicinski
380*132db935SJakub Kicinski	// Add 8 tx queues, for interface Eth0, but they are common, so are accessed
381*132db935SJakub Kicinski	// by two interfaces Eth0 and Eth1.
382*132db935SJakub Kicinski	$ ethtool -L eth1 rx 1 tx 8
383*132db935SJakub Kicinski	rx unmodified, ignoring
384*132db935SJakub Kicinski
385*132db935SJakub Kicinski2) ::
386*132db935SJakub Kicinski
387*132db935SJakub Kicinski	// Check if num of queues is set correctly:
388*132db935SJakub Kicinski	$ ethtool -l eth0
389*132db935SJakub Kicinski	Channel parameters for eth0:
390*132db935SJakub Kicinski	Pre-set maximums:
391*132db935SJakub Kicinski	RX:             8
392*132db935SJakub Kicinski	TX:             8
393*132db935SJakub Kicinski	Other:          0
394*132db935SJakub Kicinski	Combined:       0
395*132db935SJakub Kicinski	Current hardware settings:
396*132db935SJakub Kicinski	RX:             1
397*132db935SJakub Kicinski	TX:             8
398*132db935SJakub Kicinski	Other:          0
399*132db935SJakub Kicinski	Combined:       0
400*132db935SJakub Kicinski
401*132db935SJakub Kicinski3) ::
402*132db935SJakub Kicinski
403*132db935SJakub Kicinski	// TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0
404*132db935SJakub Kicinski	// and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately
405*132db935SJakub Kicinski	// for Eth0 and 30 and 10 Mb/s for Eth1.
406*132db935SJakub Kicinski	// Real speed can differ a bit due to discreetness
407*132db935SJakub Kicinski	// Leave last 4 tx queues as not rated
408*132db935SJakub Kicinski	$ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
409*132db935SJakub Kicinski	$ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
410*132db935SJakub Kicinski	$ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate
411*132db935SJakub Kicinski	$ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate
412*132db935SJakub Kicinski
413*132db935SJakub Kicinski4) ::
414*132db935SJakub Kicinski
415*132db935SJakub Kicinski	// Check maximum rate of tx (cpdma) queues:
416*132db935SJakub Kicinski	$ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
417*132db935SJakub Kicinski	40
418*132db935SJakub Kicinski	20
419*132db935SJakub Kicinski	30
420*132db935SJakub Kicinski	10
421*132db935SJakub Kicinski	0
422*132db935SJakub Kicinski	0
423*132db935SJakub Kicinski	0
424*132db935SJakub Kicinski	0
425*132db935SJakub Kicinski
426*132db935SJakub Kicinski5) ::
427*132db935SJakub Kicinski
428*132db935SJakub Kicinski	// Map skb->priority to traffic class for Eth0:
429*132db935SJakub Kicinski	// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
430*132db935SJakub Kicinski	// Map traffic class to transmit queue:
431*132db935SJakub Kicinski	// tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5)
432*132db935SJakub Kicinski	$ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
433*132db935SJakub Kicinski	map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1
434*132db935SJakub Kicinski
435*132db935SJakub Kicinski6) ::
436*132db935SJakub Kicinski
437*132db935SJakub Kicinski	// Check classes settings
438*132db935SJakub Kicinski	$ tc -g class show dev eth0
439*132db935SJakub Kicinski	+---(100:ffe2) mqprio
440*132db935SJakub Kicinski	|    +---(100:5) mqprio
441*132db935SJakub Kicinski	|    +---(100:6) mqprio
442*132db935SJakub Kicinski	|
443*132db935SJakub Kicinski	+---(100:ffe1) mqprio
444*132db935SJakub Kicinski	|    +---(100:2) mqprio
445*132db935SJakub Kicinski	|
446*132db935SJakub Kicinski	+---(100:ffe0) mqprio
447*132db935SJakub Kicinski	    +---(100:1) mqprio
448*132db935SJakub Kicinski
449*132db935SJakub Kicinski7) ::
450*132db935SJakub Kicinski
451*132db935SJakub Kicinski	// Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0
452*132db935SJakub Kicinski	// here only idle slope is important, others ignored
453*132db935SJakub Kicinski	// Real speed can differ a bit due to discreetness
454*132db935SJakub Kicinski	$ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \
455*132db935SJakub Kicinski	hicredit 62 sendslope -959000 idleslope 41000 offload 1
456*132db935SJakub Kicinski	net eth0: set FIFO3 bw = 50
457*132db935SJakub Kicinski
458*132db935SJakub Kicinski8) ::
459*132db935SJakub Kicinski
460*132db935SJakub Kicinski	// Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0
461*132db935SJakub Kicinski	$ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \
462*132db935SJakub Kicinski	hicredit 65 sendslope -979000 idleslope 21000 offload 1
463*132db935SJakub Kicinski	net eth0: set FIFO2 bw = 30
464*132db935SJakub Kicinski
465*132db935SJakub Kicinski9) ::
466*132db935SJakub Kicinski
467*132db935SJakub Kicinski	// Create vlan 100 to map sk->priority to vlan qos for Eth0
468*132db935SJakub Kicinski	$ ip link add link eth0 name eth0.100 type vlan id 100
469*132db935SJakub Kicinski	net eth0: Adding vlanid 100 to vlan filter
470*132db935SJakub Kicinski
471*132db935SJakub Kicinski10) ::
472*132db935SJakub Kicinski
473*132db935SJakub Kicinski	// Map skb->priority to L2 prio for Eth0.100, one to one
474*132db935SJakub Kicinski	$ ip link set eth0.100 type vlan \
475*132db935SJakub Kicinski	egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
476*132db935SJakub Kicinski
477*132db935SJakub Kicinski11) ::
478*132db935SJakub Kicinski
479*132db935SJakub Kicinski	// Check egress map for vlan 100
480*132db935SJakub Kicinski	$ cat /proc/net/vlan/eth0.100
481*132db935SJakub Kicinski	[...]
482*132db935SJakub Kicinski	INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
483*132db935SJakub Kicinski	EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
484*132db935SJakub Kicinski
485*132db935SJakub Kicinski12) ::
486*132db935SJakub Kicinski
487*132db935SJakub Kicinski	// Map skb->priority to traffic class for Eth1:
488*132db935SJakub Kicinski	// 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
489*132db935SJakub Kicinski	// Map traffic class to transmit queue:
490*132db935SJakub Kicinski	// tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7)
491*132db935SJakub Kicinski	$ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \
492*132db935SJakub Kicinski	map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1
493*132db935SJakub Kicinski
494*132db935SJakub Kicinski13) ::
495*132db935SJakub Kicinski
496*132db935SJakub Kicinski	// Check classes settings
497*132db935SJakub Kicinski	$ tc -g class show dev eth1
498*132db935SJakub Kicinski	+---(100:ffe2) mqprio
499*132db935SJakub Kicinski	|    +---(100:7) mqprio
500*132db935SJakub Kicinski	|    +---(100:8) mqprio
501*132db935SJakub Kicinski	|
502*132db935SJakub Kicinski	+---(100:ffe1) mqprio
503*132db935SJakub Kicinski	|    +---(100:4) mqprio
504*132db935SJakub Kicinski	|
505*132db935SJakub Kicinski	+---(100:ffe0) mqprio
506*132db935SJakub Kicinski	    +---(100:3) mqprio
507*132db935SJakub Kicinski
508*132db935SJakub Kicinski14) ::
509*132db935SJakub Kicinski
510*132db935SJakub Kicinski	// Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1
511*132db935SJakub Kicinski	// here only idle slope is important, others ignored, but calculated
512*132db935SJakub Kicinski	// for interface speed - 100Mb for eth1 port.
513*132db935SJakub Kicinski	// Set it +1 Mb for reserve (important!)
514*132db935SJakub Kicinski	$ tc qdisc add dev eth1 parent 100:3 cbs locredit -1035 \
515*132db935SJakub Kicinski	hicredit 465 sendslope -69000 idleslope 31000 offload 1
516*132db935SJakub Kicinski	net eth1: set FIFO3 bw = 31
517*132db935SJakub Kicinski
518*132db935SJakub Kicinski15) ::
519*132db935SJakub Kicinski
520*132db935SJakub Kicinski	// Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1
521*132db935SJakub Kicinski	// Set it +1 Mb for reserve (important!)
522*132db935SJakub Kicinski	$ tc qdisc add dev eth1 parent 100:4 cbs locredit -1335 \
523*132db935SJakub Kicinski	hicredit 405 sendslope -89000 idleslope 11000 offload 1
524*132db935SJakub Kicinski	net eth1: set FIFO2 bw = 11
525*132db935SJakub Kicinski
526*132db935SJakub Kicinski16) ::
527*132db935SJakub Kicinski
528*132db935SJakub Kicinski	// Create vlan 100 to map sk->priority to vlan qos for Eth1
529*132db935SJakub Kicinski	$ ip link add link eth1 name eth1.100 type vlan id 100
530*132db935SJakub Kicinski	net eth1: Adding vlanid 100 to vlan filter
531*132db935SJakub Kicinski
532*132db935SJakub Kicinski17) ::
533*132db935SJakub Kicinski
534*132db935SJakub Kicinski	// Map skb->priority to L2 prio for Eth1.100, one to one
535*132db935SJakub Kicinski	$ ip link set eth1.100 type vlan \
536*132db935SJakub Kicinski	egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
537*132db935SJakub Kicinski
538*132db935SJakub Kicinski18) ::
539*132db935SJakub Kicinski
540*132db935SJakub Kicinski	// Check egress map for vlan 100
541*132db935SJakub Kicinski	$ cat /proc/net/vlan/eth1.100
542*132db935SJakub Kicinski	[...]
543*132db935SJakub Kicinski	INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0
544*132db935SJakub Kicinski	EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
545*132db935SJakub Kicinski
546*132db935SJakub Kicinski19) ::
547*132db935SJakub Kicinski
548*132db935SJakub Kicinski	// Run appropriate tools with socket option "SO_PRIORITY" to 3
549*132db935SJakub Kicinski	// for class A and to 2 for class B. For both interfaces
550*132db935SJakub Kicinski	./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
551*132db935SJakub Kicinski	./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
552*132db935SJakub Kicinski	./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500&
553*132db935SJakub Kicinski	./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500&
554*132db935SJakub Kicinski
555*132db935SJakub Kicinski20) ::
556*132db935SJakub Kicinski
557*132db935SJakub Kicinski	// run your listener on workstation (should be in same vlan)
558*132db935SJakub Kicinski	// (I took at https://www.spinics.net/lists/netdev/msg460869.html)
559*132db935SJakub Kicinski	./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
560*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
561*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
562*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
563*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
564*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
565*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
566*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
567*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
568*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
569*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
570*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
571*132db935SJakub Kicinski	Receiving data rate: 39012 kbps
572*132db935SJakub Kicinski	Receiving data rate: 39000 kbps
573*132db935SJakub Kicinski
574*132db935SJakub Kicinski21) ::
575*132db935SJakub Kicinski
576*132db935SJakub Kicinski	// Restore default configuration if needed
577*132db935SJakub Kicinski	$ ip link del eth1.100
578*132db935SJakub Kicinski	$ ip link del eth0.100
579*132db935SJakub Kicinski	$ tc qdisc del dev eth1 root
580*132db935SJakub Kicinski	net eth1: Prev FIFO2 is shaped
581*132db935SJakub Kicinski	net eth1: set FIFO3 bw = 0
582*132db935SJakub Kicinski	net eth1: set FIFO2 bw = 0
583*132db935SJakub Kicinski	$ tc qdisc del dev eth0 root
584*132db935SJakub Kicinski	net eth0: Prev FIFO2 is shaped
585*132db935SJakub Kicinski	net eth0: set FIFO3 bw = 0
586*132db935SJakub Kicinski	net eth0: set FIFO2 bw = 0
587*132db935SJakub Kicinski	$ ethtool -L eth0 rx 1 tx 1
588