1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2
3====================================
4Marvell OcteonTx2 RVU Kernel Drivers
5====================================
6
7Copyright (c) 2020 Marvell International Ltd.
8
9Contents
10========
11
12- `Overview`_
13- `Drivers`_
14- `Basic packet flow`_
15- `Devlink health reporters`_
16
17Overview
18========
19
20Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
21resources from the network, crypto and other functional blocks into
22PCI-compatible physical and virtual functions. Each functional block
23again has multiple local functions (LFs) for provisioning to PCI devices.
24RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
25functions (VFs). PF0 is called the administrative / admin function (AF)
26and has privileges to provision RVU functional block's LFs to each of the
27PF/VF.
28
29RVU managed networking functional blocks
30 - Network pool or buffer allocator (NPA)
31 - Network interface controller (NIX)
32 - Network parser CAM (NPC)
33 - Schedule/Synchronize/Order unit (SSO)
34 - Loopback interface (LBK)
35
36RVU managed non-networking functional blocks
37 - Crypto accelerator (CPT)
38 - Scheduled timers unit (TIM)
39 - Schedule/Synchronize/Order unit (SSO)
40   Used for both networking and non networking usecases
41
42Resource provisioning examples
43 - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
44 - A PF/VF with CPT-LF resource works as a pure crypto offload device.
45
46RVU functional blocks are highly configurable as per software requirements.
47
48Firmware setups following stuff before kernel boots
49 - Enables required number of RVU PFs based on number of physical links.
50 - Number of VFs per PF are either static or configurable at compile time.
51   Based on config, firmware assigns VFs to each of the PFs.
52 - Also assigns MSIX vectors to each of PF and VFs.
53 - These are not changed after kernel boot.
54
55Drivers
56=======
57
58Linux kernel will have multiple drivers registering to different PF and VFs
59of RVU. Wrt networking there will be 3 flavours of drivers.
60
61Admin Function driver
62---------------------
63
64As mentioned above RVU PF0 is called the admin function (AF), this driver
65supports resource provisioning and configuration of functional blocks.
66Doesn't handle any I/O. It sets up few basic stuff but most of the
67funcionality is achieved via configuration requests from PFs and VFs.
68
69PF/VFs communicates with AF via a shared memory region (mailbox). Upon
70receiving requests AF does resource provisioning and other HW configuration.
71AF is always attached to host kernel, but PFs and their VFs may be used by host
72kernel itself, or attached to VMs or to userspace applications like
73DPDK etc. So AF has to handle provisioning/configuration requests sent
74by any device from any domain.
75
76AF driver also interacts with underlying firmware to
77 - Manage physical ethernet links ie CGX LMACs.
78 - Retrieve information like speed, duplex, autoneg etc
79 - Retrieve PHY EEPROM and stats.
80 - Configure FEC, PAM modes
81 - etc
82
83From pure networking side AF driver supports following functionality.
84 - Map a physical link to a RVU PF to which a netdev is registered.
85 - Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs
86   for regular networking functionality.
87 - Flow control (pause frames) enable/disable/config.
88 - HW PTP timestamping related config.
89 - NPC parser profile config, basically how to parse pkt and what info to extract.
90 - NPC extract profile config, what to extract from the pkt to match data in MCAM entries.
91 - Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules.
92 - Defines receive side scaling (RSS) algorithms.
93 - Defines segmentation offload algorithms (eg TSO)
94 - VLAN stripping, capture and insertion config.
95 - SSO and TIM blocks config which provide packet scheduling support.
96 - Debugfs support, to check current resource provising, current status of
97   NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues.
98 - And many more.
99
100Physical Function driver
101------------------------
102
103This RVU PF handles IO, is mapped to a physical ethernet link and this
104driver registers a netdev. This supports SR-IOV. As said above this driver
105communicates with AF with a mailbox. To retrieve information from physical
106links this driver talks to AF and AF gets that info from firmware and responds
107back ie cannot talk to firmware directly.
108
109Supports ethtool for configuring links, RSS, queue count, queue size,
110flow control, ntuple filters, dump PHY EEPROM, config FEC etc.
111
112Virtual Function driver
113-----------------------
114
115There are two types VFs, VFs that share the physical link with their parent
116SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK).
117
118Type1:
119 - These VFs and their parent PF share a physical link and used for outside communication.
120 - VFs cannot communicate with AF directly, they send mbox message to PF and PF
121   forwards that to AF. AF after processing, responds back to PF and PF forwards
122   the reply to VF.
123 - From functionality point of view there is no difference between PF and VF as same type
124   HW resources are attached to both. But user would be able to configure few stuff only
125   from PF as PF is treated as owner/admin of the link.
126
127Type2:
128 - RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels.
129 - A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of
130   VF0 will be received by VF1 and viceversa.
131 - These VFs can be used by applications or virtual machines to communicate between them
132   without sending traffic outside. There is no switch present in HW, hence the support
133   for loopback VFs.
134 - These communicate directly with AF (PF0) via mbox.
135
136Except for the IO channels or links used for packet reception and transmission there is
137no other difference between these VF types. AF driver takes care of IO channel mapping,
138hence same VF driver works for both types of devices.
139
140Basic packet flow
141=================
142
143Ingress
144-------
145
1461. CGX LMAC receives packet.
1472. Forwards the packet to the NIX block.
1483. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device.
1494. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF.
1505. RQ may be selected by RSS or by configuring MCAM rule with a RQ number.
1516. Packet is DMA'ed and driver is notified.
152
153Egress
154------
155
1561. Driver prepares a send descriptor and submits to SQ for transmission.
1572. The SQ is already configured (by AF) to transmit on a specific link/channel.
1583. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF.
1594. NIX block transmits the pkt on the designated channel.
1605. NPC MCAM entries can be installed to divert pkt onto a different channel.
161
162Devlink health reporters
163========================
164
165NPA Reporters
166-------------
167The NPA reporters are responsible for reporting and recovering the following group of errors:
168
1691. GENERAL events
170
171   - Error due to operation of unmapped PF.
172   - Error due to disabled alloc/free for other HW blocks (NIX, SSO, TIM, DPI and AURA).
173
1742. ERROR events
175
176   - Fault due to NPA_AQ_INST_S read or NPA_AQ_RES_S write.
177   - AQ Doorbell Error.
178
1793. RAS events
180
181   - RAS Error Reporting for NPA_AQ_INST_S/NPA_AQ_RES_S.
182
1834. RVU events
184
185   - Error due to unmapped slot.
186
187Sample Output::
188
189	~# devlink health
190	pci/0002:01:00.0:
191	  reporter hw_npa_intr
192	      state healthy error 2872 recover 2872 last_dump_date 2020-12-10 last_dump_time 09:39:09 grace_period 0 auto_recover true auto_dump true
193	  reporter hw_npa_gen
194	      state healthy error 2872 recover 2872 last_dump_date 2020-12-11 last_dump_time 04:43:04 grace_period 0 auto_recover true auto_dump true
195	  reporter hw_npa_err
196	      state healthy error 2871 recover 2871 last_dump_date 2020-12-10 last_dump_time 09:39:17 grace_period 0 auto_recover true auto_dump true
197	   reporter hw_npa_ras
198	      state healthy error 0 recover 0 last_dump_date 2020-12-10 last_dump_time 09:32:40 grace_period 0 auto_recover true auto_dump true
199
200Each reporter dumps the
201
202 - Error Type
203 - Error Register value
204 - Reason in words
205
206For example::
207
208	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_gen
209	 NPA_AF_GENERAL:
210	         NPA General Interrupt Reg : 1
211	         NIX0: free disabled RX
212	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_intr
213	 NPA_AF_RVU:
214	         NPA RVU Interrupt Reg : 1
215	         Unmap Slot Error
216	~# devlink health dump show  pci/0002:01:00.0 reporter hw_npa_err
217	 NPA_AF_ERR:
218	        NPA Error Interrupt Reg : 4096
219	        AQ Doorbell Error
220
221
222NIX Reporters
223-------------
224The NIX reporters are responsible for reporting and recovering the following group of errors:
225
2261. GENERAL events
227
228   - Receive mirror/multicast packet drop due to insufficient buffer.
229   - SMQ Flush operation.
230
2312. ERROR events
232
233   - Memory Fault due to WQE read/write from multicast/mirror buffer.
234   - Receive multicast/mirror replication list error.
235   - Receive packet on an unmapped PF.
236   - Fault due to NIX_AQ_INST_S read or NIX_AQ_RES_S write.
237   - AQ Doorbell Error.
238
2393. RAS events
240
241   - RAS Error Reporting for NIX Receive Multicast/Mirror Entry Structure.
242   - RAS Error Reporting for WQE/Packet Data read from Multicast/Mirror Buffer..
243   - RAS Error Reporting for NIX_AQ_INST_S/NIX_AQ_RES_S.
244
2454. RVU events
246
247   - Error due to unmapped slot.
248
249Sample Output::
250
251	~# ./devlink health
252	pci/0002:01:00.0:
253	  reporter hw_npa_intr
254	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
255	  reporter hw_npa_gen
256	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
257	  reporter hw_npa_err
258	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
259	  reporter hw_npa_ras
260	    state healthy error 0 recover 0 grace_period 0 auto_recover true auto_dump true
261	  reporter hw_nix_intr
262	    state healthy error 1121 recover 1121 last_dump_date 2021-01-19 last_dump_time 05:42:26 grace_period 0 auto_recover true auto_dump true
263	  reporter hw_nix_gen
264	    state healthy error 949 recover 949 last_dump_date 2021-01-19 last_dump_time 05:42:43 grace_period 0 auto_recover true auto_dump true
265	  reporter hw_nix_err
266	    state healthy error 1147 recover 1147 last_dump_date 2021-01-19 last_dump_time 05:42:59 grace_period 0 auto_recover true auto_dump true
267	  reporter hw_nix_ras
268	    state healthy error 409 recover 409 last_dump_date 2021-01-19 last_dump_time 05:43:16 grace_period 0 auto_recover true auto_dump true
269
270Each reporter dumps the
271
272 - Error Type
273 - Error Register value
274 - Reason in words
275
276For example::
277
278	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_intr
279	 NIX_AF_RVU:
280	        NIX RVU Interrupt Reg : 1
281	        Unmap Slot Error
282	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_gen
283	 NIX_AF_GENERAL:
284	        NIX General Interrupt Reg : 1
285	        Rx multicast pkt drop
286	~# devlink health dump show pci/0002:01:00.0 reporter hw_nix_err
287	 NIX_AF_ERR:
288	        NIX Error Interrupt Reg : 64
289	        Rx on unmapped PF_FUNC
290