1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2
3====================================
4Marvell OcteonTx2 RVU Kernel Drivers
5====================================
6
7Copyright (c) 2020 Marvell International Ltd.
8
9Contents
10========
11
12- `Overview`_
13- `Drivers`_
14- `Basic packet flow`_
15
16Overview
17========
18
19Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
20resources from the network, crypto and other functional blocks into
21PCI-compatible physical and virtual functions. Each functional block
22again has multiple local functions (LFs) for provisioning to PCI devices.
23RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
24functions (VFs). PF0 is called the administrative / admin function (AF)
25and has privileges to provision RVU functional block's LFs to each of the
26PF/VF.
27
28RVU managed networking functional blocks
29 - Network pool or buffer allocator (NPA)
30 - Network interface controller (NIX)
31 - Network parser CAM (NPC)
32 - Schedule/Synchronize/Order unit (SSO)
33 - Loopback interface (LBK)
34
35RVU managed non-networking functional blocks
36 - Crypto accelerator (CPT)
37 - Scheduled timers unit (TIM)
38 - Schedule/Synchronize/Order unit (SSO)
39   Used for both networking and non networking usecases
40
41Resource provisioning examples
42 - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
43 - A PF/VF with CPT-LF resource works as a pure crypto offload device.
44
45RVU functional blocks are highly configurable as per software requirements.
46
47Firmware setups following stuff before kernel boots
48 - Enables required number of RVU PFs based on number of physical links.
49 - Number of VFs per PF are either static or configurable at compile time.
50   Based on config, firmware assigns VFs to each of the PFs.
51 - Also assigns MSIX vectors to each of PF and VFs.
52 - These are not changed after kernel boot.
53
54Drivers
55=======
56
57Linux kernel will have multiple drivers registering to different PF and VFs
58of RVU. Wrt networking there will be 3 flavours of drivers.
59
60Admin Function driver
61---------------------
62
63As mentioned above RVU PF0 is called the admin function (AF), this driver
64supports resource provisioning and configuration of functional blocks.
65Doesn't handle any I/O. It sets up few basic stuff but most of the
66funcionality is achieved via configuration requests from PFs and VFs.
67
68PF/VFs communicates with AF via a shared memory region (mailbox). Upon
69receiving requests AF does resource provisioning and other HW configuration.
70AF is always attached to host kernel, but PFs and their VFs may be used by host
71kernel itself, or attached to VMs or to userspace applications like
72DPDK etc. So AF has to handle provisioning/configuration requests sent
73by any device from any domain.
74
75AF driver also interacts with underlying firmware to
76 - Manage physical ethernet links ie CGX LMACs.
77 - Retrieve information like speed, duplex, autoneg etc
78 - Retrieve PHY EEPROM and stats.
79 - Configure FEC, PAM modes
80 - etc
81
82From pure networking side AF driver supports following functionality.
83 - Map a physical link to a RVU PF to which a netdev is registered.
84 - Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs
85   for regular networking functionality.
86 - Flow control (pause frames) enable/disable/config.
87 - HW PTP timestamping related config.
88 - NPC parser profile config, basically how to parse pkt and what info to extract.
89 - NPC extract profile config, what to extract from the pkt to match data in MCAM entries.
90 - Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules.
91 - Defines receive side scaling (RSS) algorithms.
92 - Defines segmentation offload algorithms (eg TSO)
93 - VLAN stripping, capture and insertion config.
94 - SSO and TIM blocks config which provide packet scheduling support.
95 - Debugfs support, to check current resource provising, current status of
96   NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues.
97 - And many more.
98
99Physical Function driver
100------------------------
101
102This RVU PF handles IO, is mapped to a physical ethernet link and this
103driver registers a netdev. This supports SR-IOV. As said above this driver
104communicates with AF with a mailbox. To retrieve information from physical
105links this driver talks to AF and AF gets that info from firmware and responds
106back ie cannot talk to firmware directly.
107
108Supports ethtool for configuring links, RSS, queue count, queue size,
109flow control, ntuple filters, dump PHY EEPROM, config FEC etc.
110
111Virtual Function driver
112-----------------------
113
114There are two types VFs, VFs that share the physical link with their parent
115SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK).
116
117Type1:
118 - These VFs and their parent PF share a physical link and used for outside communication.
119 - VFs cannot communicate with AF directly, they send mbox message to PF and PF
120   forwards that to AF. AF after processing, responds back to PF and PF forwards
121   the reply to VF.
122 - From functionality point of view there is no difference between PF and VF as same type
123   HW resources are attached to both. But user would be able to configure few stuff only
124   from PF as PF is treated as owner/admin of the link.
125
126Type2:
127 - RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels.
128 - A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of
129   VF0 will be received by VF1 and viceversa.
130 - These VFs can be used by applications or virtual machines to communicate between them
131   without sending traffic outside. There is no switch present in HW, hence the support
132   for loopback VFs.
133 - These communicate directly with AF (PF0) via mbox.
134
135Except for the IO channels or links used for packet reception and transmission there is
136no other difference between these VF types. AF driver takes care of IO channel mapping,
137hence same VF driver works for both types of devices.
138
139Basic packet flow
140=================
141
142Ingress
143-------
144
1451. CGX LMAC receives packet.
1462. Forwards the packet to the NIX block.
1473. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device.
1484. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF.
1495. RQ may be selected by RSS or by configuring MCAM rule with a RQ number.
1506. Packet is DMA'ed and driver is notified.
151
152Egress
153------
154
1551. Driver prepares a send descriptor and submits to SQ for transmission.
1562. The SQ is already configured (by AF) to transmit on a specific link/channel.
1573. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF.
1584. NIX block transmits the pkt on the designated channel.
1595. NPC MCAM entries can be installed to divert pkt onto a different channel.
160