1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) 2 3==================================== 4Marvell OcteonTx2 RVU Kernel Drivers 5==================================== 6 7Copyright (c) 2020 Marvell International Ltd. 8 9Contents 10======== 11 12- `Overview`_ 13- `Drivers`_ 14- `Basic packet flow`_ 15- `Devlink health reporters`_ 16 17Overview 18======== 19 20Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW 21resources from the network, crypto and other functional blocks into 22PCI-compatible physical and virtual functions. Each functional block 23again has multiple local functions (LFs) for provisioning to PCI devices. 24RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual 25functions (VFs). PF0 is called the administrative / admin function (AF) 26and has privileges to provision RVU functional block's LFs to each of the 27PF/VF. 28 29RVU managed networking functional blocks 30 - Network pool or buffer allocator (NPA) 31 - Network interface controller (NIX) 32 - Network parser CAM (NPC) 33 - Schedule/Synchronize/Order unit (SSO) 34 - Loopback interface (LBK) 35 36RVU managed non-networking functional blocks 37 - Crypto accelerator (CPT) 38 - Scheduled timers unit (TIM) 39 - Schedule/Synchronize/Order unit (SSO) 40 Used for both networking and non networking usecases 41 42Resource provisioning examples 43 - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device 44 - A PF/VF with CPT-LF resource works as a pure crypto offload device. 45 46RVU functional blocks are highly configurable as per software requirements. 47 48Firmware setups following stuff before kernel boots 49 - Enables required number of RVU PFs based on number of physical links. 50 - Number of VFs per PF are either static or configurable at compile time. 51 Based on config, firmware assigns VFs to each of the PFs. 52 - Also assigns MSIX vectors to each of PF and VFs. 53 - These are not changed after kernel boot. 54 55Drivers 56======= 57 58Linux kernel will have multiple drivers registering to different PF and VFs 59of RVU. Wrt networking there will be 3 flavours of drivers. 60 61Admin Function driver 62--------------------- 63 64As mentioned above RVU PF0 is called the admin function (AF), this driver 65supports resource provisioning and configuration of functional blocks. 66Doesn't handle any I/O. It sets up few basic stuff but most of the 67funcionality is achieved via configuration requests from PFs and VFs. 68 69PF/VFs communicates with AF via a shared memory region (mailbox). Upon 70receiving requests AF does resource provisioning and other HW configuration. 71AF is always attached to host kernel, but PFs and their VFs may be used by host 72kernel itself, or attached to VMs or to userspace applications like 73DPDK etc. So AF has to handle provisioning/configuration requests sent 74by any device from any domain. 75 76AF driver also interacts with underlying firmware to 77 - Manage physical ethernet links ie CGX LMACs. 78 - Retrieve information like speed, duplex, autoneg etc 79 - Retrieve PHY EEPROM and stats. 80 - Configure FEC, PAM modes 81 - etc 82 83From pure networking side AF driver supports following functionality. 84 - Map a physical link to a RVU PF to which a netdev is registered. 85 - Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs 86 for regular networking functionality. 87 - Flow control (pause frames) enable/disable/config. 88 - HW PTP timestamping related config. 89 - NPC parser profile config, basically how to parse pkt and what info to extract. 90 - NPC extract profile config, what to extract from the pkt to match data in MCAM entries. 91 - Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules. 92 - Defines receive side scaling (RSS) algorithms. 93 - Defines segmentation offload algorithms (eg TSO) 94 - VLAN stripping, capture and insertion config. 95 - SSO and TIM blocks config which provide packet scheduling support. 96 - Debugfs support, to check current resource provising, current status of 97 NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues. 98 - And many more. 99 100Physical Function driver 101------------------------ 102 103This RVU PF handles IO, is mapped to a physical ethernet link and this 104driver registers a netdev. This supports SR-IOV. As said above this driver 105communicates with AF with a mailbox. To retrieve information from physical 106links this driver talks to AF and AF gets that info from firmware and responds 107back ie cannot talk to firmware directly. 108 109Supports ethtool for configuring links, RSS, queue count, queue size, 110flow control, ntuple filters, dump PHY EEPROM, config FEC etc. 111 112Virtual Function driver 113----------------------- 114 115There are two types VFs, VFs that share the physical link with their parent 116SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK). 117 118Type1: 119 - These VFs and their parent PF share a physical link and used for outside communication. 120 - VFs cannot communicate with AF directly, they send mbox message to PF and PF 121 forwards that to AF. AF after processing, responds back to PF and PF forwards 122 the reply to VF. 123 - From functionality point of view there is no difference between PF and VF as same type 124 HW resources are attached to both. But user would be able to configure few stuff only 125 from PF as PF is treated as owner/admin of the link. 126 127Type2: 128 - RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels. 129 - A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of 130 VF0 will be received by VF1 and viceversa. 131 - These VFs can be used by applications or virtual machines to communicate between them 132 without sending traffic outside. There is no switch present in HW, hence the support 133 for loopback VFs. 134 - These communicate directly with AF (PF0) via mbox. 135 136Except for the IO channels or links used for packet reception and transmission there is 137no other difference between these VF types. AF driver takes care of IO channel mapping, 138hence same VF driver works for both types of devices. 139 140Basic packet flow 141================= 142 143Ingress 144------- 145 1461. CGX LMAC receives packet. 1472. Forwards the packet to the NIX block. 1483. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device. 1494. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF. 1505. RQ may be selected by RSS or by configuring MCAM rule with a RQ number. 1516. Packet is DMA'ed and driver is notified. 152 153Egress 154------ 155 1561. Driver prepares a send descriptor and submits to SQ for transmission. 1572. The SQ is already configured (by AF) to transmit on a specific link/channel. 1583. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF. 1594. NIX block transmits the pkt on the designated channel. 1605. NPC MCAM entries can be installed to divert pkt onto a different channel. 161 162Devlink health reporters 163======================== 164 165NPA Reporters 166------------- 167The NPA reporters are responsible for reporting and recovering the following group of errors 1681. GENERAL events 169 - Error due to operation of unmapped PF. 170 - Error due to disabled alloc/free for other HW blocks (NIX, SSO, TIM, DPI and AURA). 1712. ERROR events 172 - Fault due to NPA_AQ_INST_S read or NPA_AQ_RES_S write. 173 - AQ Doorbell Error. 1743. RAS events 175 - RAS Error Reporting for NPA_AQ_INST_S/NPA_AQ_RES_S. 1764. RVU events 177 - Error due to unmapped slot. 178 179Sample Output 180------------- 181~# devlink health 182pci/0002:01:00.0: 183 reporter hw_npa_intr 184 state healthy error 2872 recover 2872 last_dump_date 2020-12-10 last_dump_time 09:39:09 grace_period 0 auto_recover true auto_dump true 185 reporter hw_npa_gen 186 state healthy error 2872 recover 2872 last_dump_date 2020-12-11 last_dump_time 04:43:04 grace_period 0 auto_recover true auto_dump true 187 reporter hw_npa_err 188 state healthy error 2871 recover 2871 last_dump_date 2020-12-10 last_dump_time 09:39:17 grace_period 0 auto_recover true auto_dump true 189 reporter hw_npa_ras 190 state healthy error 0 recover 0 last_dump_date 2020-12-10 last_dump_time 09:32:40 grace_period 0 auto_recover true auto_dump true 191 192Each reporter dumps the 193 - Error Type 194 - Error Register value 195 - Reason in words 196 197For eg: 198~# devlink health dump show pci/0002:01:00.0 reporter hw_npa_gen 199 NPA_AF_GENERAL: 200 NPA General Interrupt Reg : 1 201 NIX0: free disabled RX 202~# devlink health dump show pci/0002:01:00.0 reporter hw_npa_intr 203 NPA_AF_RVU: 204 NPA RVU Interrupt Reg : 1 205 Unmap Slot Error 206~# devlink health dump show pci/0002:01:00.0 reporter hw_npa_err 207 NPA_AF_ERR: 208 NPA Error Interrupt Reg : 4096 209 AQ Doorbell Error 210