1132db935SJakub Kicinski.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) 2*1ec6360dSWalter Heymans.. include:: <isonum.txt> 3132db935SJakub Kicinski 4*1ec6360dSWalter Heymans=========================================== 5*1ec6360dSWalter HeymansNetwork Flow Processor (NFP) Kernel Drivers 6*1ec6360dSWalter Heymans=========================================== 7132db935SJakub Kicinski 8*1ec6360dSWalter Heymans:Copyright: |copy| 2019, Netronome Systems, Inc. 9*1ec6360dSWalter Heymans:Copyright: |copy| 2022, Corigine, Inc. 10132db935SJakub Kicinski 11132db935SJakub KicinskiContents 12132db935SJakub Kicinski======== 13132db935SJakub Kicinski 14132db935SJakub Kicinski- `Overview`_ 15132db935SJakub Kicinski- `Acquiring Firmware`_ 16*1ec6360dSWalter Heymans- `Devlink Info`_ 17*1ec6360dSWalter Heymans- `Configure Device`_ 18*1ec6360dSWalter Heymans- `Statistics`_ 19132db935SJakub Kicinski 20132db935SJakub KicinskiOverview 21132db935SJakub Kicinski======== 22132db935SJakub Kicinski 23*1ec6360dSWalter HeymansThis driver supports Netronome and Corigine's line of Network Flow Processor 24*1ec6360dSWalter Heymansdevices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which 25*1ec6360dSWalter Heymansare also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV 26*1ec6360dSWalter Heymansphysical and virtual functions for these devices are supported by the driver. 27132db935SJakub Kicinski 28132db935SJakub KicinskiAcquiring Firmware 29132db935SJakub Kicinski================== 30132db935SJakub Kicinski 31*1ec6360dSWalter HeymansThe NFP3800, NFP4000 and NFP6000 devices require application specific firmware 32132db935SJakub Kicinskito function. Application firmware can be located either on the host file system 33132db935SJakub Kicinskior in the device flash (if supported by management firmware). 34132db935SJakub Kicinski 35132db935SJakub KicinskiFirmware files on the host filesystem contain card type (`AMDA-*` string), media 36132db935SJakub Kicinskiconfig etc. They should be placed in `/lib/firmware/netronome` directory to 37132db935SJakub Kicinskiload firmware from the host file system. 38132db935SJakub Kicinski 39132db935SJakub KicinskiFirmware for basic NIC operation is available in the upstream 40132db935SJakub Kicinski`linux-firmware.git` repository. 41132db935SJakub Kicinski 42*1ec6360dSWalter HeymansA more comprehensive list of firmware can be downloaded from the 43*1ec6360dSWalter Heymans`Corigine Support site <https://www.corigine.com/DPUDownload.html>`_. 44*1ec6360dSWalter Heymans 45132db935SJakub KicinskiFirmware in NVRAM 46132db935SJakub Kicinski----------------- 47132db935SJakub Kicinski 48132db935SJakub KicinskiRecent versions of management firmware supports loading application 49132db935SJakub Kicinskifirmware from flash when the host driver gets probed. The firmware loading 50132db935SJakub Kicinskipolicy configuration may be used to configure this feature appropriately. 51132db935SJakub Kicinski 52132db935SJakub KicinskiDevlink or ethtool can be used to update the application firmware on the device 53132db935SJakub Kicinskiflash by providing the appropriate `nic_AMDA*.nffw` file to the respective 54132db935SJakub Kicinskicommand. Users need to take care to write the correct firmware image for the 55132db935SJakub Kicinskicard and media configuration to flash. 56132db935SJakub Kicinski 57132db935SJakub KicinskiAvailable storage space in flash depends on the card being used. 58132db935SJakub Kicinski 59132db935SJakub KicinskiDealing with multiple projects 60132db935SJakub Kicinski------------------------------ 61132db935SJakub Kicinski 62132db935SJakub KicinskiNFP hardware is fully programmable therefore there can be different 63132db935SJakub Kicinskifirmware images targeting different applications. 64132db935SJakub Kicinski 65132db935SJakub KicinskiWhen using application firmware from host, we recommend placing 66132db935SJakub Kicinskiactual firmware files in application-named subdirectories in 67132db935SJakub Kicinski`/lib/firmware/netronome` and linking the desired files, e.g.:: 68132db935SJakub Kicinski 69132db935SJakub Kicinski $ tree /lib/firmware/netronome/ 70132db935SJakub Kicinski /lib/firmware/netronome/ 71132db935SJakub Kicinski ├── bpf 72132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 73132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 74132db935SJakub Kicinski ├── flower 75132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 76132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 77132db935SJakub Kicinski ├── nic 78132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 79132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 80132db935SJakub Kicinski ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw 81132db935SJakub Kicinski └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw 82132db935SJakub Kicinski 83132db935SJakub Kicinski 3 directories, 8 files 84132db935SJakub Kicinski 85132db935SJakub KicinskiYou may need to use hard instead of symbolic links on distributions 86132db935SJakub Kicinskiwhich use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu). 87132db935SJakub Kicinski 88132db935SJakub KicinskiAfter changing firmware files you may need to regenerate the initramfs 89132db935SJakub Kicinskiimage. Initramfs contains drivers and firmware files your system may 90132db935SJakub Kicinskineed to boot. Refer to the documentation of your distribution to find 91132db935SJakub Kicinskiout how to update initramfs. Good indication of stale initramfs 92132db935SJakub Kicinskiis system loading wrong driver or firmware on boot, but when driver is 93132db935SJakub Kicinskilater reloaded manually everything works correctly. 94132db935SJakub Kicinski 95132db935SJakub KicinskiSelecting firmware per device 96132db935SJakub Kicinski----------------------------- 97132db935SJakub Kicinski 98132db935SJakub KicinskiMost commonly all cards on the system use the same type of firmware. 99*1ec6360dSWalter HeymansIf you want to load a specific firmware image for a specific card, you 100*1ec6360dSWalter Heymanscan use either the PCI bus address or serial number. The driver will 101*1ec6360dSWalter Heymansprint which files it's looking for when it recognizes a NFP device:: 102132db935SJakub Kicinski 103132db935SJakub Kicinski nfp: Looking for firmware file in order of priority: 104132db935SJakub Kicinski nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found 105132db935SJakub Kicinski nfp: netronome/pci-0000:02:00.0.nffw: not found 106132db935SJakub Kicinski nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading... 107132db935SJakub Kicinski 108132db935SJakub KicinskiIn this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw* 109132db935SJakub Kicinskior *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this 110132db935SJakub Kicinskifirmware file will take precedence over `nic_AMDA*` files. 111132db935SJakub Kicinski 112132db935SJakub KicinskiNote that `serial-*` and `pci-*` files are **not** automatically included 113132db935SJakub Kicinskiin initramfs, you will have to refer to documentation of appropriate tools 114132db935SJakub Kicinskito find out how to include them. 115132db935SJakub Kicinski 116*1ec6360dSWalter HeymansRunning firmware version 117*1ec6360dSWalter Heymans------------------------ 118*1ec6360dSWalter Heymans 119*1ec6360dSWalter HeymansThe version of the loaded firmware for a particular <netdev> interface, 120*1ec6360dSWalter Heymans(e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can 121*1ec6360dSWalter Heymansbe displayed with the ethtool command:: 122*1ec6360dSWalter Heymans 123*1ec6360dSWalter Heymans $ ethtool -i <netdev> 124*1ec6360dSWalter Heymans 125132db935SJakub KicinskiFirmware loading policy 126132db935SJakub Kicinski----------------------- 127132db935SJakub Kicinski 128132db935SJakub KicinskiFirmware loading policy is controlled via three HWinfo parameters 129132db935SJakub Kicinskistored as key value pairs in the device flash: 130132db935SJakub Kicinski 131132db935SJakub Kicinskiapp_fw_from_flash 132132db935SJakub Kicinski Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or 133132db935SJakub Kicinski the 'Preferred' (2) firmware. When 'Preferred' is selected, the management 134132db935SJakub Kicinski firmware makes the decision over which firmware will be loaded by comparing 135132db935SJakub Kicinski versions of the flash firmware and the host supplied firmware. 136132db935SJakub Kicinski This variable is configurable using the 'fw_load_policy' 137132db935SJakub Kicinski devlink parameter. 138132db935SJakub Kicinski 139132db935SJakub Kicinskiabi_drv_reset 140132db935SJakub Kicinski Defines if the driver should reset the firmware when 141132db935SJakub Kicinski the driver is probed, either 'Disk' (0) if firmware was found on disk, 142132db935SJakub Kicinski 'Always' (1) reset or 'Never' (2) reset. Note that the device is always 143132db935SJakub Kicinski reset on driver unload if firmware was loaded when the driver was probed. 144132db935SJakub Kicinski This variable is configurable using the 'reset_dev_on_drv_probe' 145132db935SJakub Kicinski devlink parameter. 146132db935SJakub Kicinski 147132db935SJakub Kicinskiabi_drv_load_ifc 148132db935SJakub Kicinski Defines a list of PF devices allowed to load FW on the device. 149132db935SJakub Kicinski This variable is not currently user configurable. 150132db935SJakub Kicinski 151*1ec6360dSWalter HeymansDevlink Info 152*1ec6360dSWalter Heymans============ 153*1ec6360dSWalter Heymans 154*1ec6360dSWalter HeymansThe devlink info command displays the running and stored firmware versions 155*1ec6360dSWalter Heymanson the device, serial number and board information. 156*1ec6360dSWalter Heymans 157*1ec6360dSWalter HeymansDevlink info command example (replace PCI address):: 158*1ec6360dSWalter Heymans 159*1ec6360dSWalter Heymans $ devlink dev info pci/0000:03:00.0 160*1ec6360dSWalter Heymans pci/0000:03:00.0: 161*1ec6360dSWalter Heymans driver nfp 162*1ec6360dSWalter Heymans serial_number CSAAMDA2001-1003000111 163*1ec6360dSWalter Heymans versions: 164*1ec6360dSWalter Heymans fixed: 165*1ec6360dSWalter Heymans board.id AMDA2001-1003 166*1ec6360dSWalter Heymans board.rev 01 167*1ec6360dSWalter Heymans board.manufacture CSA 168*1ec6360dSWalter Heymans board.model mozart 169*1ec6360dSWalter Heymans running: 170*1ec6360dSWalter Heymans fw.mgmt 22.10.0-rc3 171*1ec6360dSWalter Heymans fw.cpld 0x1000003 172*1ec6360dSWalter Heymans fw.app nic-22.09.0 173*1ec6360dSWalter Heymans chip.init AMDA-2001-1003 1003000111 174*1ec6360dSWalter Heymans stored: 175*1ec6360dSWalter Heymans fw.bundle_id bspbundle_1003000111 176*1ec6360dSWalter Heymans fw.mgmt 22.10.0-rc3 177*1ec6360dSWalter Heymans fw.cpld 0x0 178*1ec6360dSWalter Heymans chip.init AMDA-2001-1003 1003000111 179*1ec6360dSWalter Heymans 180*1ec6360dSWalter HeymansConfigure Device 181*1ec6360dSWalter Heymans================ 182*1ec6360dSWalter Heymans 183*1ec6360dSWalter HeymansThis section explains how to use Agilio SmartNICs running basic NIC firmware. 184*1ec6360dSWalter Heymans 185*1ec6360dSWalter HeymansConfigure interface link-speed 186*1ec6360dSWalter Heymans------------------------------ 187*1ec6360dSWalter HeymansThe following steps explains how to change between 10G mode and 25G mode on 188*1ec6360dSWalter HeymansAgilio CX 2x25GbE cards. The changing of port speed must be done in order, 189*1ec6360dSWalter Heymansport 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G. 190*1ec6360dSWalter Heymans 191*1ec6360dSWalter HeymansDown the respective interface(s):: 192*1ec6360dSWalter Heymans 193*1ec6360dSWalter Heymans $ ip link set dev <netdev port 0> down 194*1ec6360dSWalter Heymans $ ip link set dev <netdev port 1> down 195*1ec6360dSWalter Heymans 196*1ec6360dSWalter HeymansSet interface link-speed to 10G:: 197*1ec6360dSWalter Heymans 198*1ec6360dSWalter Heymans $ ethtool -s <netdev port 0> speed 10000 199*1ec6360dSWalter Heymans $ ethtool -s <netdev port 1> speed 10000 200*1ec6360dSWalter Heymans 201*1ec6360dSWalter HeymansSet interface link-speed to 25G:: 202*1ec6360dSWalter Heymans 203*1ec6360dSWalter Heymans $ ethtool -s <netdev port 0> speed 25000 204*1ec6360dSWalter Heymans $ ethtool -s <netdev port 1> speed 25000 205*1ec6360dSWalter Heymans 206*1ec6360dSWalter HeymansReload driver for changes to take effect:: 207*1ec6360dSWalter Heymans 208*1ec6360dSWalter Heymans $ rmmod nfp; modprobe nfp 209*1ec6360dSWalter Heymans 210*1ec6360dSWalter HeymansConfigure interface Maximum Transmission Unit (MTU) 211*1ec6360dSWalter Heymans--------------------------------------------------- 212*1ec6360dSWalter Heymans 213*1ec6360dSWalter HeymansThe MTU of interfaces can temporarily be set using the iproute2, ip link or 214*1ec6360dSWalter Heymansifconfig tools. Note that this change will not persist. Setting this via 215*1ec6360dSWalter HeymansNetwork Manager, or another appropriate OS configuration tool, is 216*1ec6360dSWalter Heymansrecommended as changes to the MTU using Network Manager can be made to 217*1ec6360dSWalter Heymanspersist. 218*1ec6360dSWalter Heymans 219*1ec6360dSWalter HeymansSet interface MTU to 9000 bytes:: 220*1ec6360dSWalter Heymans 221*1ec6360dSWalter Heymans $ ip link set dev <netdev port> mtu 9000 222*1ec6360dSWalter Heymans 223*1ec6360dSWalter HeymansIt is the responsibility of the user or the orchestration layer to set 224*1ec6360dSWalter Heymansappropriate MTU values when handling jumbo frames or utilizing tunnels. For 225*1ec6360dSWalter Heymansexample, if packets sent from a VM are to be encapsulated on the card and 226*1ec6360dSWalter Heymansegress a physical port, then the MTU of the VF should be set to lower than 227*1ec6360dSWalter Heymansthat of the physical port to account for the extra bytes added by the 228*1ec6360dSWalter Heymansadditional header. If a setup is expected to see fallback traffic between 229*1ec6360dSWalter Heymansthe SmartNIC and the kernel then the user should also ensure that the PF MTU 230*1ec6360dSWalter Heymansis appropriately set to avoid unexpected drops on this path. 231*1ec6360dSWalter Heymans 232*1ec6360dSWalter HeymansConfigure Forward Error Correction (FEC) modes 233*1ec6360dSWalter Heymans---------------------------------------------- 234*1ec6360dSWalter Heymans 235*1ec6360dSWalter HeymansAgilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R, 236*1ec6360dSWalter HeymansReedSolomon and Off modes. Each physical port's FEC mode can be set 237*1ec6360dSWalter Heymansindependently using ethtool. The supported FEC modes for an interface can 238*1ec6360dSWalter Heymansbe viewed using:: 239*1ec6360dSWalter Heymans 240*1ec6360dSWalter Heymans $ ethtool <netdev> 241*1ec6360dSWalter Heymans 242*1ec6360dSWalter HeymansThe currently configured FEC mode can be viewed using:: 243*1ec6360dSWalter Heymans 244*1ec6360dSWalter Heymans $ ethtool --show-fec <netdev> 245*1ec6360dSWalter Heymans 246*1ec6360dSWalter HeymansTo force the FEC mode for a particular port, auto-negotiation must be disabled 247*1ec6360dSWalter Heymans(see the `Auto-negotiation`_ section). An example of how to set the FEC mode 248*1ec6360dSWalter Heymansto Reed-Solomon is:: 249*1ec6360dSWalter Heymans 250*1ec6360dSWalter Heymans $ ethtool --set-fec <netdev> encoding rs 251*1ec6360dSWalter Heymans 252*1ec6360dSWalter HeymansAuto-negotiation 253*1ec6360dSWalter Heymans---------------- 254*1ec6360dSWalter Heymans 255*1ec6360dSWalter HeymansTo change auto-negotiation settings, the link must first be put down. After the 256*1ec6360dSWalter Heymanslink is down, auto-negotiation can be enabled or disabled using:: 257*1ec6360dSWalter Heymans 258*1ec6360dSWalter Heymans ethtool -s <netdev> autoneg <on|off> 259*1ec6360dSWalter Heymans 260132db935SJakub KicinskiStatistics 261132db935SJakub Kicinski========== 262132db935SJakub Kicinski 263132db935SJakub KicinskiFollowing device statistics are available through the ``ethtool -S`` interface: 264132db935SJakub Kicinski 265132db935SJakub Kicinski.. flat-table:: NFP device statistics 266132db935SJakub Kicinski :header-rows: 1 267132db935SJakub Kicinski :widths: 3 1 11 268132db935SJakub Kicinski 269132db935SJakub Kicinski * - Name 270132db935SJakub Kicinski - ID 271132db935SJakub Kicinski - Meaning 272132db935SJakub Kicinski 273132db935SJakub Kicinski * - dev_rx_discards 274132db935SJakub Kicinski - 1 275132db935SJakub Kicinski - Packet can be discarded on the RX path for one of the following reasons: 276132db935SJakub Kicinski 277132db935SJakub Kicinski * The NIC is not in promisc mode, and the destination MAC address 278132db935SJakub Kicinski doesn't match the interfaces' MAC address. 279132db935SJakub Kicinski * The received packet is larger than the max buffer size on the host. 280132db935SJakub Kicinski I.e. it exceeds the Layer 3 MRU. 281132db935SJakub Kicinski * There is no freelist descriptor available on the host for the packet. 282132db935SJakub Kicinski It is likely that the NIC couldn't cache one in time. 283132db935SJakub Kicinski * A BPF program discarded the packet. 284132db935SJakub Kicinski * The datapath drop action was executed. 285132db935SJakub Kicinski * The MAC discarded the packet due to lack of ingress buffer space 286132db935SJakub Kicinski on the NIC. 287132db935SJakub Kicinski 288132db935SJakub Kicinski * - dev_rx_errors 289132db935SJakub Kicinski - 2 290132db935SJakub Kicinski - A packet can be counted (and dropped) as RX error for the following 291132db935SJakub Kicinski reasons: 292132db935SJakub Kicinski 293132db935SJakub Kicinski * A problem with the VEB lookup (only when SR-IOV is used). 294132db935SJakub Kicinski * A physical layer problem that causes Ethernet errors, like FCS or 295132db935SJakub Kicinski alignment errors. The cause is usually faulty cables or SFPs. 296132db935SJakub Kicinski 297132db935SJakub Kicinski * - dev_rx_bytes 298132db935SJakub Kicinski - 3 299132db935SJakub Kicinski - Total number of bytes received. 300132db935SJakub Kicinski 301132db935SJakub Kicinski * - dev_rx_uc_bytes 302132db935SJakub Kicinski - 4 303132db935SJakub Kicinski - Unicast bytes received. 304132db935SJakub Kicinski 305132db935SJakub Kicinski * - dev_rx_mc_bytes 306132db935SJakub Kicinski - 5 307132db935SJakub Kicinski - Multicast bytes received. 308132db935SJakub Kicinski 309132db935SJakub Kicinski * - dev_rx_bc_bytes 310132db935SJakub Kicinski - 6 311132db935SJakub Kicinski - Broadcast bytes received. 312132db935SJakub Kicinski 313132db935SJakub Kicinski * - dev_rx_pkts 314132db935SJakub Kicinski - 7 315132db935SJakub Kicinski - Total number of packets received. 316132db935SJakub Kicinski 317132db935SJakub Kicinski * - dev_rx_mc_pkts 318132db935SJakub Kicinski - 8 319132db935SJakub Kicinski - Multicast packets received. 320132db935SJakub Kicinski 321132db935SJakub Kicinski * - dev_rx_bc_pkts 322132db935SJakub Kicinski - 9 323132db935SJakub Kicinski - Broadcast packets received. 324132db935SJakub Kicinski 325132db935SJakub Kicinski * - dev_tx_discards 326132db935SJakub Kicinski - 10 327132db935SJakub Kicinski - A packet can be discarded in the TX direction if the MAC is 328132db935SJakub Kicinski being flow controlled and the NIC runs out of TX queue space. 329132db935SJakub Kicinski 330132db935SJakub Kicinski * - dev_tx_errors 331132db935SJakub Kicinski - 11 332132db935SJakub Kicinski - A packet can be counted as TX error (and dropped) for one for the 333132db935SJakub Kicinski following reasons: 334132db935SJakub Kicinski 335132db935SJakub Kicinski * The packet is an LSO segment, but the Layer 3 or Layer 4 offset 336132db935SJakub Kicinski could not be determined. Therefore LSO could not continue. 337132db935SJakub Kicinski * An invalid packet descriptor was received over PCIe. 338132db935SJakub Kicinski * The packet Layer 3 length exceeds the device MTU. 339132db935SJakub Kicinski * An error on the MAC/physical layer. Usually due to faulty cables or 340132db935SJakub Kicinski SFPs. 341132db935SJakub Kicinski * A CTM buffer could not be allocated. 342132db935SJakub Kicinski * The packet offset was incorrect and could not be fixed by the NIC. 343132db935SJakub Kicinski 344132db935SJakub Kicinski * - dev_tx_bytes 345132db935SJakub Kicinski - 12 346132db935SJakub Kicinski - Total number of bytes transmitted. 347132db935SJakub Kicinski 348132db935SJakub Kicinski * - dev_tx_uc_bytes 349132db935SJakub Kicinski - 13 350132db935SJakub Kicinski - Unicast bytes transmitted. 351132db935SJakub Kicinski 352132db935SJakub Kicinski * - dev_tx_mc_bytes 353132db935SJakub Kicinski - 14 354132db935SJakub Kicinski - Multicast bytes transmitted. 355132db935SJakub Kicinski 356132db935SJakub Kicinski * - dev_tx_bc_bytes 357132db935SJakub Kicinski - 15 358132db935SJakub Kicinski - Broadcast bytes transmitted. 359132db935SJakub Kicinski 360132db935SJakub Kicinski * - dev_tx_pkts 361132db935SJakub Kicinski - 16 362132db935SJakub Kicinski - Total number of packets transmitted. 363132db935SJakub Kicinski 364132db935SJakub Kicinski * - dev_tx_mc_pkts 365132db935SJakub Kicinski - 17 366132db935SJakub Kicinski - Multicast packets transmitted. 367132db935SJakub Kicinski 368132db935SJakub Kicinski * - dev_tx_bc_pkts 369132db935SJakub Kicinski - 18 370132db935SJakub Kicinski - Broadcast packets transmitted. 371132db935SJakub Kicinski 372132db935SJakub KicinskiNote that statistics unknown to the driver will be displayed as 373132db935SJakub Kicinski``dev_unknown_stat$ID``, where ``$ID`` refers to the second column 374132db935SJakub Kicinskiabove. 375