1132db935SJakub Kicinski.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) 2132db935SJakub Kicinski 3132db935SJakub Kicinski============================================= 4132db935SJakub KicinskiNetronome Flow Processor (NFP) Kernel Drivers 5132db935SJakub Kicinski============================================= 6132db935SJakub Kicinski 7132db935SJakub KicinskiCopyright (c) 2019, Netronome Systems, Inc. 8132db935SJakub Kicinski 9132db935SJakub KicinskiContents 10132db935SJakub Kicinski======== 11132db935SJakub Kicinski 12132db935SJakub Kicinski- `Overview`_ 13132db935SJakub Kicinski- `Acquiring Firmware`_ 14132db935SJakub Kicinski 15132db935SJakub KicinskiOverview 16132db935SJakub Kicinski======== 17132db935SJakub Kicinski 18132db935SJakub KicinskiThis driver supports Netronome's line of Flow Processor devices, 19132db935SJakub Kicinskiincluding the NFP4000, NFP5000, and NFP6000 models, which are also 20132db935SJakub Kicinskiincorporated in the company's family of Agilio SmartNICs. The SR-IOV 21132db935SJakub Kicinskiphysical and virtual functions for these devices are supported by 22132db935SJakub Kicinskithe driver. 23132db935SJakub Kicinski 24132db935SJakub KicinskiAcquiring Firmware 25132db935SJakub Kicinski================== 26132db935SJakub Kicinski 27132db935SJakub KicinskiThe NFP4000 and NFP6000 devices require application specific firmware 28132db935SJakub Kicinskito function. Application firmware can be located either on the host file system 29132db935SJakub Kicinskior in the device flash (if supported by management firmware). 30132db935SJakub Kicinski 31132db935SJakub KicinskiFirmware files on the host filesystem contain card type (`AMDA-*` string), media 32132db935SJakub Kicinskiconfig etc. They should be placed in `/lib/firmware/netronome` directory to 33132db935SJakub Kicinskiload firmware from the host file system. 34132db935SJakub Kicinski 35132db935SJakub KicinskiFirmware for basic NIC operation is available in the upstream 36132db935SJakub Kicinski`linux-firmware.git` repository. 37132db935SJakub Kicinski 38132db935SJakub KicinskiFirmware in NVRAM 39132db935SJakub Kicinski----------------- 40132db935SJakub Kicinski 41132db935SJakub KicinskiRecent versions of management firmware supports loading application 42132db935SJakub Kicinskifirmware from flash when the host driver gets probed. The firmware loading 43132db935SJakub Kicinskipolicy configuration may be used to configure this feature appropriately. 44132db935SJakub Kicinski 45132db935SJakub KicinskiDevlink or ethtool can be used to update the application firmware on the device 46132db935SJakub Kicinskiflash by providing the appropriate `nic_AMDA*.nffw` file to the respective 47132db935SJakub Kicinskicommand. Users need to take care to write the correct firmware image for the 48132db935SJakub Kicinskicard and media configuration to flash. 49132db935SJakub Kicinski 50132db935SJakub KicinskiAvailable storage space in flash depends on the card being used. 51132db935SJakub Kicinski 52132db935SJakub KicinskiDealing with multiple projects 53132db935SJakub Kicinski------------------------------ 54132db935SJakub Kicinski 55132db935SJakub KicinskiNFP hardware is fully programmable therefore there can be different 56132db935SJakub Kicinskifirmware images targeting different applications. 57132db935SJakub Kicinski 58132db935SJakub KicinskiWhen using application firmware from host, we recommend placing 59132db935SJakub Kicinskiactual firmware files in application-named subdirectories in 60132db935SJakub Kicinski`/lib/firmware/netronome` and linking the desired files, e.g.:: 61132db935SJakub Kicinski 62132db935SJakub Kicinski $ tree /lib/firmware/netronome/ 63132db935SJakub Kicinski /lib/firmware/netronome/ 64132db935SJakub Kicinski ├── bpf 65132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 66132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 67132db935SJakub Kicinski ├── flower 68132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 69132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 70132db935SJakub Kicinski ├── nic 71132db935SJakub Kicinski │ ├── nic_AMDA0081-0001_1x40.nffw 72132db935SJakub Kicinski │ └── nic_AMDA0081-0001_4x10.nffw 73132db935SJakub Kicinski ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw 74132db935SJakub Kicinski └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw 75132db935SJakub Kicinski 76132db935SJakub Kicinski 3 directories, 8 files 77132db935SJakub Kicinski 78132db935SJakub KicinskiYou may need to use hard instead of symbolic links on distributions 79132db935SJakub Kicinskiwhich use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu). 80132db935SJakub Kicinski 81132db935SJakub KicinskiAfter changing firmware files you may need to regenerate the initramfs 82132db935SJakub Kicinskiimage. Initramfs contains drivers and firmware files your system may 83132db935SJakub Kicinskineed to boot. Refer to the documentation of your distribution to find 84132db935SJakub Kicinskiout how to update initramfs. Good indication of stale initramfs 85132db935SJakub Kicinskiis system loading wrong driver or firmware on boot, but when driver is 86132db935SJakub Kicinskilater reloaded manually everything works correctly. 87132db935SJakub Kicinski 88132db935SJakub KicinskiSelecting firmware per device 89132db935SJakub Kicinski----------------------------- 90132db935SJakub Kicinski 91132db935SJakub KicinskiMost commonly all cards on the system use the same type of firmware. 92132db935SJakub KicinskiIf you want to load specific firmware image for a specific card, you 93132db935SJakub Kicinskican use either the PCI bus address or serial number. Driver will print 94132db935SJakub Kicinskiwhich files it's looking for when it recognizes a NFP device:: 95132db935SJakub Kicinski 96132db935SJakub Kicinski nfp: Looking for firmware file in order of priority: 97132db935SJakub Kicinski nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found 98132db935SJakub Kicinski nfp: netronome/pci-0000:02:00.0.nffw: not found 99132db935SJakub Kicinski nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading... 100132db935SJakub Kicinski 101132db935SJakub KicinskiIn this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw* 102132db935SJakub Kicinskior *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this 103132db935SJakub Kicinskifirmware file will take precedence over `nic_AMDA*` files. 104132db935SJakub Kicinski 105132db935SJakub KicinskiNote that `serial-*` and `pci-*` files are **not** automatically included 106132db935SJakub Kicinskiin initramfs, you will have to refer to documentation of appropriate tools 107132db935SJakub Kicinskito find out how to include them. 108132db935SJakub Kicinski 109132db935SJakub KicinskiFirmware loading policy 110132db935SJakub Kicinski----------------------- 111132db935SJakub Kicinski 112132db935SJakub KicinskiFirmware loading policy is controlled via three HWinfo parameters 113132db935SJakub Kicinskistored as key value pairs in the device flash: 114132db935SJakub Kicinski 115132db935SJakub Kicinskiapp_fw_from_flash 116132db935SJakub Kicinski Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or 117132db935SJakub Kicinski the 'Preferred' (2) firmware. When 'Preferred' is selected, the management 118132db935SJakub Kicinski firmware makes the decision over which firmware will be loaded by comparing 119132db935SJakub Kicinski versions of the flash firmware and the host supplied firmware. 120132db935SJakub Kicinski This variable is configurable using the 'fw_load_policy' 121132db935SJakub Kicinski devlink parameter. 122132db935SJakub Kicinski 123132db935SJakub Kicinskiabi_drv_reset 124132db935SJakub Kicinski Defines if the driver should reset the firmware when 125132db935SJakub Kicinski the driver is probed, either 'Disk' (0) if firmware was found on disk, 126132db935SJakub Kicinski 'Always' (1) reset or 'Never' (2) reset. Note that the device is always 127132db935SJakub Kicinski reset on driver unload if firmware was loaded when the driver was probed. 128132db935SJakub Kicinski This variable is configurable using the 'reset_dev_on_drv_probe' 129132db935SJakub Kicinski devlink parameter. 130132db935SJakub Kicinski 131132db935SJakub Kicinskiabi_drv_load_ifc 132132db935SJakub Kicinski Defines a list of PF devices allowed to load FW on the device. 133132db935SJakub Kicinski This variable is not currently user configurable. 134132db935SJakub Kicinski 135132db935SJakub KicinskiStatistics 136132db935SJakub Kicinski========== 137132db935SJakub Kicinski 138132db935SJakub KicinskiFollowing device statistics are available through the ``ethtool -S`` interface: 139132db935SJakub Kicinski 140132db935SJakub Kicinski.. flat-table:: NFP device statistics 141132db935SJakub Kicinski :header-rows: 1 142132db935SJakub Kicinski :widths: 3 1 11 143132db935SJakub Kicinski 144132db935SJakub Kicinski * - Name 145132db935SJakub Kicinski - ID 146132db935SJakub Kicinski - Meaning 147132db935SJakub Kicinski 148132db935SJakub Kicinski * - dev_rx_discards 149132db935SJakub Kicinski - 1 150132db935SJakub Kicinski - Packet can be discarded on the RX path for one of the following reasons: 151132db935SJakub Kicinski 152132db935SJakub Kicinski * The NIC is not in promisc mode, and the destination MAC address 153132db935SJakub Kicinski doesn't match the interfaces' MAC address. 154132db935SJakub Kicinski * The received packet is larger than the max buffer size on the host. 155132db935SJakub Kicinski I.e. it exceeds the Layer 3 MRU. 156132db935SJakub Kicinski * There is no freelist descriptor available on the host for the packet. 157132db935SJakub Kicinski It is likely that the NIC couldn't cache one in time. 158132db935SJakub Kicinski * A BPF program discarded the packet. 159132db935SJakub Kicinski * The datapath drop action was executed. 160132db935SJakub Kicinski * The MAC discarded the packet due to lack of ingress buffer space 161132db935SJakub Kicinski on the NIC. 162132db935SJakub Kicinski 163132db935SJakub Kicinski * - dev_rx_errors 164132db935SJakub Kicinski - 2 165132db935SJakub Kicinski - A packet can be counted (and dropped) as RX error for the following 166132db935SJakub Kicinski reasons: 167132db935SJakub Kicinski 168132db935SJakub Kicinski * A problem with the VEB lookup (only when SR-IOV is used). 169132db935SJakub Kicinski * A physical layer problem that causes Ethernet errors, like FCS or 170132db935SJakub Kicinski alignment errors. The cause is usually faulty cables or SFPs. 171132db935SJakub Kicinski 172132db935SJakub Kicinski * - dev_rx_bytes 173132db935SJakub Kicinski - 3 174132db935SJakub Kicinski - Total number of bytes received. 175132db935SJakub Kicinski 176132db935SJakub Kicinski * - dev_rx_uc_bytes 177132db935SJakub Kicinski - 4 178132db935SJakub Kicinski - Unicast bytes received. 179132db935SJakub Kicinski 180132db935SJakub Kicinski * - dev_rx_mc_bytes 181132db935SJakub Kicinski - 5 182132db935SJakub Kicinski - Multicast bytes received. 183132db935SJakub Kicinski 184132db935SJakub Kicinski * - dev_rx_bc_bytes 185132db935SJakub Kicinski - 6 186132db935SJakub Kicinski - Broadcast bytes received. 187132db935SJakub Kicinski 188132db935SJakub Kicinski * - dev_rx_pkts 189132db935SJakub Kicinski - 7 190132db935SJakub Kicinski - Total number of packets received. 191132db935SJakub Kicinski 192132db935SJakub Kicinski * - dev_rx_mc_pkts 193132db935SJakub Kicinski - 8 194132db935SJakub Kicinski - Multicast packets received. 195132db935SJakub Kicinski 196132db935SJakub Kicinski * - dev_rx_bc_pkts 197132db935SJakub Kicinski - 9 198132db935SJakub Kicinski - Broadcast packets received. 199132db935SJakub Kicinski 200132db935SJakub Kicinski * - dev_tx_discards 201132db935SJakub Kicinski - 10 202132db935SJakub Kicinski - A packet can be discarded in the TX direction if the MAC is 203132db935SJakub Kicinski being flow controlled and the NIC runs out of TX queue space. 204132db935SJakub Kicinski 205132db935SJakub Kicinski * - dev_tx_errors 206132db935SJakub Kicinski - 11 207132db935SJakub Kicinski - A packet can be counted as TX error (and dropped) for one for the 208132db935SJakub Kicinski following reasons: 209132db935SJakub Kicinski 210132db935SJakub Kicinski * The packet is an LSO segment, but the Layer 3 or Layer 4 offset 211132db935SJakub Kicinski could not be determined. Therefore LSO could not continue. 212132db935SJakub Kicinski * An invalid packet descriptor was received over PCIe. 213132db935SJakub Kicinski * The packet Layer 3 length exceeds the device MTU. 214132db935SJakub Kicinski * An error on the MAC/physical layer. Usually due to faulty cables or 215132db935SJakub Kicinski SFPs. 216132db935SJakub Kicinski * A CTM buffer could not be allocated. 217132db935SJakub Kicinski * The packet offset was incorrect and could not be fixed by the NIC. 218132db935SJakub Kicinski 219132db935SJakub Kicinski * - dev_tx_bytes 220132db935SJakub Kicinski - 12 221132db935SJakub Kicinski - Total number of bytes transmitted. 222132db935SJakub Kicinski 223132db935SJakub Kicinski * - dev_tx_uc_bytes 224132db935SJakub Kicinski - 13 225132db935SJakub Kicinski - Unicast bytes transmitted. 226132db935SJakub Kicinski 227132db935SJakub Kicinski * - dev_tx_mc_bytes 228132db935SJakub Kicinski - 14 229132db935SJakub Kicinski - Multicast bytes transmitted. 230132db935SJakub Kicinski 231132db935SJakub Kicinski * - dev_tx_bc_bytes 232132db935SJakub Kicinski - 15 233132db935SJakub Kicinski - Broadcast bytes transmitted. 234132db935SJakub Kicinski 235132db935SJakub Kicinski * - dev_tx_pkts 236132db935SJakub Kicinski - 16 237132db935SJakub Kicinski - Total number of packets transmitted. 238132db935SJakub Kicinski 239132db935SJakub Kicinski * - dev_tx_mc_pkts 240132db935SJakub Kicinski - 17 241132db935SJakub Kicinski - Multicast packets transmitted. 242132db935SJakub Kicinski 243132db935SJakub Kicinski * - dev_tx_bc_pkts 244132db935SJakub Kicinski - 18 245132db935SJakub Kicinski - Broadcast packets transmitted. 246132db935SJakub Kicinski 247132db935SJakub KicinskiNote that statistics unknown to the driver will be displayed as 248132db935SJakub Kicinski``dev_unknown_stat$ID``, where ``$ID`` refers to the second column 249132db935SJakub Kicinskiabove. 250