1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2.. include:: <isonum.txt>
3
4===========================================
5Network Flow Processor (NFP) Kernel Drivers
6===========================================
7
8:Copyright: |copy| 2019, Netronome Systems, Inc.
9:Copyright: |copy| 2022, Corigine, Inc.
10
11Contents
12========
13
14- `Overview`_
15- `Acquiring Firmware`_
16- `Devlink Info`_
17- `Configure Device`_
18- `Statistics`_
19
20Overview
21========
22
23This driver supports Netronome and Corigine's line of Network Flow Processor
24devices, including the NFP3800, NFP4000, NFP5000, and NFP6000 models, which
25are also incorporated in the companies' family of Agilio SmartNICs. The SR-IOV
26physical and virtual functions for these devices are supported by the driver.
27
28Acquiring Firmware
29==================
30
31The NFP3800, NFP4000 and NFP6000 devices require application specific firmware
32to function. Application firmware can be located either on the host file system
33or in the device flash (if supported by management firmware).
34
35Firmware files on the host filesystem contain card type (`AMDA-*` string), media
36config etc. They should be placed in `/lib/firmware/netronome` directory to
37load firmware from the host file system.
38
39Firmware for basic NIC operation is available in the upstream
40`linux-firmware.git` repository.
41
42A more comprehensive list of firmware can be downloaded from the
43`Corigine Support site <https://www.corigine.com/DPUDownload.html>`_.
44
45Firmware in NVRAM
46-----------------
47
48Recent versions of management firmware supports loading application
49firmware from flash when the host driver gets probed. The firmware loading
50policy configuration may be used to configure this feature appropriately.
51
52Devlink or ethtool can be used to update the application firmware on the device
53flash by providing the appropriate `nic_AMDA*.nffw` file to the respective
54command. Users need to take care to write the correct firmware image for the
55card and media configuration to flash.
56
57Available storage space in flash depends on the card being used.
58
59Dealing with multiple projects
60------------------------------
61
62NFP hardware is fully programmable therefore there can be different
63firmware images targeting different applications.
64
65When using application firmware from host, we recommend placing
66actual firmware files in application-named subdirectories in
67`/lib/firmware/netronome` and linking the desired files, e.g.::
68
69    $ tree /lib/firmware/netronome/
70    /lib/firmware/netronome/
71    ├── bpf
72    │   ├── nic_AMDA0081-0001_1x40.nffw
73    │   └── nic_AMDA0081-0001_4x10.nffw
74    ├── flower
75    │   ├── nic_AMDA0081-0001_1x40.nffw
76    │   └── nic_AMDA0081-0001_4x10.nffw
77    ├── nic
78    │   ├── nic_AMDA0081-0001_1x40.nffw
79    │   └── nic_AMDA0081-0001_4x10.nffw
80    ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw
81    └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw
82
83    3 directories, 8 files
84
85You may need to use hard instead of symbolic links on distributions
86which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu).
87
88After changing firmware files you may need to regenerate the initramfs
89image. Initramfs contains drivers and firmware files your system may
90need to boot. Refer to the documentation of your distribution to find
91out how to update initramfs. Good indication of stale initramfs
92is system loading wrong driver or firmware on boot, but when driver is
93later reloaded manually everything works correctly.
94
95Selecting firmware per device
96-----------------------------
97
98Most commonly all cards on the system use the same type of firmware.
99If you want to load a specific firmware image for a specific card, you
100can use either the PCI bus address or serial number. The driver will
101print which files it's looking for when it recognizes a NFP device::
102
103    nfp: Looking for firmware file in order of priority:
104    nfp:  netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found
105    nfp:  netronome/pci-0000:02:00.0.nffw: not found
106    nfp:  netronome/nic_AMDA0081-0001_1x40.nffw: found, loading...
107
108In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw*
109or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this
110firmware file will take precedence over `nic_AMDA*` files.
111
112Note that `serial-*` and `pci-*` files are **not** automatically included
113in initramfs, you will have to refer to documentation of appropriate tools
114to find out how to include them.
115
116Running firmware version
117------------------------
118
119The version of the loaded firmware for a particular <netdev> interface,
120(e.g. enp4s0), or an interface's port <netdev port> (e.g. enp4s0np0) can
121be displayed with the ethtool command::
122
123  $ ethtool -i <netdev>
124
125Firmware loading policy
126-----------------------
127
128Firmware loading policy is controlled via three HWinfo parameters
129stored as key value pairs in the device flash:
130
131app_fw_from_flash
132    Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or
133    the 'Preferred' (2) firmware. When 'Preferred' is selected, the management
134    firmware makes the decision over which firmware will be loaded by comparing
135    versions of the flash firmware and the host supplied firmware.
136    This variable is configurable using the 'fw_load_policy'
137    devlink parameter.
138
139abi_drv_reset
140    Defines if the driver should reset the firmware when
141    the driver is probed, either 'Disk' (0) if firmware was found on disk,
142    'Always' (1) reset or 'Never' (2) reset. Note that the device is always
143    reset on driver unload if firmware was loaded when the driver was probed.
144    This variable is configurable using the 'reset_dev_on_drv_probe'
145    devlink parameter.
146
147abi_drv_load_ifc
148    Defines a list of PF devices allowed to load FW on the device.
149    This variable is not currently user configurable.
150
151Devlink Info
152============
153
154The devlink info command displays the running and stored firmware versions
155on the device, serial number and board information.
156
157Devlink info command example (replace PCI address)::
158
159  $ devlink dev info pci/0000:03:00.0
160    pci/0000:03:00.0:
161      driver nfp
162      serial_number CSAAMDA2001-1003000111
163      versions:
164          fixed:
165            board.id AMDA2001-1003
166            board.rev 01
167            board.manufacture CSA
168            board.model mozart
169          running:
170            fw.mgmt 22.10.0-rc3
171            fw.cpld 0x1000003
172            fw.app nic-22.09.0
173            chip.init AMDA-2001-1003  1003000111
174          stored:
175            fw.bundle_id bspbundle_1003000111
176            fw.mgmt 22.10.0-rc3
177            fw.cpld 0x0
178            chip.init AMDA-2001-1003  1003000111
179
180Configure Device
181================
182
183This section explains how to use Agilio SmartNICs running basic NIC firmware.
184
185Configure interface link-speed
186------------------------------
187The following steps explains how to change between 10G mode and 25G mode on
188Agilio CX 2x25GbE cards. The changing of port speed must be done in order,
189port 0 (p0) must be set to 10G before port 1 (p1) may be set to 10G.
190
191Down the respective interface(s)::
192
193  $ ip link set dev <netdev port 0> down
194  $ ip link set dev <netdev port 1> down
195
196Set interface link-speed to 10G::
197
198  $ ethtool -s <netdev port 0> speed 10000
199  $ ethtool -s <netdev port 1> speed 10000
200
201Set interface link-speed to 25G::
202
203  $ ethtool -s <netdev port 0> speed 25000
204  $ ethtool -s <netdev port 1> speed 25000
205
206Reload driver for changes to take effect::
207
208  $ rmmod nfp; modprobe nfp
209
210Configure interface Maximum Transmission Unit (MTU)
211---------------------------------------------------
212
213The MTU of interfaces can temporarily be set using the iproute2, ip link or
214ifconfig tools. Note that this change will not persist. Setting this via
215Network Manager, or another appropriate OS configuration tool, is
216recommended as changes to the MTU using Network Manager can be made to
217persist.
218
219Set interface MTU to 9000 bytes::
220
221  $ ip link set dev <netdev port> mtu 9000
222
223It is the responsibility of the user or the orchestration layer to set
224appropriate MTU values when handling jumbo frames or utilizing tunnels. For
225example, if packets sent from a VM are to be encapsulated on the card and
226egress a physical port, then the MTU of the VF should be set to lower than
227that of the physical port to account for the extra bytes added by the
228additional header. If a setup is expected to see fallback traffic between
229the SmartNIC and the kernel then the user should also ensure that the PF MTU
230is appropriately set to avoid unexpected drops on this path.
231
232Configure Forward Error Correction (FEC) modes
233----------------------------------------------
234
235Agilio SmartNICs support FEC mode configuration, e.g. Auto, Firecode Base-R,
236ReedSolomon and Off modes. Each physical port's FEC mode can be set
237independently using ethtool. The supported FEC modes for an interface can
238be viewed using::
239
240  $ ethtool <netdev>
241
242The currently configured FEC mode can be viewed using::
243
244  $ ethtool --show-fec <netdev>
245
246To force the FEC mode for a particular port, auto-negotiation must be disabled
247(see the `Auto-negotiation`_ section). An example of how to set the FEC mode
248to Reed-Solomon is::
249
250  $ ethtool --set-fec <netdev> encoding rs
251
252Auto-negotiation
253----------------
254
255To change auto-negotiation settings, the link must first be put down. After the
256link is down, auto-negotiation can be enabled or disabled using::
257
258  ethtool -s <netdev> autoneg <on|off>
259
260Statistics
261==========
262
263Following device statistics are available through the ``ethtool -S`` interface:
264
265.. flat-table:: NFP device statistics
266   :header-rows: 1
267   :widths: 3 1 11
268
269   * - Name
270     - ID
271     - Meaning
272
273   * - dev_rx_discards
274     - 1
275     - Packet can be discarded on the RX path for one of the following reasons:
276
277        * The NIC is not in promisc mode, and the destination MAC address
278          doesn't match the interfaces' MAC address.
279        * The received packet is larger than the max buffer size on the host.
280          I.e. it exceeds the Layer 3 MRU.
281        * There is no freelist descriptor available on the host for the packet.
282          It is likely that the NIC couldn't cache one in time.
283        * A BPF program discarded the packet.
284        * The datapath drop action was executed.
285        * The MAC discarded the packet due to lack of ingress buffer space
286          on the NIC.
287
288   * - dev_rx_errors
289     - 2
290     - A packet can be counted (and dropped) as RX error for the following
291       reasons:
292
293       * A problem with the VEB lookup (only when SR-IOV is used).
294       * A physical layer problem that causes Ethernet errors, like FCS or
295         alignment errors. The cause is usually faulty cables or SFPs.
296
297   * - dev_rx_bytes
298     - 3
299     - Total number of bytes received.
300
301   * - dev_rx_uc_bytes
302     - 4
303     - Unicast bytes received.
304
305   * - dev_rx_mc_bytes
306     - 5
307     - Multicast bytes received.
308
309   * - dev_rx_bc_bytes
310     - 6
311     - Broadcast bytes received.
312
313   * - dev_rx_pkts
314     - 7
315     - Total number of packets received.
316
317   * - dev_rx_mc_pkts
318     - 8
319     - Multicast packets received.
320
321   * - dev_rx_bc_pkts
322     - 9
323     - Broadcast packets received.
324
325   * - dev_tx_discards
326     - 10
327     - A packet can be discarded in the TX direction if the MAC is
328       being flow controlled and the NIC runs out of TX queue space.
329
330   * - dev_tx_errors
331     - 11
332     - A packet can be counted as TX error (and dropped) for one for the
333       following reasons:
334
335       * The packet is an LSO segment, but the Layer 3 or Layer 4 offset
336         could not be determined. Therefore LSO could not continue.
337       * An invalid packet descriptor was received over PCIe.
338       * The packet Layer 3 length exceeds the device MTU.
339       * An error on the MAC/physical layer. Usually due to faulty cables or
340         SFPs.
341       * A CTM buffer could not be allocated.
342       * The packet offset was incorrect and could not be fixed by the NIC.
343
344   * - dev_tx_bytes
345     - 12
346     - Total number of bytes transmitted.
347
348   * - dev_tx_uc_bytes
349     - 13
350     - Unicast bytes transmitted.
351
352   * - dev_tx_mc_bytes
353     - 14
354     - Multicast bytes transmitted.
355
356   * - dev_tx_bc_bytes
357     - 15
358     - Broadcast bytes transmitted.
359
360   * - dev_tx_pkts
361     - 16
362     - Total number of packets transmitted.
363
364   * - dev_tx_mc_pkts
365     - 17
366     - Multicast packets transmitted.
367
368   * - dev_tx_bc_pkts
369     - 18
370     - Broadcast packets transmitted.
371
372Note that statistics unknown to the driver will be displayed as
373``dev_unknown_stat$ID``, where ``$ID`` refers to the second column
374above.
375