1.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
2
3=============================================
4Netronome Flow Processor (NFP) Kernel Drivers
5=============================================
6
7Copyright (c) 2019, Netronome Systems, Inc.
8
9Contents
10========
11
12- `Overview`_
13- `Acquiring Firmware`_
14
15Overview
16========
17
18This driver supports Netronome's line of Flow Processor devices,
19including the NFP4000, NFP5000, and NFP6000 models, which are also
20incorporated in the company's family of Agilio SmartNICs. The SR-IOV
21physical and virtual functions for these devices are supported by
22the driver.
23
24Acquiring Firmware
25==================
26
27The NFP4000 and NFP6000 devices require application specific firmware
28to function.  Application firmware can be located either on the host file system
29or in the device flash (if supported by management firmware).
30
31Firmware files on the host filesystem contain card type (`AMDA-*` string), media
32config etc.  They should be placed in `/lib/firmware/netronome` directory to
33load firmware from the host file system.
34
35Firmware for basic NIC operation is available in the upstream
36`linux-firmware.git` repository.
37
38Firmware in NVRAM
39-----------------
40
41Recent versions of management firmware supports loading application
42firmware from flash when the host driver gets probed.  The firmware loading
43policy configuration may be used to configure this feature appropriately.
44
45Devlink or ethtool can be used to update the application firmware on the device
46flash by providing the appropriate `nic_AMDA*.nffw` file to the respective
47command.  Users need to take care to write the correct firmware image for the
48card and media configuration to flash.
49
50Available storage space in flash depends on the card being used.
51
52Dealing with multiple projects
53------------------------------
54
55NFP hardware is fully programmable therefore there can be different
56firmware images targeting different applications.
57
58When using application firmware from host, we recommend placing
59actual firmware files in application-named subdirectories in
60`/lib/firmware/netronome` and linking the desired files, e.g.::
61
62    $ tree /lib/firmware/netronome/
63    /lib/firmware/netronome/
64    ├── bpf
65    │   ├── nic_AMDA0081-0001_1x40.nffw
66    │   └── nic_AMDA0081-0001_4x10.nffw
67    ├── flower
68    │   ├── nic_AMDA0081-0001_1x40.nffw
69    │   └── nic_AMDA0081-0001_4x10.nffw
70    ├── nic
71    │   ├── nic_AMDA0081-0001_1x40.nffw
72    │   └── nic_AMDA0081-0001_4x10.nffw
73    ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw
74    └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw
75
76    3 directories, 8 files
77
78You may need to use hard instead of symbolic links on distributions
79which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu).
80
81After changing firmware files you may need to regenerate the initramfs
82image.  Initramfs contains drivers and firmware files your system may
83need to boot.  Refer to the documentation of your distribution to find
84out how to update initramfs.  Good indication of stale initramfs
85is system loading wrong driver or firmware on boot, but when driver is
86later reloaded manually everything works correctly.
87
88Selecting firmware per device
89-----------------------------
90
91Most commonly all cards on the system use the same type of firmware.
92If you want to load specific firmware image for a specific card, you
93can use either the PCI bus address or serial number.  Driver will print
94which files it's looking for when it recognizes a NFP device::
95
96    nfp: Looking for firmware file in order of priority:
97    nfp:  netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found
98    nfp:  netronome/pci-0000:02:00.0.nffw: not found
99    nfp:  netronome/nic_AMDA0081-0001_1x40.nffw: found, loading...
100
101In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw*
102or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this
103firmware file will take precedence over `nic_AMDA*` files.
104
105Note that `serial-*` and `pci-*` files are **not** automatically included
106in initramfs, you will have to refer to documentation of appropriate tools
107to find out how to include them.
108
109Firmware loading policy
110-----------------------
111
112Firmware loading policy is controlled via three HWinfo parameters
113stored as key value pairs in the device flash:
114
115app_fw_from_flash
116    Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or
117    the 'Preferred' (2) firmware. When 'Preferred' is selected, the management
118    firmware makes the decision over which firmware will be loaded by comparing
119    versions of the flash firmware and the host supplied firmware.
120    This variable is configurable using the 'fw_load_policy'
121    devlink parameter.
122
123abi_drv_reset
124    Defines if the driver should reset the firmware when
125    the driver is probed, either 'Disk' (0) if firmware was found on disk,
126    'Always' (1) reset or 'Never' (2) reset. Note that the device is always
127    reset on driver unload if firmware was loaded when the driver was probed.
128    This variable is configurable using the 'reset_dev_on_drv_probe'
129    devlink parameter.
130
131abi_drv_load_ifc
132    Defines a list of PF devices allowed to load FW on the device.
133    This variable is not currently user configurable.
134
135Statistics
136==========
137
138Following device statistics are available through the ``ethtool -S`` interface:
139
140.. flat-table:: NFP device statistics
141   :header-rows: 1
142   :widths: 3 1 11
143
144   * - Name
145     - ID
146     - Meaning
147
148   * - dev_rx_discards
149     - 1
150     - Packet can be discarded on the RX path for one of the following reasons:
151
152        * The NIC is not in promisc mode, and the destination MAC address
153          doesn't match the interfaces' MAC address.
154        * The received packet is larger than the max buffer size on the host.
155          I.e. it exceeds the Layer 3 MRU.
156        * There is no freelist descriptor available on the host for the packet.
157          It is likely that the NIC couldn't cache one in time.
158        * A BPF program discarded the packet.
159        * The datapath drop action was executed.
160        * The MAC discarded the packet due to lack of ingress buffer space
161          on the NIC.
162
163   * - dev_rx_errors
164     - 2
165     - A packet can be counted (and dropped) as RX error for the following
166       reasons:
167
168       * A problem with the VEB lookup (only when SR-IOV is used).
169       * A physical layer problem that causes Ethernet errors, like FCS or
170         alignment errors. The cause is usually faulty cables or SFPs.
171
172   * - dev_rx_bytes
173     - 3
174     - Total number of bytes received.
175
176   * - dev_rx_uc_bytes
177     - 4
178     - Unicast bytes received.
179
180   * - dev_rx_mc_bytes
181     - 5
182     - Multicast bytes received.
183
184   * - dev_rx_bc_bytes
185     - 6
186     - Broadcast bytes received.
187
188   * - dev_rx_pkts
189     - 7
190     - Total number of packets received.
191
192   * - dev_rx_mc_pkts
193     - 8
194     - Multicast packets received.
195
196   * - dev_rx_bc_pkts
197     - 9
198     - Broadcast packets received.
199
200   * - dev_tx_discards
201     - 10
202     - A packet can be discarded in the TX direction if the MAC is
203       being flow controlled and the NIC runs out of TX queue space.
204
205   * - dev_tx_errors
206     - 11
207     - A packet can be counted as TX error (and dropped) for one for the
208       following reasons:
209
210       * The packet is an LSO segment, but the Layer 3 or Layer 4 offset
211         could not be determined. Therefore LSO could not continue.
212       * An invalid packet descriptor was received over PCIe.
213       * The packet Layer 3 length exceeds the device MTU.
214       * An error on the MAC/physical layer. Usually due to faulty cables or
215         SFPs.
216       * A CTM buffer could not be allocated.
217       * The packet offset was incorrect and could not be fixed by the NIC.
218
219   * - dev_tx_bytes
220     - 12
221     - Total number of bytes transmitted.
222
223   * - dev_tx_uc_bytes
224     - 13
225     - Unicast bytes transmitted.
226
227   * - dev_tx_mc_bytes
228     - 14
229     - Multicast bytes transmitted.
230
231   * - dev_tx_bc_bytes
232     - 15
233     - Broadcast bytes transmitted.
234
235   * - dev_tx_pkts
236     - 16
237     - Total number of packets transmitted.
238
239   * - dev_tx_mc_pkts
240     - 17
241     - Multicast packets transmitted.
242
243   * - dev_tx_bc_pkts
244     - 18
245     - Broadcast packets transmitted.
246
247Note that statistics unknown to the driver will be displayed as
248``dev_unknown_stat$ID``, where ``$ID`` refers to the second column
249above.
250