1.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
2.. include:: <isonum.txt>
3
4================
5Ethtool counters
6================
7
8:Copyright: |copy| 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
9
10Contents
11========
12
13- `Overview`_
14- `Groups`_
15- `Types`_
16- `Descriptions`_
17
18Overview
19========
20
21There are several counter groups based on where the counter is being counted. In
22addition, each group of counters may have different counter types.
23
24These counter groups are based on which component in a networking setup,
25illustrated below, that they describe::
26
27                                                  ----------------------------------------
28                                                  |                                      |
29    ----------------------------------------    ---------------------------------------- |
30    |              Hypervisor              |    |                  VM                  | |
31    |                                      |    |                                      | |
32    | -------------------  --------------- |    | -------------------  --------------- | |
33    | | Ethernet driver |  | RDMA driver | |    | | Ethernet driver |  | RDMA driver | | |
34    | -------------------  --------------- |    | -------------------  --------------- | |
35    |           |                 |        |    |           |                 |        | |
36    |           -------------------        |    |           -------------------        | |
37    |                   |                  |    |                   |                  |--
38    ----------------------------------------    ----------------------------------------
39                        |                                           |
40            -------------               -----------------------------
41            |                           |
42         ------                      ------ ------ ------         ------      ------      ------
43    -----| PF |----------------------| VF |-| VF |-| VF |-----  --| PF |--- --| PF |--- --| PF |---
44    |    ------                      ------ ------ ------    |  | ------  | | ------  | | ------  |
45    |                                                        |  |         | |         | |         |
46    |                                                        |  |         | |         | |         |
47    |                                                        |  |         | |         | |         |
48    | eSwitch                                                |  | eSwitch | | eSwitch | | eSwitch |
49    ----------------------------------------------------------  ----------- ----------- -----------
50               -------------------------------------------------------------------------------
51               |                                                                             |
52               |                                                                             |
53               | Uplink (no counters)                                                        |
54               -------------------------------------------------------------------------------
55                       ---------------------------------------------------------------
56                       |                                                             |
57                       |                                                             |
58                       | MPFS (no counters)                                          |
59                       ---------------------------------------------------------------
60                                                     |
61                                                     |
62                                                     | Port
63
64Groups
65======
66
67Ring
68  Software counters populated by the driver stack.
69
70Netdev
71  An aggregation of software ring counters.
72
73vPort counters
74  Traffic counters and drops due to steering or no buffers. May indicate issues
75  with NIC. These counters include Ethernet traffic counters (including Raw
76  Ethernet) and RDMA/RoCE traffic counters.
77
78Physical port counters
79  Counters that collect statistics about the PFs and VFs. May indicate issues
80  with NIC, link, or network. This measuring point holds information on
81  standardized counters like IEEE 802.3, RFC2863, RFC 2819, RFC 3635 and
82  additional counters like flow control, FEC and more. Physical port counters
83  are not exposed to virtual machines.
84
85Priority Port Counters
86  A set of the physical port counters, per priority per port.
87
88Types
89=====
90
91Counters are divided into three types.
92
93Traffic Informative Counters
94  Counters which count traffic. These counters can be used for load estimation
95  or for general debug.
96
97Traffic Acceleration Counters
98  Counters which count traffic that was accelerated by Mellanox driver or by
99  hardware. The counters are an additional layer to the informative counter set,
100  and the same traffic is counted in both informative and acceleration counters.
101
102.. [#accel] Traffic acceleration counter.
103
104Error Counters
105  Increment of these counters might indicate a problem. Each of these counters
106  has an explanation and correction action.
107
108Statistic can be fetched via the `ip link` or `ethtool` commands. `ethtool`
109provides more detailed information.::
110
111    ip –s link show <if-name>
112    ethtool -S <if-name>
113
114Descriptions
115============
116
117XSK, PTP, and QoS counters that are similar to counters defined previously will
118not be separately listed. For example, `ptp_tx[i]_packets` will not be
119explicitly documented since `tx[i]_packets` describes the behavior of both
120counters, except `ptp_tx[i]_packets` is only counted when precision time
121protocol is used.
122
123Ring / Netdev Counter
124----------------------------
125The following counters are available per ring or software port.
126
127These counters provide information on the amount of traffic that was accelerated
128by the NIC. The counters are counting the accelerated traffic in addition to the
129standard counters which counts it (i.e. accelerated traffic is counted twice).
130
131The counter names in the table below refers to both ring and port counters. The
132notation for ring counters includes the [i] index without the braces. The
133notation for port counters doesn't include the [i]. A counter name
134`rx[i]_packets` will be printed as `rx0_packets` for ring 0 and `rx_packets` for
135the software port.
136
137.. flat-table:: Ring / Software Port Counter Table
138   :widths: 2 3 1
139
140   * - Counter
141     - Description
142     - Type
143
144   * - `rx[i]_packets`
145     - The number of packets received on ring i.
146     - Informative
147
148   * - `rx[i]_bytes`
149     - The number of bytes received on ring i.
150     - Informative
151
152   * - `tx[i]_packets`
153     - The number of packets transmitted on ring i.
154     - Informative
155
156   * - `tx[i]_bytes`
157     - The number of bytes transmitted on ring i.
158     - Informative
159
160   * - `tx[i]_recover`
161     - The number of times the SQ was recovered.
162     - Error
163
164   * - `tx[i]_cqes`
165     - Number of CQEs events on SQ issued on ring i.
166     - Informative
167
168   * - `tx[i]_cqe_err`
169     - The number of error CQEs encountered on the SQ for ring i.
170     - Error
171
172   * - `tx[i]_tso_packets`
173     - The number of TSO packets transmitted on ring i [#accel]_.
174     - Acceleration
175
176   * - `tx[i]_tso_bytes`
177     - The number of TSO bytes transmitted on ring i [#accel]_.
178     - Acceleration
179
180   * - `tx[i]_tso_inner_packets`
181     - The number of TSO packets which are indicated to be carry internal
182       encapsulation transmitted on ring i [#accel]_.
183     - Acceleration
184
185   * - `tx[i]_tso_inner_bytes`
186     - The number of TSO bytes which are indicated to be carry internal
187       encapsulation transmitted on ring i [#accel]_.
188     - Acceleration
189
190   * - `rx[i]_gro_packets`
191     - Number of received packets processed using hardware-accelerated GRO. The
192       number of hardware GRO offloaded packets received on ring i.
193     - Acceleration
194
195   * - `rx[i]_gro_bytes`
196     - Number of received bytes processed using hardware-accelerated GRO. The
197       number of hardware GRO offloaded bytes received on ring i.
198     - Acceleration
199
200   * - `rx[i]_gro_skbs`
201     - The number of receive SKBs constructed while performing
202       hardware-accelerated GRO.
203     - Informative
204
205   * - `rx[i]_gro_match_packets`
206     - Number of received packets processed using hardware-accelerated GRO that
207       met the flow table match criteria.
208     - Informative
209
210   * - `rx[i]_gro_large_hds`
211     - Number of receive packets using hardware-accelerated GRO that have large
212       headers that require additional memory to be allocated.
213     - Informative
214
215   * - `rx[i]_lro_packets`
216     - The number of LRO packets received on ring i [#accel]_.
217     - Acceleration
218
219   * - `rx[i]_lro_bytes`
220     - The number of LRO bytes received on ring i [#accel]_.
221     - Acceleration
222
223   * - `rx[i]_ecn_mark`
224     - The number of received packets where the ECN mark was turned on.
225     - Informative
226
227   * - `rx_oversize_pkts_buffer`
228     - The number of dropped received packets due to length which arrived to RQ
229       and exceed software buffer size allocated by the device for incoming
230       traffic. It might imply that the device MTU is larger than the software
231       buffers size.
232     - Error
233
234   * - `rx_oversize_pkts_sw_drop`
235     - Number of received packets dropped in software because the CQE data is
236       larger than the MTU size.
237     - Error
238
239   * - `rx[i]_csum_unnecessary`
240     - Packets received with a `CHECKSUM_UNNECESSARY` on ring i [#accel]_.
241     - Acceleration
242
243   * - `rx[i]_csum_unnecessary_inner`
244     - Packets received with inner encapsulation with a `CHECKSUM_UNNECESSARY`
245       on ring i [#accel]_.
246     - Acceleration
247
248   * - `rx[i]_csum_none`
249     - Packets received with a `CHECKSUM_NONE` on ring i [#accel]_.
250     - Acceleration
251
252   * - `rx[i]_csum_complete`
253     - Packets received with a `CHECKSUM_COMPLETE` on ring i [#accel]_.
254     - Acceleration
255
256   * - `rx[i]_csum_complete_tail`
257     - Number of received packets that had checksum calculation computed,
258       potentially needed padding, and were able to do so with
259       `CHECKSUM_PARTIAL`.
260     - Informative
261
262   * - `rx[i]_csum_complete_tail_slow`
263     - Number of received packets that need padding larger than eight bytes for
264       the checksum.
265     - Informative
266
267   * - `tx[i]_csum_partial`
268     - Packets transmitted with a `CHECKSUM_PARTIAL` on ring i [#accel]_.
269     - Acceleration
270
271   * - `tx[i]_csum_partial_inner`
272     - Packets transmitted with inner encapsulation with a `CHECKSUM_PARTIAL` on
273       ring i [#accel]_.
274     - Acceleration
275
276   * - `tx[i]_csum_none`
277     - Packets transmitted with no hardware checksum acceleration on ring i.
278     - Informative
279
280   * - `tx[i]_stopped` / `tx_queue_stopped` [#ring_global]_
281     - Events where SQ was full on ring i. If this counter is increased, check
282       the amount of buffers allocated for transmission.
283     - Informative
284
285   * - `tx[i]_wake` / `tx_queue_wake` [#ring_global]_
286     - Events where SQ was full and has become not full on ring i.
287     - Informative
288
289   * - `tx[i]_dropped` / `tx_queue_dropped` [#ring_global]_
290     - Packets transmitted that were dropped due to DMA mapping failure on
291       ring i. If this counter is increased, check the amount of buffers
292       allocated for transmission.
293     - Error
294
295   * - `tx[i]_nop`
296     - The number of nop WQEs (empty WQEs) inserted to the SQ (related to
297       ring i) due to the reach of the end of the cyclic buffer. When reaching
298       near to the end of cyclic buffer the driver may add those empty WQEs to
299       avoid handling a state the a WQE start in the end of the queue and ends
300       in the beginning of the queue. This is a normal condition.
301     - Informative
302
303   * - `tx[i]_added_vlan_packets`
304     - The number of packets sent where vlan tag insertion was offloaded to the
305       hardware.
306     - Acceleration
307
308   * - `rx[i]_removed_vlan_packets`
309     - The number of packets received where vlan tag stripping was offloaded to
310       the hardware.
311     - Acceleration
312
313   * - `rx[i]_wqe_err`
314     - The number of wrong opcodes received on ring i.
315     - Error
316
317   * - `rx[i]_mpwqe_frag`
318     - The number of WQEs that failed to allocate compound page and hence
319       fragmented MPWQE’s (Multi Packet WQEs) were used on ring i. If this
320       counter raise, it may suggest that there is no enough memory for large
321       pages, the driver allocated fragmented pages. This is not abnormal
322       condition.
323     - Informative
324
325   * - `rx[i]_mpwqe_filler_cqes`
326     - The number of filler CQEs events that were issued on ring i.
327     - Informative
328
329   * - `rx[i]_mpwqe_filler_strides`
330     - The number of strides consumed by filler CQEs on ring i.
331     - Informative
332
333   * - `tx[i]_mpwqe_blks`
334     - The number of send blocks processed from Multi-Packet WQEs (mpwqe).
335     - Informative
336
337   * - `tx[i]_mpwqe_pkts`
338     - The number of send packets processed from Multi-Packet WQEs (mpwqe).
339     - Informative
340
341   * - `rx[i]_cqe_compress_blks`
342     - The number of receive blocks with CQE compression on ring i [#accel]_.
343     - Acceleration
344
345   * - `rx[i]_cqe_compress_pkts`
346     - The number of receive packets with CQE compression on ring i [#accel]_.
347     - Acceleration
348
349   * - `rx[i]_arfs_err`
350     - Number of flow rules that failed to be added to the flow table.
351     - Error
352
353   * - `rx[i]_recover`
354     - The number of times the RQ was recovered.
355     - Error
356
357   * - `tx[i]_xmit_more`
358     - The number of packets sent with `xmit_more` indication set on the skbuff
359       (no doorbell).
360     - Acceleration
361
362   * - `ch[i]_poll`
363     - The number of invocations of NAPI poll of channel i.
364     - Informative
365
366   * - `ch[i]_arm`
367     - The number of times the NAPI poll function completed and armed the
368       completion queues on channel i.
369     - Informative
370
371   * - `ch[i]_aff_change`
372     - The number of times the NAPI poll function explicitly stopped execution
373       on a CPU due to a change in affinity, on channel i.
374     - Informative
375
376   * - `ch[i]_events`
377     - The number of hard interrupt events on the completion queues of channel i.
378     - Informative
379
380   * - `ch[i]_eq_rearm`
381     - The number of times the EQ was recovered.
382     - Error
383
384   * - `ch[i]_force_irq`
385     - Number of times NAPI is triggered by XSK wakeups by posting a NOP to
386       ICOSQ.
387     - Acceleration
388
389   * - `rx[i]_congst_umr`
390     - The number of times an outstanding UMR request is delayed due to
391       congestion, on ring i.
392     - Informative
393
394   * - `rx_pp_alloc_fast`
395     - Number of successful fast path allocations.
396     - Informative
397
398   * - `rx_pp_alloc_slow`
399     - Number of slow path order-0 allocations.
400     - Informative
401
402   * - `rx_pp_alloc_slow_high_order`
403     - Number of slow path high order allocations.
404     - Informative
405
406   * - `rx_pp_alloc_empty`
407     - Counter is incremented when ptr ring is empty, so a slow path allocation
408       was forced.
409     - Informative
410
411   * - `rx_pp_alloc_refill`
412     - Counter is incremented when an allocation which triggered a refill of the
413       cache.
414     - Informative
415
416   * - `rx_pp_alloc_waive`
417     - Counter is incremented when pages obtained from the ptr ring that cannot
418       be added to the cache due to a NUMA mismatch.
419     - Informative
420
421   * - `rx_pp_recycle_cached`
422     - Counter is incremented when recycling placed page in the page pool cache.
423     - Informative
424
425   * - `rx_pp_recycle_cache_full`
426     - Counter is incremented when page pool cache was full.
427     - Informative
428
429   * - `rx_pp_recycle_ring`
430     - Counter is incremented when page placed into the ptr ring.
431     - Informative
432
433   * - `rx_pp_recycle_ring_full`
434     - Counter is incremented when page released from page pool because the ptr
435       ring was full.
436     - Informative
437
438   * - `rx_pp_recycle_released_ref`
439     - Counter is incremented when page released (and not recycled) because
440       refcnt > 1.
441     - Informative
442
443   * - `rx[i]_xsk_buff_alloc_err`
444     - The number of times allocating an skb or XSK buffer failed in the XSK RQ
445       context.
446     - Error
447
448   * - `rx[i]_xsk_arfs_err`
449     - aRFS (accelerated Receive Flow Steering) does not occur in the XSK RQ
450       context, so this counter should never increment.
451     - Error
452
453   * - `rx[i]_xdp_tx_xmit`
454     - The number of packets forwarded back to the port due to XDP program
455       `XDP_TX` action (bouncing). these packets are not counted by other
456       software counters. These packets are counted by physical port and vPort
457       counters.
458     - Informative
459
460   * - `rx[i]_xdp_tx_mpwqe`
461     - Number of multi-packet WQEs transmitted by the netdev and `XDP_TX`-ed by
462       the netdev during the RQ context.
463     - Acceleration
464
465   * - `rx[i]_xdp_tx_inlnw`
466     - Number of WQE data segments transmitted where the data could be inlined
467       in the WQE and then `XDP_TX`-ed during the RQ context.
468     - Acceleration
469
470   * - `rx[i]_xdp_tx_nops`
471     - Number of NOP WQEBBs (WQE building blocks) received posted to the XDP SQ.
472     - Acceleration
473
474   * - `rx[i]_xdp_tx_full`
475     - The number of packets that should have been forwarded back to the port
476       due to `XDP_TX` action but were dropped due to full tx queue. These packets
477       are not counted by other software counters. These packets are counted by
478       physical port and vPort counters. You may open more rx queues and spread
479       traffic rx over all queues and/or increase rx ring size.
480     - Error
481
482   * - `rx[i]_xdp_tx_err`
483     - The number of times an `XDP_TX` error such as frame too long and frame
484       too short occurred on `XDP_TX` ring of RX ring.
485     - Error
486
487   * - `rx[i]_xdp_tx_cqes` / `rx_xdp_tx_cqe` [#ring_global]_
488     - The number of completions received on the CQ of the `XDP_TX` ring.
489     - Informative
490
491   * - `rx[i]_xdp_drop`
492     - The number of packets dropped due to XDP program `XDP_DROP` action. these
493       packets are not counted by other software counters. These packets are
494       counted by physical port and vPort counters.
495     - Informative
496
497   * - `rx[i]_xdp_redirect`
498     - The number of times an XDP redirect action was triggered on ring i.
499     - Acceleration
500
501   * - `tx[i]_xdp_xmit`
502     - The number of packets redirected to the interface(due to XDP redirect).
503       These packets are not counted by other software counters. These packets
504       are counted by physical port and vPort counters.
505     - Informative
506
507   * - `tx[i]_xdp_full`
508     - The number of packets redirected to the interface(due to XDP redirect),
509       but were dropped due to full tx queue. these packets are not counted by
510       other software counters. you may enlarge tx queues.
511     - Informative
512
513   * - `tx[i]_xdp_mpwqe`
514     - Number of multi-packet WQEs offloaded onto the NIC that were
515       `XDP_REDIRECT`-ed from other netdevs.
516     - Acceleration
517
518   * - `tx[i]_xdp_inlnw`
519     - Number of WQE data segments where the data could be inlined in the WQE
520       where the data segments were `XDP_REDIRECT`-ed from other netdevs.
521     - Acceleration
522
523   * - `tx[i]_xdp_nops`
524     - Number of NOP WQEBBs (WQE building blocks) posted to the SQ that were
525       `XDP_REDIRECT`-ed from other netdevs.
526     - Acceleration
527
528   * - `tx[i]_xdp_err`
529     - The number of packets redirected to the interface(due to XDP redirect)
530       but were dropped due to error such as frame too long and frame too short.
531     - Error
532
533   * - `tx[i]_xdp_cqes`
534     - The number of completions received for packets redirected to the
535       interface(due to XDP redirect) on the CQ.
536     - Informative
537
538   * - `tx[i]_xsk_xmit`
539     - The number of packets transmitted using XSK zerocopy functionality.
540     - Acceleration
541
542   * - `tx[i]_xsk_mpwqe`
543     - Number of multi-packet WQEs offloaded onto the NIC that were
544       `XDP_REDIRECT`-ed from other netdevs.
545     - Acceleration
546
547   * - `tx[i]_xsk_inlnw`
548     - Number of WQE data segments where the data could be inlined in the WQE
549       that are transmitted using XSK zerocopy.
550     - Acceleration
551
552   * - `tx[i]_xsk_full`
553     - Number of times doorbell is rung in XSK zerocopy mode when SQ is full.
554     - Error
555
556   * - `tx[i]_xsk_err`
557     - Number of errors that occurred in XSK zerocopy mode such as if the data
558       size is larger than the MTU size.
559     - Error
560
561   * - `tx[i]_xsk_cqes`
562     - Number of CQEs processed in XSK zerocopy mode.
563     - Acceleration
564
565   * - `tx_tls_ctx`
566     - Number of TLS TX HW offload contexts added to device for encryption.
567     - Acceleration
568
569   * - `tx_tls_del`
570     - Number of TLS TX HW offload contexts removed from device (connection
571       closed).
572     - Acceleration
573
574   * - `tx_tls_pool_alloc`
575     - Number of times a unit of work is successfully allocated in the TLS HW
576       offload pool.
577     - Acceleration
578
579   * - `tx_tls_pool_free`
580     - Number of times a unit of work is freed in the TLS HW offload pool.
581     - Acceleration
582
583   * - `rx_tls_ctx`
584     - Number of TLS RX HW offload contexts added to device for decryption.
585     - Acceleration
586
587   * - `rx_tls_del`
588     - Number of TLS RX HW offload contexts deleted from device (connection has
589       finished).
590     - Acceleration
591
592   * - `rx[i]_tls_decrypted_packets`
593     - Number of successfully decrypted RX packets which were part of a TLS
594       stream.
595     - Acceleration
596
597   * - `rx[i]_tls_decrypted_bytes`
598     - Number of TLS payload bytes in RX packets which were successfully
599       decrypted.
600     - Acceleration
601
602   * - `rx[i]_tls_resync_req_pkt`
603     - Number of received TLS packets with a resync request.
604     - Acceleration
605
606   * - `rx[i]_tls_resync_req_start`
607     - Number of times the TLS async resync request was started.
608     - Acceleration
609
610   * - `rx[i]_tls_resync_req_end`
611     - Number of times the TLS async resync request properly ended with
612       providing the HW tracked tcp-seq.
613     - Acceleration
614
615   * - `rx[i]_tls_resync_req_skip`
616     - Number of times the TLS async resync request procedure was started but
617       not properly ended.
618     - Error
619
620   * - `rx[i]_tls_resync_res_ok`
621     - Number of times the TLS resync response call to the driver was
622       successfully handled.
623     - Acceleration
624
625   * - `rx[i]_tls_resync_res_retry`
626     - Number of times the TLS resync response call to the driver was
627       reattempted when ICOSQ is full.
628     - Error
629
630   * - `rx[i]_tls_resync_res_skip`
631     - Number of times the TLS resync response call to the driver was terminated
632       unsuccessfully.
633     - Error
634
635   * - `rx[i]_tls_err`
636     - Number of times when CQE TLS offload was problematic.
637     - Error
638
639   * - `tx[i]_tls_encrypted_packets`
640     - The number of send packets that are TLS encrypted by the kernel.
641     - Acceleration
642
643   * - `tx[i]_tls_encrypted_bytes`
644     - The number of send bytes that are TLS encrypted by the kernel.
645     - Acceleration
646
647   * - `tx[i]_tls_ooo`
648     - Number of times out of order TLS SQE fragments were handled on ring i.
649     - Acceleration
650
651   * - `tx[i]_tls_dump_packets`
652     - Number of TLS decrypted packets copied over from NIC over DMA.
653     - Acceleration
654
655   * - `tx[i]_tls_dump_bytes`
656     - Number of TLS decrypted bytes copied over from NIC over DMA.
657     - Acceleration
658
659   * - `tx[i]_tls_resync_bytes`
660     - Number of TLS bytes requested to be resynchronized in order to be
661       decrypted.
662     - Acceleration
663
664   * - `tx[i]_tls_skip_no_sync_data`
665     - Number of TLS send data that can safely be skipped / do not need to be
666       decrypted.
667     - Acceleration
668
669   * - `tx[i]_tls_drop_no_sync_data`
670     - Number of TLS send data that were dropped due to retransmission of TLS
671       data.
672     - Acceleration
673
674   * - `ptp_cq[i]_abort`
675     - Number of times a CQE has to be skipped in precision time protocol due to
676       a skew between the port timestamp and CQE timestamp being greater than
677       128 seconds.
678     - Error
679
680   * - `ptp_cq[i]_abort_abs_diff_ns`
681     - Accumulation of time differences between the port timestamp and CQE
682       timestamp when the difference is greater than 128 seconds in precision
683       time protocol.
684     - Error
685
686.. [#ring_global] The corresponding ring and global counters do not share the
687                  same name (i.e. do not follow the common naming scheme).
688
689vPort Counters
690--------------
691Counters on the NIC port that is connected to a eSwitch.
692
693.. flat-table:: vPort Counter Table
694   :widths: 2 3 1
695
696   * - Counter
697     - Description
698     - Type
699
700   * - `rx_vport_unicast_packets`
701     - Unicast packets received, steered to a port including Raw Ethernet
702       QP/DPDK traffic, excluding RDMA traffic.
703     - Informative
704
705   * - `rx_vport_unicast_bytes`
706     - Unicast bytes received, steered to a port including Raw Ethernet QP/DPDK
707       traffic, excluding RDMA traffic.
708     - Informative
709
710   * - `tx_vport_unicast_packets`
711     - Unicast packets transmitted, steered from a port including Raw Ethernet
712       QP/DPDK traffic, excluding RDMA traffic.
713     - Informative
714
715   * - `tx_vport_unicast_bytes`
716     - Unicast bytes transmitted, steered from a port including Raw Ethernet
717       QP/DPDK traffic, excluding RDMA traffic.
718     - Informative
719
720   * - `rx_vport_multicast_packets`
721     - Multicast packets received, steered to a port including Raw Ethernet
722       QP/DPDK traffic, excluding RDMA traffic.
723     - Informative
724
725   * - `rx_vport_multicast_bytes`
726     - Multicast bytes received, steered to a port including Raw Ethernet
727       QP/DPDK traffic, excluding RDMA traffic.
728     - Informative
729
730   * - `tx_vport_multicast_packets`
731     - Multicast packets transmitted, steered from a port including Raw Ethernet
732       QP/DPDK traffic, excluding RDMA traffic.
733     - Informative
734
735   * - `tx_vport_multicast_bytes`
736     - Multicast bytes transmitted, steered from a port including Raw Ethernet
737       QP/DPDK traffic, excluding RDMA traffic.
738     - Informative
739
740   * - `rx_vport_broadcast_packets`
741     - Broadcast packets received, steered to a port including Raw Ethernet
742       QP/DPDK traffic, excluding RDMA traffic.
743     - Informative
744
745   * - `rx_vport_broadcast_bytes`
746     - Broadcast bytes received, steered to a port including Raw Ethernet
747       QP/DPDK traffic, excluding RDMA traffic.
748     - Informative
749
750   * - `tx_vport_broadcast_packets`
751     - Broadcast packets transmitted, steered from a port including Raw Ethernet
752       QP/DPDK traffic, excluding RDMA traffic.
753     - Informative
754
755   * - `tx_vport_broadcast_bytes`
756     - Broadcast bytes transmitted, steered from a port including Raw Ethernet
757       QP/DPDK traffic, excluding RDMA traffic.
758     - Informative
759
760   * - `rx_vport_rdma_unicast_packets`
761     - RDMA unicast packets received, steered to a port (counters counts
762       RoCE/UD/RC traffic) [#accel]_.
763     - Acceleration
764
765   * - `rx_vport_rdma_unicast_bytes`
766     - RDMA unicast bytes received, steered to a port (counters counts
767       RoCE/UD/RC traffic) [#accel]_.
768     - Acceleration
769
770   * - `tx_vport_rdma_unicast_packets`
771     - RDMA unicast packets transmitted, steered from a port (counters counts
772       RoCE/UD/RC traffic) [#accel]_.
773     - Acceleration
774
775   * - `tx_vport_rdma_unicast_bytes`
776     - RDMA unicast bytes transmitted, steered from a port (counters counts
777       RoCE/UD/RC traffic) [#accel]_.
778     - Acceleration
779
780   * - `rx_vport_rdma_multicast_packets`
781     - RDMA multicast packets received, steered to a port (counters counts
782       RoCE/UD/RC traffic) [#accel]_.
783     - Acceleration
784
785   * - `rx_vport_rdma_multicast_bytes`
786     - RDMA multicast bytes received, steered to a port (counters counts
787       RoCE/UD/RC traffic) [#accel]_.
788     - Acceleration
789
790   * - `tx_vport_rdma_multicast_packets`
791     - RDMA multicast packets transmitted, steered from a port (counters counts
792       RoCE/UD/RC traffic) [#accel]_.
793     - Acceleration
794
795   * - `tx_vport_rdma_multicast_bytes`
796     - RDMA multicast bytes transmitted, steered from a port (counters counts
797       RoCE/UD/RC traffic) [#accel]_.
798     - Acceleration
799
800   * - `rx_steer_missed_packets`
801     - Number of packets that was received by the NIC, however was discarded
802       because it did not match any flow in the NIC flow table.
803     - Error
804
805   * - `rx_packets`
806     - Representor only: packets received, that were handled by the hypervisor.
807     - Informative
808
809   * - `rx_bytes`
810     - Representor only: bytes received, that were handled by the hypervisor.
811     - Informative
812
813   * - `tx_packets`
814     - Representor only: packets transmitted, that were handled by the
815       hypervisor.
816     - Informative
817
818   * - `tx_bytes`
819     - Representor only: bytes transmitted, that were handled by the hypervisor.
820     - Informative
821
822   * - `dev_internal_queue_oob`
823     - The number of dropped packets due to lack of receive WQEs for an internal
824       device RQ.
825     - Error
826
827Physical Port Counters
828----------------------
829The physical port counters are the counters on the external port connecting the
830adapter to the network. This measuring point holds information on standardized
831counters like IEEE 802.3, RFC2863, RFC 2819, RFC 3635 and additional counters
832like flow control, FEC and more.
833
834.. flat-table:: Physical Port Counter Table
835   :widths: 2 3 1
836
837   * - Counter
838     - Description
839     - Type
840
841   * - `rx_packets_phy`
842     - The number of packets received on the physical port. This counter doesn’t
843       include packets that were discarded due to FCS, frame size and similar
844       errors.
845     - Informative
846
847   * - `tx_packets_phy`
848     - The number of packets transmitted on the physical port.
849     - Informative
850
851   * - `rx_bytes_phy`
852     - The number of bytes received on the physical port, including Ethernet
853       header and FCS.
854     - Informative
855
856   * - `tx_bytes_phy`
857     - The number of bytes transmitted on the physical port.
858     - Informative
859
860   * - `rx_multicast_phy`
861     - The number of multicast packets received on the physical port.
862     - Informative
863
864   * - `tx_multicast_phy`
865     - The number of multicast packets transmitted on the physical port.
866     - Informative
867
868   * - `rx_broadcast_phy`
869     - The number of broadcast packets received on the physical port.
870     - Informative
871
872   * - `tx_broadcast_phy`
873     - The number of broadcast packets transmitted on the physical port.
874     - Informative
875
876   * - `rx_crc_errors_phy`
877     - The number of dropped received packets due to FCS (Frame Check Sequence)
878       error on the physical port. If this counter is increased in high rate,
879       check the link quality using `rx_symbol_error_phy` and
880       `rx_corrected_bits_phy` counters below.
881     - Error
882
883   * - `rx_in_range_len_errors_phy`
884     - The number of received packets dropped due to length/type errors on a
885       physical port.
886     - Error
887
888   * - `rx_out_of_range_len_phy`
889     - The number of received packets dropped due to length greater than allowed
890       on a physical port. If this counter is increasing, it implies that the
891       peer connected to the adapter has a larger MTU configured. Using same MTU
892       configuration shall resolve this issue.
893     - Error
894
895   * - `rx_oversize_pkts_phy`
896     - The number of dropped received packets due to length which exceed MTU
897       size on a physical port. If this counter is increasing, it implies that
898       the peer connected to the adapter has a larger MTU configured. Using same
899       MTU configuration shall resolve this issue.
900     - Error
901
902   * - `rx_symbol_err_phy`
903     - The number of received packets dropped due to physical coding errors
904       (symbol errors) on a physical port.
905     - Error
906
907   * - `rx_mac_control_phy`
908     - The number of MAC control packets received on the physical port.
909     - Informative
910
911   * - `tx_mac_control_phy`
912     - The number of MAC control packets transmitted on the physical port.
913     - Informative
914
915   * - `rx_pause_ctrl_phy`
916     - The number of link layer pause packets received on a physical port. If
917       this counter is increasing, it implies that the network is congested and
918       cannot absorb the traffic coming from to the adapter.
919     - Informative
920
921   * - `tx_pause_ctrl_phy`
922     - The number of link layer pause packets transmitted on a physical port. If
923       this counter is increasing, it implies that the NIC is congested and
924       cannot absorb the traffic coming from the network.
925     - Informative
926
927   * - `rx_unsupported_op_phy`
928     - The number of MAC control packets received with unsupported opcode on a
929       physical port.
930     - Error
931
932   * - `rx_discards_phy`
933     - The number of received packets dropped due to lack of buffers on a
934       physical port. If this counter is increasing, it implies that the adapter
935       is congested and cannot absorb the traffic coming from the network.
936     - Error
937
938   * - `tx_discards_phy`
939     - The number of packets which were discarded on transmission, even no
940       errors were detected. the drop might occur due to link in down state,
941       head of line drop, pause from the network, etc.
942     - Error
943
944   * - `tx_errors_phy`
945     - The number of transmitted packets dropped due to a length which exceed
946       MTU size on a physical port.
947     - Error
948
949   * - `rx_undersize_pkts_phy`
950     - The number of received packets dropped due to length which is shorter
951       than 64 bytes on a physical port. If this counter is increasing, it
952       implies that the peer connected to the adapter has a non-standard MTU
953       configured or malformed packet had arrived.
954     - Error
955
956   * - `rx_fragments_phy`
957     - The number of received packets dropped due to a length which is shorter
958       than 64 bytes and has FCS error on a physical port. If this counter is
959       increasing, it implies that the peer connected to the adapter has a
960       non-standard MTU configured.
961     - Error
962
963   * - `rx_jabbers_phy`
964     - The number of received packets d due to a length which is longer than 64
965       bytes and had FCS error on a physical port.
966     - Error
967
968   * - `rx_64_bytes_phy`
969     - The number of packets received on the physical port with size of 64 bytes.
970     - Informative
971
972   * - `rx_65_to_127_bytes_phy`
973     - The number of packets received on the physical port with size of 65 to
974       127 bytes.
975     - Informative
976
977   * - `rx_128_to_255_bytes_phy`
978     - The number of packets received on the physical port with size of 128 to
979       255 bytes.
980     - Informative
981
982   * - `rx_256_to_511_bytes_phy`
983     - The number of packets received on the physical port with size of 256 to
984       512 bytes.
985     - Informative
986
987   * - `rx_512_to_1023_bytes_phy`
988     - The number of packets received on the physical port with size of 512 to
989       1023 bytes.
990     - Informative
991
992   * - `rx_1024_to_1518_bytes_phy`
993     - The number of packets received on the physical port with size of 1024 to
994       1518 bytes.
995     - Informative
996
997   * - `rx_1519_to_2047_bytes_phy`
998     - The number of packets received on the physical port with size of 1519 to
999       2047 bytes.
1000     - Informative
1001
1002   * - `rx_2048_to_4095_bytes_phy`
1003     - The number of packets received on the physical port with size of 2048 to
1004       4095 bytes.
1005     - Informative
1006
1007   * - `rx_4096_to_8191_bytes_phy`
1008     - The number of packets received on the physical port with size of 4096 to
1009       8191 bytes.
1010     - Informative
1011
1012   * - `rx_8192_to_10239_bytes_phy`
1013     - The number of packets received on the physical port with size of 8192 to
1014       10239 bytes.
1015     - Informative
1016
1017   * - `link_down_events_phy`
1018     - The number of times where the link operative state changed to down. In
1019       case this counter is increasing it may imply on port flapping. You may
1020       need to replace the cable/transceiver.
1021     - Error
1022
1023   * - `rx_out_of_buffer`
1024     - Number of times receive queue had no software buffers allocated for the
1025       adapter's incoming traffic.
1026     - Error
1027
1028   * - `module_bus_stuck`
1029     - The number of times that module's I\ :sup:`2`\C bus (data or clock)
1030       short-wire was detected. You may need to replace the cable/transceiver.
1031     - Error
1032
1033   * - `module_high_temp`
1034     - The number of times that the module temperature was too high. If this
1035       issue persist, you may need to check the ambient temperature or replace
1036       the cable/transceiver module.
1037     - Error
1038
1039   * - `module_bad_shorted`
1040     - The number of times that the module cables were shorted. You may need to
1041       replace the cable/transceiver module.
1042     - Error
1043
1044   * - `module_unplug`
1045     - The number of times that module was ejected.
1046     - Informative
1047
1048   * - `rx_buffer_passed_thres_phy`
1049     - The number of events where the port receive buffer was over 85% full.
1050     - Informative
1051
1052   * - `tx_pause_storm_warning_events`
1053     - The number of times the device was sending pauses for a long period of
1054       time.
1055     - Informative
1056
1057   * - `tx_pause_storm_error_events`
1058     - The number of times the device was sending pauses for a long period of
1059       time, reaching time out and disabling transmission of pause frames. on
1060       the period where pause frames were disabled, drop could have been
1061       occurred.
1062     - Error
1063
1064   * - `rx[i]_buff_alloc_err`
1065     - Failed to allocate a buffer to received packet (or SKB) on ring i.
1066     - Error
1067
1068   * - `rx_bits_phy`
1069     - This counter provides information on the total amount of traffic that
1070       could have been received and can be used as a guideline to measure the
1071       ratio of errored traffic in `rx_pcs_symbol_err_phy` and
1072       `rx_corrected_bits_phy`.
1073     - Informative
1074
1075   * - `rx_pcs_symbol_err_phy`
1076     - This counter counts the number of symbol errors that wasn’t corrected by
1077       FEC correction algorithm or that FEC algorithm was not active on this
1078       interface. If this counter is increasing, it implies that the link
1079       between the NIC and the network is suffering from high BER, and that
1080       traffic is lost. You may need to replace the cable/transceiver. The error
1081       rate is the number of `rx_pcs_symbol_err_phy` divided by the number of
1082       `rx_bits_phy` on a specific time frame.
1083     - Error
1084
1085   * - `rx_corrected_bits_phy`
1086     - The number of corrected bits on this port according to active FEC
1087       (RS/FC). If this counter is increasing, it implies that the link between
1088       the NIC and the network is suffering from high BER. The corrected bit
1089       rate is the number of `rx_corrected_bits_phy` divided by the number of
1090       `rx_bits_phy` on a specific time frame.
1091     - Error
1092
1093   * - `rx_err_lane_[l]_phy`
1094     - This counter counts the number of physical raw errors per lane l index.
1095       The counter counts errors before FEC corrections. If this counter is
1096       increasing, it implies that the link between the NIC and the network is
1097       suffering from high BER, and that traffic might be lost. You may need to
1098       replace the cable/transceiver. Please check in accordance with
1099       `rx_corrected_bits_phy`.
1100     - Error
1101
1102   * - `rx_global_pause`
1103     - The number of pause packets received on the physical port. If this
1104       counter is increasing, it implies that the network is congested and
1105       cannot absorb the traffic coming from the adapter. Note: This counter is
1106       only enabled when global pause mode is enabled.
1107     - Informative
1108
1109   * - `rx_global_pause_duration`
1110     - The duration of pause received (in microSec) on the physical port. The
1111       counter represents the time the port did not send any traffic. If this
1112       counter is increasing, it implies that the network is congested and
1113       cannot absorb the traffic coming from the adapter. Note: This counter is
1114       only enabled when global pause mode is enabled.
1115     - Informative
1116
1117   * - `tx_global_pause`
1118     - The number of pause packets transmitted on a physical port. If this
1119       counter is increasing, it implies that the adapter is congested and
1120       cannot absorb the traffic coming from the network. Note: This counter is
1121       only enabled when global pause mode is enabled.
1122     - Informative
1123
1124   * - `tx_global_pause_duration`
1125     - The duration of pause transmitter (in microSec) on the physical port.
1126       Note: This counter is only enabled when global pause mode is enabled.
1127     - Informative
1128
1129   * - `rx_global_pause_transition`
1130     - The number of times a transition from Xoff to Xon on the physical port
1131       has occurred. Note: This counter is only enabled when global pause mode
1132       is enabled.
1133     - Informative
1134
1135   * - `rx_if_down_packets`
1136     - The number of received packets that were dropped due to interface down.
1137     - Informative
1138
1139Priority Port Counters
1140----------------------
1141The following counters are physical port counters that are counted per L2
1142priority (0-7).
1143
1144**Note:** `p` in the counter name represents the priority.
1145
1146.. flat-table:: Priority Port Counter Table
1147   :widths: 2 3 1
1148
1149   * - Counter
1150     - Description
1151     - Type
1152
1153   * - `rx_prio[p]_bytes`
1154     - The number of bytes received with priority p on the physical port.
1155     - Informative
1156
1157   * - `rx_prio[p]_packets`
1158     - The number of packets received with priority p on the physical port.
1159     - Informative
1160
1161   * - `tx_prio[p]_bytes`
1162     - The number of bytes transmitted on priority p on the physical port.
1163     - Informative
1164
1165   * - `tx_prio[p]_packets`
1166     - The number of packets transmitted on priority p on the physical port.
1167     - Informative
1168
1169   * - `rx_prio[p]_pause`
1170     - The number of pause packets received with priority p on a physical port.
1171       If this counter is increasing, it implies that the network is congested
1172       and cannot absorb the traffic coming from the adapter. Note: This counter
1173       is available only if PFC was enabled on priority p.
1174     - Informative
1175
1176   * - `rx_prio[p]_pause_duration`
1177     - The duration of pause received (in microSec) on priority p on the
1178       physical port. The counter represents the time the port did not send any
1179       traffic on this priority. If this counter is increasing, it implies that
1180       the network is congested and cannot absorb the traffic coming from the
1181       adapter. Note: This counter is available only if PFC was enabled on
1182       priority p.
1183     - Informative
1184
1185   * - `rx_prio[p]_pause_transition`
1186     - The number of times a transition from Xoff to Xon on priority p on the
1187       physical port has occurred. Note: This counter is available only if PFC
1188       was enabled on priority p.
1189     - Informative
1190
1191   * - `tx_prio[p]_pause`
1192     - The number of pause packets transmitted on priority p on a physical port.
1193       If this counter is increasing, it implies that the adapter is congested
1194       and cannot absorb the traffic coming from the network. Note: This counter
1195       is available only if PFC was enabled on priority p.
1196     - Informative
1197
1198   * - `tx_prio[p]_pause_duration`
1199     - The duration of pause transmitter (in microSec) on priority p on the
1200       physical port. Note: This counter is available only if PFC was enabled on
1201       priority p.
1202     - Informative
1203
1204   * - `rx_prio[p]_buf_discard`
1205     - The number of packets discarded by device due to lack of per host receive
1206       buffers.
1207     - Informative
1208
1209   * - `rx_prio[p]_cong_discard`
1210     - The number of packets discarded by device due to per host congestion.
1211     - Informative
1212
1213   * - `rx_prio[p]_marked`
1214     - The number of packets ecn marked by device due to per host congestion.
1215     - Informative
1216
1217   * - `rx_prio[p]_discards`
1218     - The number of packets discarded by device due to lack of receive buffers.
1219     - Informative
1220
1221Device Counters
1222---------------
1223.. flat-table:: Device Counter Table
1224   :widths: 2 3 1
1225
1226   * - Counter
1227     - Description
1228     - Type
1229
1230   * - `rx_pci_signal_integrity`
1231     - Counts physical layer PCIe signal integrity errors, the number of
1232       transitions to recovery due to Framing errors and CRC (dlp and tlp). If
1233       this counter is raising, try moving the adapter card to a different slot
1234       to rule out a bad PCI slot. Validate that you are running with the latest
1235       firmware available and latest server BIOS version.
1236     - Error
1237
1238   * - `tx_pci_signal_integrity`
1239     - Counts physical layer PCIe signal integrity errors, the number of
1240       transition to recovery initiated by the other side (moving to recovery
1241       due to getting TS/EIEOS). If this counter is raising, try moving the
1242       adapter card to a different slot to rule out a bad PCI slot. Validate
1243       that you are running with the latest firmware available and latest server
1244       BIOS version.
1245     - Error
1246
1247   * - `outbound_pci_buffer_overflow`
1248     - The number of packets dropped due to pci buffer overflow. If this counter
1249       is raising in high rate, it might indicate that the receive traffic rate
1250       for a host is larger than the PCIe bus and therefore a congestion occurs.
1251     - Informative
1252
1253   * - `outbound_pci_stalled_rd`
1254     - The percentage (in the range 0...100) of time within the last second that
1255       the NIC had outbound non-posted reads requests but could not perform the
1256       operation due to insufficient posted credits.
1257     - Informative
1258
1259   * - `outbound_pci_stalled_wr`
1260     - The percentage (in the range 0...100) of time within the last second that
1261       the NIC had outbound posted writes requests but could not perform the
1262       operation due to insufficient posted credits.
1263     - Informative
1264
1265   * - `outbound_pci_stalled_rd_events`
1266     - The number of seconds where `outbound_pci_stalled_rd` was above 30%.
1267     - Informative
1268
1269   * - `outbound_pci_stalled_wr_events`
1270     - The number of seconds where `outbound_pci_stalled_wr` was above 30%.
1271     - Informative
1272
1273   * - `dev_out_of_buffer`
1274     - The number of times the device owned queue had not enough buffers
1275       allocated.
1276     - Error
1277