Documentation/networking/snmp_counter.rst

17 .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
30 .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
41 .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
60 .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
73 .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
81 .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
98 .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
111 .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
118 .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
168 .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169 .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170 .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172 .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173 .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174 .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175 .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176 .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177 .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178 .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
180 .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181 .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182 .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184 .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185 .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186 .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187 .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188 .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189 .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190 .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
204 .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
221 .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222 .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
230 ---------------------------------
255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
284 .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
286 It means the TCP layer sends a SYN, and come into the SYN-SENT
294 .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
297 the SYN-RCVD state.
320   retransmission but including data-in-SYN). This counter is different from
330   retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
344 When kernel receives a SYN from a client, and if the TCP accept queue
356 queue is full. On the old kernel, TCP stack won't drop the SYN, it
357 would complete the 3-way handshake. As the accept queue is full, TCP
358 stack will keep the socket in the TCP half-open queue. As it is in the
359 half open queue, TCP stack will send SYN+ACK on an exponential backoff
361 queue is still full, if it is not full, moves the socket to the accept
362 queue, if it is full, keeps the socket in the half-open queue, at next
364 to the accept queue.
373 .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
379 .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
388 .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
396 queue.
410 The spurious retransmission timeout detected by the `F-RTO`_
413 .. _F-RTO: https://tools.ietf.org/html/rfc5682
424   - A zero window was announced from us
425   - zero window probing
427   - Out of order segments arrived.
428   - Urgent data is expected.
429   - There is no buffer space left
430   - Unexpected TCP flags/window values/header lengths are received
432   - Data is sent in both directions. The fast path only supports pure senders
435   - Unexpected TCP option.
472 .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
475 will return immediately and kernel will try to send the in-flight data
478 wait for the in-flight data are acked by the other side, the max wait
506 .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
530 .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
622 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
718 likely to re-transmit any packets, and we still receive an invalid
728 to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
753 to queue it.
774 .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
779 Packets are dropped by PAWS in Syn-Sent status.
783 Packets are dropped by PAWS in any status other than Syn-Sent.
795 .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
799 The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
802 in the Syn-Recv status. But in several scenarios, the TCP stack need
814 numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
815 or Time-Wait statuses, the skipped ACK would be counted to
823 check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
827 The ACK is skipped in Fin-Wait-2 status, the reason would be either
832 The ACK is skipped in Time-Wait status, the reason would be either
844 .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
845 .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
846 .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
853 window to zero. But the receive window might still be a no-zero
860 The TCP receive window is set to zero from a no-zero value.
864 The TCP receive window is set to no-zero value from zero.
901 .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
914 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
921 When the TCP stack receives an ACK packet in the SYN-SENT status, and
932 after the 3-way handshake, the retransmission timeout happens
933 net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
950 fastopenq->max_qlen, the TCP stack will reject the fast open request
953 TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
1004 reclaim memory from the receiving queue and out of order queue. One of
1012 counter, the TCP stack will try to collapse the out of order queue and
1013 the receiving queue. If the memory is still not enough, the TCP stack
1014 will try to discard packets from the out of order queue (and update the
1019 The TCP stack tries to discard packet on the out of order queue.
1023 After 'collapse' and discard packets from the out of order queue, if
1035 ---------
1038   nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1042   --- 8.8.8.8 ping statistics ---
1048   nstatuser@nstat-a:~$ nstat
1078 tcp 3-way handshake
1079 -------------------
1082   nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1087   nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1091 completed the 3-way handshake.
1095   nstatuser@nstat-b:~$ nstat | grep -i tcp
1103   nstatuser@nstat-a:~$ nstat | grep -i tcp
1109 SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1112 of the 3-way handshake is a pure ACK without data, so
1115 When the client sent SYN, the client came into the SYN-SENT state, so
1121 ------------------
1124   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1129   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1130   Connection to nstat-b 9000 port [tcp/*] succeeded!
1134   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1135   Connection to nstat-b 9000 port [tcp/*] succeeded!
1140   nstatuser@nstat-a:~$ nstat
1155   nstatuser@nstat-b:~$ nstat
1168   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1169   Connection to nstat-b 9000 port [tcp/*] succeeded!
1175   nstatuser@nstat-a:~$ nstat
1191   nstatuser@nstat-b:~$ nstat
1203 Compare the first client-side nstat and the second client-side nstat,
1205 but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1206 nstat and the second server-side nstat had a difference too: the
1207 second server-side nstat had a TcpExtTCPHPHits, but the first
1208 server-side nstat didn't have it. The network traffic patterns were
1219   nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1220   Netid    Recv-Q     Send-Q            Local Address:Port             Peer Address:Port
1244 ---------------------
1264   nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1268 read it yet. We type Ctrl-C to terminate the server script. Then we
1271   nstatuser@nstat-b:~$ nstat | grep -i abort
1275 RST after we type Ctrl-C.
1278 ---------------------------------------------------
1283   sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1287   nstatuser@nstat-a:~$ cat client_orphan.py
1291   server = 'nstat-b' # server address
1309   nstatuser@nstat-b:~$ cat server_orphan.py
1337   sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1339 Type Ctrl-C on client, stop client_orphan.py.
1343   nstatuser@nstat-a:~$ nstat | grep -i abort
1348   nstatuser@nstat-a:~$ ss -s
1353   *         0         -         -
1363 the client, type Ctrl-C on client_orphan.py, the system of the client
1372 the 'ss -s' command shows the system has 10 orphan sockets, and the
1376 exactly orphan socket count by the 'ss -s' command, but when kernel
1396   nstatuser@nstat-a:~$ nstat | grep -i abort
1400 ----------------------
1403   nstatuser@nstat-b:~$ cat server_linger.py
1418   nstatuser@nstat-a:~$ cat client_linger.py
1422   server = 'nstat-b' # server address
1427   s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1433   nstatuser@nstat-b:~$ python3 server_linger.py
1437   nstatuser@nstat-a:~$ python3 client_linger.py
1441   nstatuser@nstat-a:~$ nstat | grep -i abort
1445 --------------------
1466   server = 'nstat-b'
1473   nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1475 We use '-i' to come into the interactive mode, then a packet::
1477   >>> s.send(b'foo')
1487   ubuntu@nstat-b:~$ nstat
1501 the receiving queue. So the TCP layer merged the two packets, and we
1505 -------------------------------------------
1508   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1513   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1514   Connection to nstat-b 9000 port [tcp/*] succeeded!
1516 The nc command only accepts 1 connection, and the accept queue length
1517 is 1. On current linux implementation, set queue length to n means the
1518 actual queue length is n+1. Now we create 3 connections, 1 is accepted
1519 by nc, 2 in accepted queue, so the accept queue is full.
1523   nstatuser@nstat-b:~$ nstat -n
1527   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1531 will drop the SYN if the accept queue is full. If the nc client is running
1534 on half open queue. I did the test on kernel 4.15. Below is the nstat
1537   nstatuser@nstat-b:~$ nstat
1553 -------------------------------------------------
1562   $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1563   $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1564   $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1565   $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1574   $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1578   $ nc -v 8.8.8.8 53
1592 re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1598   $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1609   $ nc -v 8.8.8.8 53
1634   $ ping -c 1 8.8.8.8
1648 --------------------------
1650 first SYN will let server create a socket, set it to Syn-Recv status,
1659   nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1660   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1664   nstatuser@nstat-a:~$ nc nstat-b 9000
1666 As the nstat-b didn't listen on port 9000, it should reply a RST, and
1672   nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1674 On nstat-b, we run nc to listen on port 9000::
1676   nstatuser@nstat-b:~$ nc -lkv 9000
1679 On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1680 RST to nstat-b::
1682   nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1684 Send 3 SYN repeatedly to nstat-b::
1686   nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1688 Check snmp counter on nstat-b::
1690   nstatuser@nstat-b:~$ nstat | grep -i skip
1696 -----------------------
1699 On nstat-b, let nc listen on port 9000::
1701   nstatuser@nstat-b:~$ nc -lkv 9000
1704 On nstat-a, run tcpdump to capture a SYN::
1706   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1707   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1709 On nstat-a, run nc as a client to connect nstat-b::
1711   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1712   Connection to nstat-b 9000 port [tcp/*] succeeded!
1717   nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1721   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1723 On nstat-b, check the snmp counter::
1725   nstatuser@nstat-b:~$ nstat | grep -i skip
1729 failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1733 ----------------------
1743 On nstat-b, open two terminals, run two nc commands to listen on both
1746   nstatuser@nstat-b:~$ nc -lkv 9000
1749   nstatuser@nstat-b:~$ nc -lkv 9001
1752 On nstat-a, run two nc clients::
1754   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1755   Connection to nstat-b 9000 port [tcp/*] succeeded!
1757   nstatuser@nstat-a:~$ nc -v nstat-b 9001
1758   Connection to nstat-b 9001 port [tcp/*] succeeded!
1760 On nstat-a, run tcpdump to capture an ACK::
1762   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1763   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1765 On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1766 string 'foo' in our example::
1768   nstatuser@nstat-b:~$ nc -lkv 9001
1770   Connection from nstat-a 42132 received!
1771   foo
1773 On nstat-a, the tcpdump should have captured the ACK. We should check
1776   nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1777   State  Recv-Q   Send-Q         Local Address:Port           Peer Address:Port
1784 …nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r…
1786 Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1788   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1790 Check TcpExtTCPACKSkippedSeq on nstat-b::
1792   nstatuser@nstat-b:~$ nstat | grep -i skip