Documentation/networking/snmp_counter.rst

17 .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
20 beginning of ip_rcv function, always be updated together with
30 .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
41 .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
43 The number of packets sent via IP layer, for both single cast and
44 multicast packets, and would always be updated together with
58 `Explicit Congestion Notification`_ for more details.
60 .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
64 for the same packet, you might find that IpInReceives count 1, but
73 .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
81 .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
86 packet and can't find a route for it from the route table. It might
88 not a local address and there is no route for the destination IP
95 raw socket, kernel will always deliver the packet to the raw socket
98 .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
102 For IPv4 packet, it means the actual data size is smaller than the
111 .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
118 .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
123 dropped in the IP sending path and no route is found for it.
125 .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
133 .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134 .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
168 .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169 .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170 .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171 .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172 .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173 .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174 .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175 .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176 .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177 .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178 .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
180 .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181 .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182 .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183 .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184 .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185 .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186 .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187 .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188 .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189 .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190 .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
192 Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
204 .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
206 For example, if the Linux kernel sends an ICMP Echo packet, the
221 .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222 .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
227 is increased, IcmpInErrors would always be increased too.
230 ---------------------------------
231 The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
255 .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
272 .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
275 it excludes the retransmitted packets. But it includes the SYN, ACK
284 .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
286 It means the TCP layer sends a SYN, and come into the SYN-SENT
287 state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
294 .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
296 It means the TCP layer receives a SYN, replies a SYN+ACK, come into
297 the SYN-RCVD state.
310 a bigger one. This counter increase 1 for every packet merged in such
311 situation. Please refer to the LWN article for more details:
320   retransmission but including data-in-SYN). This counter is different from
329   TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
330   retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
348 kernel would always add 1 to TcpExtListenDrops. So increase
357 would complete the 3-way handshake. As the accept queue is full, TCP
358 stack will keep the socket in the TCP half-open queue. As it is in the
359 half open queue, TCP stack will send SYN+ACK on an exponential backoff
360 timer, after client replies ACK, TCP stack checks whether the accept
362 queue, if it is full, keeps the socket in the half-open queue, at next
363 time client replies ACK, this socket will get another chance to move
373 .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
379 .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
388 .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
410 The spurious retransmission timeout detected by the `F-RTO`_
413 .. _F-RTO: https://tools.ietf.org/html/rfc5682
424   - A zero window was announced from us
425   - zero window probing
427   - Out of order segments arrived.
428   - Urgent data is expected.
429   - There is no buffer space left
430   - Unexpected TCP flags/window values/header lengths are received
432   - Data is sent in both directions. The fast path only supports pure senders
433     or pure receivers (this means either the sequence number or the ack
435   - Unexpected TCP option.
440 good. Kernel would also come into slow path if the "Delayed ack" is
441 used, because when using "Delayed ack", the data is sent in both
450 If a packet set ACK flag and has no data, it is a pure ACK packet, if
457 If a TCP packet has data (which means it is not a pure ACK packet),
472 .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
475 will return immediately and kernel will try to send the in-flight data
478 wait for the in-flight data are acked by the other side, the max wait
496 becomes an orphan socket, kernel waits for the reply of the other side,
506 .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
519 for the fin packet from the other side, kernel could send a RST and
530 .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
537 approached. The two pieces of information are ACK train length and
538 increase in packet delay. For detail information, please refer the
539 `Hybrid Slow Start paper`_. Either ACK train length or packet delay
550 How many times the ACK train length threshold is detected
554 The sum of CWND detected by ACK train length. Dividing this value by
556 ACK train length.
580 Recovery and Loss. For details about these states, please refer page 5
602 the RTO expires for this packet, then the sender assumes this packet
609 the duplicate ACK number. E.g., if retransmission is triggered, and
611 order, the receiver would acknowledge multiple times, one for the
612 retransmitted packet, another for the arriving of the original out of
620 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
622 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
623 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
633 the sender has received SACKs for packet 2 and 5, now the sender
634 receives SACK for packet 4 and the sender doesn't retransmit the
636 stack of kernel will increase TcpExtTCPSACKReorder for both of the
697 number of the SACK block. For more details, please refer the comment
702 `Add counters for discarded SACK blocks`_ patch has additional
705 .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvald…
710 SACK block is caused by ACK recording, the TCP stack will only ignore
718 likely to re-transmit any packets, and we still receive an invalid
726 The linux networking stack stores data in sk_buff struct (skb for
728 to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
745 A skb should be shifted or merged, but the TCP stack doesn't do it for
771 timestamps. For detail information, please refer the `timestamp wiki`_
774 .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
779 Packets are dropped by PAWS in Syn-Sent status.
783 Packets are dropped by PAWS in any status other than Syn-Sent.
785 TCP ACK skip
789 section of the `sysctl document`_. When kernel decides to skip an ACK
791 counters to indicate the ACK is skipped in which scenario. The ACK
795 .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
799 The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
800 TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
801 waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
802 in the Syn-Recv status. But in several scenarios, the TCP stack need
803 to send an ACK. E.g., the TCP stack receives the same SYN packet
806 the TCP stack needs to send ACK. If the ACk sending frequency is higher than
807 tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
813 The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
814 numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
815 or Time-Wait statuses, the skipped ACK would be counted to
817 TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
823 check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
827 The ACK is skipped in Fin-Wait-2 status, the reason would be either
832 The ACK is skipped in Time-Wait status, the reason would be either
837 The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
838 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
841 send challenge ACKs if the ACK number is before the first
844 .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
845 .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
846 .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
853 window to zero. But the receive window might still be a no-zero
854 value. For example, if the previous window size is 10, and the TCP
860 The TCP receive window is set to zero from a no-zero value.
864 The TCP receive window is set to no-zero value from zero.
867 Delayed ACK
869 The TCP Delayed ACK is a technique which is used for reducing the
870 packet count in the network. For more details, please refer the
871 `Delayed ACK wiki`_
873 .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
877 A delayed ACK timer expires. The TCP stack will send a pure ACK packet
878 and exit the delayed ACK mode.
882 A delayed ACK timer expires, but the TCP stack can't send an ACK
884 TCP stack will send a pure ACK later (after the userspace program
885 unlock the socket). When the TCP stack sends the pure ACK later, the
886 TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
892 ACKed. A Delayed ACK loss might cause this issue, but it would also be
898 TLP is an algorithm which is used to detect TCP packet loss. For more
901 .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
914 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
921 When the TCP stack receives an ACK packet in the SYN-SENT status, and
922 the ACK packet acknowledges the data in the SYN packet, the TCP stack
932 after the 3-way handshake, the retransmission timeout happens
933 net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
950 fastopenq->max_qlen, the TCP stack will reject the fast open request
953 TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
955 net.core.somaxconn. For example:
966 SYN cookies are used to mitigate SYN flood, for details, please refer
985 Challenge ACK
987 For details of challenge ACK, please refer the explanation of
997 updates this counter, the TCP stack might send a challenge ACK and
1011 The TCP stack tries to reclaim memory for a socket. After updates this
1035 ---------
1038   nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1042   --- 8.8.8.8 ping statistics ---
1048   nstatuser@nstat-a:~$ nstat
1078 tcp 3-way handshake
1079 -------------------
1082   nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1087   nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1091 completed the 3-way handshake.
1095   nstatuser@nstat-b:~$ nstat | grep -i tcp
1103   nstatuser@nstat-a:~$ nstat | grep -i tcp
1108 When the server received the first SYN, it replied a SYN+ACK, and came into
1109 SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1110 SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
1111 packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
1112 of the 3-way handshake is a pure ACK without data, so
1115 When the client sent SYN, the client came into the SYN-SENT state, so
1116 TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
1117 ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
1121 ------------------
1124   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1129   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1130   Connection to nstat-b 9000 port [tcp/*] succeeded!
1134   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1135   Connection to nstat-b 9000 port [tcp/*] succeeded!
1140   nstatuser@nstat-a:~$ nstat
1155   nstatuser@nstat-b:~$ nstat
1168   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1169   Connection to nstat-b 9000 port [tcp/*] succeeded!
1175   nstatuser@nstat-a:~$ nstat
1191   nstatuser@nstat-b:~$ nstat
1203 Compare the first client-side nstat and the second client-side nstat,
1205 but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1206 nstat and the second server-side nstat had a difference too: the
1207 second server-side nstat had a TcpExtTCPHPHits, but the first
1208 server-side nstat didn't have it. The network traffic patterns were
1210 replied an ACK. But kernel handled them in different ways. When the
1219   nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1220   Netid    Recv-Q     Send-Q            Local Address:Port             Peer Address:Port
1228 reply an ACK, when kernel handled this ACK, the fast path was not
1229 enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
1232 and received another ACK from the server, in this time, the fast path is
1233 enabled, and the ACK was qualified for fast path, so it was handled by
1234 the fast path, so this ACK was counted into TcpExtTCPHPAcks.
1240 and the packet received from client qualified for fast path, so it
1244 ---------------------
1264   nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1268 read it yet. We type Ctrl-C to terminate the server script. Then we
1271   nstatuser@nstat-b:~$ nstat | grep -i abort
1275 RST after we type Ctrl-C.
1278 ---------------------------------------------------
1283   sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1287   nstatuser@nstat-a:~$ cat client_orphan.py
1291   server = 'nstat-b' # server address
1298   for i in range(64):
1309   nstatuser@nstat-b:~$ cat server_orphan.py
1337   sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1339 Type Ctrl-C on client, stop client_orphan.py.
1343   nstatuser@nstat-a:~$ nstat | grep -i abort
1348   nstatuser@nstat-a:~$ ss -s
1353   *         0         -         -
1363 the client, type Ctrl-C on client_orphan.py, the system of the client
1370 only keep 10 orphan sockets, for all other orphan sockets, the client
1371 system sent RST for them and delete them. We have 64 connections, so
1372 the 'ss -s' command shows the system has 10 orphan sockets, and the
1376 exactly orphan socket count by the 'ss -s' command, but when kernel
1378 doesn't always check the exactly orphan socket count. For increasing
1390 Continue the previous test, we wait for several minutes. Because of the
1393 FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
1396   nstatuser@nstat-a:~$ nstat | grep -i abort
1400 ----------------------
1403   nstatuser@nstat-b:~$ cat server_linger.py
1418   nstatuser@nstat-a:~$ cat client_linger.py
1422   server = 'nstat-b' # server address
1427   s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1433   nstatuser@nstat-b:~$ python3 server_linger.py
1437   nstatuser@nstat-a:~$ python3 client_linger.py
1441   nstatuser@nstat-a:~$ nstat | grep -i abort
1445 --------------------
1466   server = 'nstat-b'
1473   nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1475 We use '-i' to come into the interactive mode, then a packet::
1487   ubuntu@nstat-b:~$ nstat
1505 -------------------------------------------
1508   nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1513   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1514   Connection to nstat-b 9000 port [tcp/*] succeeded!
1523   nstatuser@nstat-b:~$ nstat -n
1527   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1537   nstatuser@nstat-b:~$ nstat
1553 -------------------------------------------------
1560 Prepare on server B, disable send_redirects for all interfaces::
1562   $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1563   $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1564   $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1565   $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1574   $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1578   $ nc -v 8.8.8.8 53
1592 re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1598   $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1609   $ nc -v 8.8.8.8 53
1628 this packet. We have deleted the default route, there was no route for
1634   $ ping -c 1 8.8.8.8
1644 a route for the 8.8.8.8 IP address, so server B increased
1648 --------------------------
1650 first SYN will let server create a socket, set it to Syn-Recv status,
1651 and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
1652 again, and record the reply time (the duplicate ACK reply time). The
1653 third SYN will let server check the previous duplicate ACK reply time,
1654 and decide to skip the duplicate ACK, then increase the
1659   nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1660   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1664   nstatuser@nstat-a:~$ nc nstat-b 9000
1666 As the nstat-b didn't listen on port 9000, it should reply a RST, and
1667 the nc command exited immediately. It was enough for the tcpdump
1669 offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
1672   nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1674 On nstat-b, we run nc to listen on port 9000::
1676   nstatuser@nstat-b:~$ nc -lkv 9000
1679 On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1680 RST to nstat-b::
1682   nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1684 Send 3 SYN repeatedly to nstat-b::
1686   nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1688 Check snmp counter on nstat-b::
1690   nstatuser@nstat-b:~$ nstat | grep -i skip
1696 -----------------------
1699 On nstat-b, let nc listen on port 9000::
1701   nstatuser@nstat-b:~$ nc -lkv 9000
1704 On nstat-a, run tcpdump to capture a SYN::
1706   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1707   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1709 On nstat-a, run nc as a client to connect nstat-b::
1711   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1712   Connection to nstat-b 9000 port [tcp/*] succeeded!
1717   nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1721   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1723 On nstat-b, check the snmp counter::
1725   nstatuser@nstat-b:~$ nstat | grep -i skip
1729 failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1730 for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
1733 ----------------------
1737 data, so we need a pure ACK packet. To generate such a packet, we
1739 we capture an ACK on port 9001, change the source/destination port
1743 On nstat-b, open two terminals, run two nc commands to listen on both
1746   nstatuser@nstat-b:~$ nc -lkv 9000
1749   nstatuser@nstat-b:~$ nc -lkv 9001
1752 On nstat-a, run two nc clients::
1754   nstatuser@nstat-a:~$ nc -v nstat-b 9000
1755   Connection to nstat-b 9000 port [tcp/*] succeeded!
1757   nstatuser@nstat-a:~$ nc -v nstat-b 9001
1758   Connection to nstat-b 9001 port [tcp/*] succeeded!
1760 On nstat-a, run tcpdump to capture an ACK::
1762   nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1763   tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1765 On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1768   nstatuser@nstat-b:~$ nc -lkv 9001
1770   Connection from nstat-a 42132 received!
1773 On nstat-a, the tcpdump should have captured the ACK. We should check
1776   nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1777   State  Recv-Q   Send-Q         Local Address:Port           Peer Address:Port
1784 …nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r…
1786 Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1788   nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1790 Check TcpExtTCPACKSkippedSeq on nstat-b::
1792   nstatuser@nstat-b:~$ nstat | grep -i skip