1============
2SNMP counter
3============
4
5This document explains the meaning of SNMP counters.
6
7General IPv4 counters
8=====================
9All layer 4 packets and ICMP packets will change these counters, but
10these counters won't be changed by layer 2 packets (such as STP) or
11ARP packets.
12
13* IpInReceives
14
15Defined in `RFC1213 ipInReceives`_
16
17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
18
19The number of packets received by the IP layer. It gets increasing at the
20beginning of ip_rcv function, always be updated together with
21IpExtInOctets. It will be increased even if the packet is dropped
22later (e.g. due to the IP header is invalid or the checksum is wrong
23and so on).  It indicates the number of aggregated segments after
24GRO/LRO.
25
26* IpInDelivers
27
28Defined in `RFC1213 ipInDelivers`_
29
30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
31
32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
33ICMP and so on. If no one listens on a raw socket, only kernel
34supported protocols will be delivered, if someone listens on the raw
35socket, all valid IP packets will be delivered.
36
37* IpOutRequests
38
39Defined in `RFC1213 ipOutRequests`_
40
41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
42
43The number of packets sent via IP layer, for both single cast and
44multicast packets, and would always be updated together with
45IpExtOutOctets.
46
47* IpExtInOctets and IpExtOutOctets
48
49They are Linux kernel extensions, no RFC definitions. Please note,
50RFC1213 indeed defines ifInOctets  and ifOutOctets, but they
51are different things. The ifInOctets and ifOutOctets include the MAC
52layer header size but IpExtInOctets and IpExtOutOctets don't, they
53only include the IP layer header and the IP layer data.
54
55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
56
57They indicate the number of four kinds of ECN IP packets, please refer
58`Explicit Congestion Notification`_ for more details.
59
60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
61
62These 4 counters calculate how many packets received per ECN
63status. They count the real frame number regardless the LRO/GRO. So
64for the same packet, you might find that IpInReceives count 1, but
65IpExtInNoECTPkts counts 2 or more.
66
67* IpInHdrErrors
68
69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
70dropped due to the IP header error. It might happen in both IP input
71and IP forward paths.
72
73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
74
75* IpInAddrErrors
76
77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
78scenarios: (1) The IP address is invalid. (2) The destination IP
79address is not a local address and IP forwarding is not enabled
80
81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
82
83* IpExtInNoRoutes
84
85This counter means the packet is dropped when the IP stack receives a
86packet and can't find a route for it from the route table. It might
87happen when IP forwarding is enabled and the destination IP address is
88not a local address and there is no route for the destination IP
89address.
90
91* IpInUnknownProtos
92
93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
94layer 4 protocol is unsupported by kernel. If an application is using
95raw socket, kernel will always deliver the packet to the raw socket
96and this counter won't be increased.
97
98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
99
100* IpExtInTruncatedPkts
101
102For IPv4 packet, it means the actual data size is smaller than the
103"Total Length" field in the IPv4 header.
104
105* IpInDiscards
106
107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
108in the IP receiving path and due to kernel internal reasons (e.g. no
109enough memory).
110
111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
112
113* IpOutDiscards
114
115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
116dropped in the IP sending path and due to kernel internal reasons.
117
118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
119
120* IpOutNoRoutes
121
122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
123dropped in the IP sending path and no route is found for it.
124
125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
126
127ICMP counters
128=============
129* IcmpInMsgs and IcmpOutMsgs
130
131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
132
133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
135
136As mentioned in the RFC1213, these two counters include errors, they
137would be increased even if the ICMP packet has an invalid type. The
138ICMP output path will check the header of a raw socket, so the
139IcmpOutMsgs would still be updated if the IP header is constructed by
140a userspace program.
141
142* ICMP named types
143
144| These counters include most of common ICMP types, they are:
145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
149| IcmpInRedirects: `RFC1213 icmpInRedirects`_
150| IcmpInEchos: `RFC1213 icmpInEchos`_
151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
161| IcmpOutEchos: `RFC1213 icmpOutEchos`_
162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
167
168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
179
180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
191
192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
194straightforward. The 'In' counter means kernel receives such a packet
195and the 'Out' counter means kernel sends such a packet.
196
197* ICMP numeric types
198
199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
200ICMP type number. These counters track all kinds of ICMP packets. The
201ICMP type number definition could be found in the `ICMP parameters`_
202document.
203
204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
205
206For example, if the Linux kernel sends an ICMP Echo packet, the
207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
208packet, IcmpMsgInType0 would increase 1.
209
210* IcmpInCsumErrors
211
212This counter indicates the checksum of the ICMP packet is
213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
214before updating IcmpMsgInType[N]. If a packet has bad checksum, the
215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
216
217* IcmpInErrors and IcmpOutErrors
218
219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
220
221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
223
224When an error occurs in the ICMP packet handler path, these two
225counters would be updated. The receiving packet path use IcmpInErrors
226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
227is increased, IcmpInErrors would always be increased too.
228
229relationship of the ICMP counters
230---------------------------------
231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
232are updated at the same time. The sum of IcmpMsgInType[N] plus
233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
234receives an ICMP packet, kernel follows below logic:
235
2361. increase IcmpInMsgs
2372. if has any error, update IcmpInErrors and finish the process
2383. update IcmpMsgOutType[N]
2394. handle the packet depending on the type, if has any error, update
240   IcmpInErrors and finish the process
241
242So if all errors occur in step (2), IcmpInMsgs should be equal to the
243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
244step (4), IcmpInMsgs should be equal to the sum of
245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
247IcmpInErrors.
248
249General TCP counters
250====================
251* TcpInSegs
252
253Defined in `RFC1213 tcpInSegs`_
254
255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
256
257The number of packets received by the TCP layer. As mentioned in
258RFC1213, it includes the packets received in error, such as checksum
259error, invalid TCP header and so on. Only one error won't be included:
260if the layer 2 destination address is not the NIC's layer 2
261address. It might happen if the packet is a multicast or broadcast
262packet, or the NIC is in promiscuous mode. In these situations, the
263packets would be delivered to the TCP layer, but the TCP layer will discard
264these packets before increasing TcpInSegs. The TcpInSegs counter
265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
266counter would only increase 1.
267
268* TcpOutSegs
269
270Defined in `RFC1213 tcpOutSegs`_
271
272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
273
274The number of packets sent by the TCP layer. As mentioned in RFC1213,
275it excludes the retransmitted packets. But it includes the SYN, ACK
276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
278increase 2.
279
280* TcpActiveOpens
281
282Defined in `RFC1213 tcpActiveOpens`_
283
284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
285
286It means the TCP layer sends a SYN, and come into the SYN-SENT
287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
288increase 1.
289
290* TcpPassiveOpens
291
292Defined in `RFC1213 tcpPassiveOpens`_
293
294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
295
296It means the TCP layer receives a SYN, replies a SYN+ACK, come into
297the SYN-RCVD state.
298
299* TcpExtTCPRcvCoalesce
300
301When packets are received by the TCP layer and are not be read by the
302application, the TCP layer will try to merge them. This counter
303indicate how many packets are merged in such situation. If GRO is
304enabled, lots of packets would be merged by GRO, these packets
305wouldn't be counted to TcpExtTCPRcvCoalesce.
306
307* TcpExtTCPAutoCorking
308
309When sending packets, the TCP layer will try to merge small packets to
310a bigger one. This counter increase 1 for every packet merged in such
311situation. Please refer to the LWN article for more details:
312https://lwn.net/Articles/576263/
313
314* TcpExtTCPOrigDataSent
315
316This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
317explaination below::
318
319  TCPOrigDataSent: number of outgoing packets with original data (excluding
320  retransmission but including data-in-SYN). This counter is different from
321  TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
322  more useful to track the TCP retransmission rate.
323
324* TCPSynRetrans
325
326This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
327explaination below::
328
329  TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
330  retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
331
332* TCPFastOpenActiveFail
333
334This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
335explaination below::
336
337  TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
338  the remote does not accept it or the attempts timed out.
339
340.. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
341
342* TcpExtListenOverflows and TcpExtListenDrops
343
344When kernel receives a SYN from a client, and if the TCP accept queue
345is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
346At the same time kernel will also add 1 to TcpExtListenDrops. When a
347TCP socket is in LISTEN state, and kernel need to drop a packet,
348kernel would always add 1 to TcpExtListenDrops. So increase
349TcpExtListenOverflows would let TcpExtListenDrops increasing at the
350same time, but TcpExtListenDrops would also increase without
351TcpExtListenOverflows increasing, e.g. a memory allocation fail would
352also let TcpExtListenDrops increase.
353
354Note: The above explanation is based on kernel 4.10 or above version, on
355an old kernel, the TCP stack has different behavior when TCP accept
356queue is full. On the old kernel, TCP stack won't drop the SYN, it
357would complete the 3-way handshake. As the accept queue is full, TCP
358stack will keep the socket in the TCP half-open queue. As it is in the
359half open queue, TCP stack will send SYN+ACK on an exponential backoff
360timer, after client replies ACK, TCP stack checks whether the accept
361queue is still full, if it is not full, moves the socket to the accept
362queue, if it is full, keeps the socket in the half-open queue, at next
363time client replies ACK, this socket will get another chance to move
364to the accept queue.
365
366
367TCP Fast Open
368=============
369* TcpEstabResets
370
371Defined in `RFC1213 tcpEstabResets`_.
372
373.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
374
375* TcpAttemptFails
376
377Defined in `RFC1213 tcpAttemptFails`_.
378
379.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
380
381* TcpOutRsts
382
383Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
384the 'segments sent containing the RST flag', but in linux kernel, this
385couner indicates the segments kerenl tried to send. The sending
386process might be failed due to some errors (e.g. memory alloc failed).
387
388.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
389
390* TcpExtTCPSpuriousRtxHostQueues
391
392When the TCP stack wants to retransmit a packet, and finds that packet
393is not lost in the network, but the packet is not sent yet, the TCP
394stack would give up the retransmission and update this counter. It
395might happen if a packet stays too long time in a qdisc or driver
396queue.
397
398* TcpEstabResets
399
400The socket receives a RST packet in Establish or CloseWait state.
401
402* TcpExtTCPKeepAlive
403
404This counter indicates many keepalive packets were sent. The keepalive
405won't be enabled by default. A userspace program could enable it by
406setting the SO_KEEPALIVE socket option.
407
408* TcpExtTCPSpuriousRTOs
409
410The spurious retransmission timeout detected by the `F-RTO`_
411algorithm.
412
413.. _F-RTO: https://tools.ietf.org/html/rfc5682
414
415TCP Fast Path
416=============
417When kernel receives a TCP packet, it has two paths to handler the
418packet, one is fast path, another is slow path. The comment in kernel
419code provides a good explanation of them, I pasted them below::
420
421  It is split into a fast path and a slow path. The fast path is
422  disabled when:
423
424  - A zero window was announced from us
425  - zero window probing
426    is only handled properly on the slow path.
427  - Out of order segments arrived.
428  - Urgent data is expected.
429  - There is no buffer space left
430  - Unexpected TCP flags/window values/header lengths are received
431    (detected by checking the TCP header against pred_flags)
432  - Data is sent in both directions. The fast path only supports pure senders
433    or pure receivers (this means either the sequence number or the ack
434    value must stay constant)
435  - Unexpected TCP option.
436
437Kernel will try to use fast path unless any of the above conditions
438are satisfied. If the packets are out of order, kernel will handle
439them in slow path, which means the performance might be not very
440good. Kernel would also come into slow path if the "Delayed ack" is
441used, because when using "Delayed ack", the data is sent in both
442directions. When the TCP window scale option is not used, kernel will
443try to enable fast path immediately when the connection comes into the
444established state, but if the TCP window scale option is used, kernel
445will disable the fast path at first, and try to enable it after kernel
446receives packets.
447
448* TcpExtTCPPureAcks and TcpExtTCPHPAcks
449
450If a packet set ACK flag and has no data, it is a pure ACK packet, if
451kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
452if kernel handles it in the slow path, TcpExtTCPPureAcks will
453increase 1.
454
455* TcpExtTCPHPHits
456
457If a TCP packet has data (which means it is not a pure ACK packet),
458and this packet is handled in the fast path, TcpExtTCPHPHits will
459increase 1.
460
461
462TCP abort
463=========
464* TcpExtTCPAbortOnData
465
466It means TCP layer has data in flight, but need to close the
467connection. So TCP layer sends a RST to the other side, indicate the
468connection is not closed very graceful. An easy way to increase this
469counter is using the SO_LINGER option. Please refer to the SO_LINGER
470section of the `socket man page`_:
471
472.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
473
474By default, when an application closes a connection, the close function
475will return immediately and kernel will try to send the in-flight data
476async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
477to a positive number, the close function won't return immediately, but
478wait for the in-flight data are acked by the other side, the max wait
479time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
480when the application closes a connection, kernel will send a RST
481immediately and increase the TcpExtTCPAbortOnData counter.
482
483* TcpExtTCPAbortOnClose
484
485This counter means the application has unread data in the TCP layer when
486the application wants to close the TCP connection. In such a situation,
487kernel will send a RST to the other side of the TCP connection.
488
489* TcpExtTCPAbortOnMemory
490
491When an application closes a TCP connection, kernel still need to track
492the connection, let it complete the TCP disconnect process. E.g. an
493app calls the close method of a socket, kernel sends fin to the other
494side of the connection, then the app has no relationship with the
495socket any more, but kernel need to keep the socket, this socket
496becomes an orphan socket, kernel waits for the reply of the other side,
497and would come to the TIME_WAIT state finally. When kernel has no
498enough memory to keep the orphan socket, kernel would send an RST to
499the other side, and delete the socket, in such situation, kernel will
500increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
501TcpExtTCPAbortOnMemory:
502
5031. the memory used by the TCP protocol is higher than the third value of
504the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
505
506.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
507
5082. the orphan socket count is higher than net.ipv4.tcp_max_orphans
509
510
511* TcpExtTCPAbortOnTimeout
512
513This counter will increase when any of the TCP timers expire. In such
514situation, kernel won't send RST, just give up the connection.
515
516* TcpExtTCPAbortOnLinger
517
518When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
519for the fin packet from the other side, kernel could send a RST and
520delete the socket immediately. This is not the default behavior of
521Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
522you could let kernel follow this behavior.
523
524* TcpExtTCPAbortFailed
525
526The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
527satisfied. If an internal error occurs during this process,
528TcpExtTCPAbortFailed will be increased.
529
530.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
531
532TCP Hybrid Slow Start
533=====================
534The Hybrid Slow Start algorithm is an enhancement of the traditional
535TCP congestion window Slow Start algorithm. It uses two pieces of
536information to detect whether the max bandwidth of the TCP path is
537approached. The two pieces of information are ACK train length and
538increase in packet delay. For detail information, please refer the
539`Hybrid Slow Start paper`_. Either ACK train length or packet delay
540hits a specific threshold, the congestion control algorithm will come
541into the Congestion Avoidance state. Until v4.20, two congestion
542control algorithms are using Hybrid Slow Start, they are cubic (the
543default congestion control algorithm) and cdg. Four snmp counters
544relate with the Hybrid Slow Start algorithm.
545
546.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
547
548* TcpExtTCPHystartTrainDetect
549
550How many times the ACK train length threshold is detected
551
552* TcpExtTCPHystartTrainCwnd
553
554The sum of CWND detected by ACK train length. Dividing this value by
555TcpExtTCPHystartTrainDetect is the average CWND which detected by the
556ACK train length.
557
558* TcpExtTCPHystartDelayDetect
559
560How many times the packet delay threshold is detected.
561
562* TcpExtTCPHystartDelayCwnd
563
564The sum of CWND detected by packet delay. Dividing this value by
565TcpExtTCPHystartDelayDetect is the average CWND which detected by the
566packet delay.
567
568TCP retransmission and congestion control
569=========================================
570The TCP protocol has two retransmission mechanisms: SACK and fast
571recovery. They are exclusive with each other. When SACK is enabled,
572the kernel TCP stack would use SACK, or kernel would use fast
573recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
574the fast recovery is defined in `RFC6582`_, which is also called
575'Reno'.
576
577The TCP congestion control is a big and complex topic. To understand
578the related snmp counter, we need to know the states of the congestion
579control state machine. There are 5 states: Open, Disorder, CWR,
580Recovery and Loss. For details about these states, please refer page 5
581and page 6 of this document:
582https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
583
584.. _RFC2018: https://tools.ietf.org/html/rfc2018
585.. _RFC6582: https://tools.ietf.org/html/rfc6582
586
587* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
588
589When the congestion control comes into Recovery state, if sack is
590used, TcpExtTCPSackRecovery increases 1, if sack is not used,
591TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
592stack begins to retransmit the lost packets.
593
594* TcpExtTCPSACKReneging
595
596A packet was acknowledged by SACK, but the receiver has dropped this
597packet, so the sender needs to retransmit this packet. In this
598situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
599could drop a packet which has been acknowledged by SACK, although it is
600unusual, it is allowed by the TCP protocol. The sender doesn't really
601know what happened on the receiver side. The sender just waits until
602the RTO expires for this packet, then the sender assumes this packet
603has been dropped by the receiver.
604
605* TcpExtTCPRenoReorder
606
607The reorder packet is detected by fast recovery. It would only be used
608if SACK is disabled. The fast recovery algorithm detects recorder by
609the duplicate ACK number. E.g., if retransmission is triggered, and
610the original retransmitted packet is not lost, it is just out of
611order, the receiver would acknowledge multiple times, one for the
612retransmitted packet, another for the arriving of the original out of
613order packet. Thus the sender would find more ACks than its
614expectation, and the sender knows out of order occurs.
615
616* TcpExtTCPTSReorder
617
618The reorder packet is detected when a hole is filled. E.g., assume the
619sender sends packet 1,2,3,4,5, and the receiving order is
6201,2,4,5,3. When the sender receives the ACK of packet 3 (which will
621fill the hole), two conditions will let TcpExtTCPTSReorder increase
6221: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
6233 is retransmitted but the timestamp of the packet 3's ACK is earlier
624than the retransmission timestamp.
625
626* TcpExtTCPSACKReorder
627
628The reorder packet detected by SACK. The SACK has two methods to
629detect reorder: (1) DSACK is received by the sender. It means the
630sender sends the same packet more than one times. And the only reason
631is the sender believes an out of order packet is lost so it sends the
632packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
633the sender has received SACKs for packet 2 and 5, now the sender
634receives SACK for packet 4 and the sender doesn't retransmit the
635packet yet, the sender would know packet 4 is out of order. The TCP
636stack of kernel will increase TcpExtTCPSACKReorder for both of the
637above scenarios.
638
639* TcpExtTCPSlowStartRetrans
640
641The TCP stack wants to retransmit a packet and the congestion control
642state is 'Loss'.
643
644* TcpExtTCPFastRetrans
645
646The TCP stack wants to retransmit a packet and the congestion control
647state is not 'Loss'.
648
649* TcpExtTCPLostRetransmit
650
651A SACK points out that a retransmission packet is lost again.
652
653* TcpExtTCPRetransFail
654
655The TCP stack tries to deliver a retransmission packet to lower layers
656but the lower layers return an error.
657
658* TcpExtTCPSynRetrans
659
660The TCP stack retransmits a SYN packet.
661
662DSACK
663=====
664The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
665duplicate packets to the sender. There are two kinds of
666duplications: (1) a packet which has been acknowledged is
667duplicate. (2) an out of order packet is duplicate. The TCP stack
668counts these two kinds of duplications on both receiver side and
669sender side.
670
671.. _RFC2883 : https://tools.ietf.org/html/rfc2883
672
673* TcpExtTCPDSACKOldSent
674
675The TCP stack receives a duplicate packet which has been acked, so it
676sends a DSACK to the sender.
677
678* TcpExtTCPDSACKOfoSent
679
680The TCP stack receives an out of order duplicate packet, so it sends a
681DSACK to the sender.
682
683* TcpExtTCPDSACKRecv
684
685The TCP stack receives a DSACK, which indicates an acknowledged
686duplicate packet is received.
687
688* TcpExtTCPDSACKOfoRecv
689
690The TCP stack receives a DSACK, which indicate an out of order
691duplicate packet is received.
692
693invalid SACK and DSACK
694======================
695When a SACK (or DSACK) block is invalid, a corresponding counter would
696be updated. The validation method is base on the start/end sequence
697number of the SACK block. For more details, please refer the comment
698of the function tcp_is_sackblock_valid in the kernel source code. A
699SACK option could have up to 4 blocks, they are checked
700individually. E.g., if 3 blocks of a SACk is invalid, the
701corresponding counter would be updated 3 times. The comment of the
702`Add counters for discarded SACK blocks`_ patch has additional
703explaination:
704
705.. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
706
707* TcpExtTCPSACKDiscard
708
709This counter indicates how many SACK blocks are invalid. If the invalid
710SACK block is caused by ACK recording, the TCP stack will only ignore
711it and won't update this counter.
712
713* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
714
715When a DSACK block is invalid, one of these two counters would be
716updated. Which counter will be updated depends on the undo_marker flag
717of the TCP socket. If the undo_marker is not set, the TCP stack isn't
718likely to re-transmit any packets, and we still receive an invalid
719DSACK block, the reason might be that the packet is duplicated in the
720middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
721will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
722will be updated. As implied in its name, it might be an old packet.
723
724SACK shift
725==========
726The linux networking stack stores data in sk_buff struct (skb for
727short). If a SACK block acrosses multiple skb, the TCP stack will try
728to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
72910 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
73015 in skb2 would be moved to skb1. This operation is 'shift'. If a
731SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
732seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
733discard, this operation is 'merge'.
734
735* TcpExtTCPSackShifted
736
737A skb is shifted
738
739* TcpExtTCPSackMerged
740
741A skb is merged
742
743* TcpExtTCPSackShiftFallback
744
745A skb should be shifted or merged, but the TCP stack doesn't do it for
746some reasons.
747
748TCP out of order
749================
750* TcpExtTCPOFOQueue
751
752The TCP layer receives an out of order packet and has enough memory
753to queue it.
754
755* TcpExtTCPOFODrop
756
757The TCP layer receives an out of order packet but doesn't have enough
758memory, so drops it. Such packets won't be counted into
759TcpExtTCPOFOQueue.
760
761* TcpExtTCPOFOMerge
762
763The received out of order packet has an overlay with the previous
764packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
765packets will also be counted into TcpExtTCPOFOQueue.
766
767TCP PAWS
768========
769PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
770which is used to drop old packets. It depends on the TCP
771timestamps. For detail information, please refer the `timestamp wiki`_
772and the `RFC of PAWS`_.
773
774.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
775.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
776
777* TcpExtPAWSActive
778
779Packets are dropped by PAWS in Syn-Sent status.
780
781* TcpExtPAWSEstab
782
783Packets are dropped by PAWS in any status other than Syn-Sent.
784
785TCP ACK skip
786============
787In some scenarios, kernel would avoid sending duplicate ACKs too
788frequently. Please find more details in the tcp_invalid_ratelimit
789section of the `sysctl document`_. When kernel decides to skip an ACK
790due to tcp_invalid_ratelimit, kernel would update one of below
791counters to indicate the ACK is skipped in which scenario. The ACK
792would only be skipped if the received packet is either a SYN packet or
793it has no data.
794
795.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
796
797* TcpExtTCPACKSkippedSynRecv
798
799The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
800TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
801waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
802in the Syn-Recv status. But in several scenarios, the TCP stack need
803to send an ACK. E.g., the TCP stack receives the same SYN packet
804repeately, the received packet does not pass the PAWS check, or the
805received packet sequence number is out of window. In these scenarios,
806the TCP stack needs to send ACK. If the ACk sending frequency is higher than
807tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
808increase TcpExtTCPACKSkippedSynRecv.
809
810
811* TcpExtTCPACKSkippedPAWS
812
813The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
814numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
815or Time-Wait statuses, the skipped ACK would be counted to
816TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
817TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
818would be counted to TcpExtTCPACKSkippedPAWS.
819
820* TcpExtTCPACKSkippedSeq
821
822The sequence number is out of window and the timestamp passes the PAWS
823check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
824
825* TcpExtTCPACKSkippedFinWait2
826
827The ACK is skipped in Fin-Wait-2 status, the reason would be either
828PAWS check fails or the received sequence number is out of window.
829
830* TcpExtTCPACKSkippedTimeWait
831
832Tha ACK is skipped in Time-Wait status, the reason would be either
833PAWS check failed or the received sequence number is out of window.
834
835* TcpExtTCPACKSkippedChallenge
836
837The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
8383 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
839`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
840three scenarios, In some TCP status, the linux TCP stack would also
841send challenge ACKs if the ACK number is before the first
842unacknowledged number (more strict than `RFC 5961 section 5.2`_).
843
844.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
845.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
846.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
847
848TCP receive window
849==================
850* TcpExtTCPWantZeroWindowAdv
851
852Depending on current memory usage, the TCP stack tries to set receive
853window to zero. But the receive window might still be a no-zero
854value. For example, if the previous window size is 10, and the TCP
855stack receives 3 bytes, the current window size would be 7 even if the
856window size calculated by the memory usage is zero.
857
858* TcpExtTCPToZeroWindowAdv
859
860The TCP receive window is set to zero from a no-zero value.
861
862* TcpExtTCPFromZeroWindowAdv
863
864The TCP receive window is set to no-zero value from zero.
865
866
867Delayed ACK
868===========
869The TCP Delayed ACK is a technique which is used for reducing the
870packet count in the network. For more details, please refer the
871`Delayed ACK wiki`_
872
873.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
874
875* TcpExtDelayedACKs
876
877A delayed ACK timer expires. The TCP stack will send a pure ACK packet
878and exit the delayed ACK mode.
879
880* TcpExtDelayedACKLocked
881
882A delayed ACK timer expires, but the TCP stack can't send an ACK
883immediately due to the socket is locked by a userspace program. The
884TCP stack will send a pure ACK later (after the userspace program
885unlock the socket). When the TCP stack sends the pure ACK later, the
886TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
887mode.
888
889* TcpExtDelayedACKLost
890
891It will be updated when the TCP stack receives a packet which has been
892ACKed. A Delayed ACK loss might cause this issue, but it would also be
893triggered by other reasons, such as a packet is duplicated in the
894network.
895
896Tail Loss Probe (TLP)
897=====================
898TLP is an algorithm which is used to detect TCP packet loss. For more
899details, please refer the `TLP paper`_.
900
901.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
902
903* TcpExtTCPLossProbes
904
905A TLP probe packet is sent.
906
907* TcpExtTCPLossProbeRecovery
908
909A packet loss is detected and recovered by TLP.
910
911TCP Fast Open
912=============
913TCP Fast Open is a technology which allows data transfer before the
9143-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
915general description.
916
917.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
918
919* TcpExtTCPFastOpenActive
920
921When the TCP stack receives an ACK packet in the SYN-SENT status, and
922the ACK packet acknowledges the data in the SYN packet, the TCP stack
923understand the TFO cookie is accepted by the other side, then it
924updates this counter.
925
926* TcpExtTCPFastOpenActiveFail
927
928This counter indicates that the TCP stack initiated a TCP Fast Open,
929but it failed. This counter would be updated in three scenarios: (1)
930the other side doesn't acknowledge the data in the SYN packet. (2) The
931SYN packet which has the TFO cookie is timeout at least once. (3)
932after the 3-way handshake, the retransmission timeout happens
933net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
934fast open after the handshake.
935
936* TcpExtTCPFastOpenPassive
937
938This counter indicates how many times the TCP stack accepts the fast
939open request.
940
941* TcpExtTCPFastOpenPassiveFail
942
943This counter indicates how many times the TCP stack rejects the fast
944open request. It is caused by either the TFO cookie is invalid or the
945TCP stack finds an error during the socket creating process.
946
947* TcpExtTCPFastOpenListenOverflow
948
949When the pending fast open request number is larger than
950fastopenq->max_qlen, the TCP stack will reject the fast open request
951and update this counter. When this counter is updated, the TCP stack
952won't update TcpExtTCPFastOpenPassive or
953TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
954TCP_FASTOPEN socket operation and it could not be larger than
955net.core.somaxconn. For example:
956
957setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
958
959* TcpExtTCPFastOpenCookieReqd
960
961This counter indicates how many times a client wants to request a TFO
962cookie.
963
964SYN cookies
965===========
966SYN cookies are used to mitigate SYN flood, for details, please refer
967the `SYN cookies wiki`_.
968
969.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
970
971* TcpExtSyncookiesSent
972
973It indicates how many SYN cookies are sent.
974
975* TcpExtSyncookiesRecv
976
977How many reply packets of the SYN cookies the TCP stack receives.
978
979* TcpExtSyncookiesFailed
980
981The MSS decoded from the SYN cookie is invalid. When this counter is
982updated, the received packet won't be treated as a SYN cookie and the
983TcpExtSyncookiesRecv counter wont be updated.
984
985Challenge ACK
986=============
987For details of challenge ACK, please refer the explaination of
988TcpExtTCPACKSkippedChallenge.
989
990* TcpExtTCPChallengeACK
991
992The number of challenge acks sent.
993
994* TcpExtTCPSYNChallenge
995
996The number of challenge acks sent in response to SYN packets. After
997updates this counter, the TCP stack might send a challenge ACK and
998update the TcpExtTCPChallengeACK counter, or it might also skip to
999send the challenge and update the TcpExtTCPACKSkippedChallenge.
1000
1001prune
1002=====
1003When a socket is under memory pressure, the TCP stack will try to
1004reclaim memory from the receiving queue and out of order queue. One of
1005the reclaiming method is 'collapse', which means allocate a big sbk,
1006copy the contiguous skbs to the single big skb, and free these
1007contiguous skbs.
1008
1009* TcpExtPruneCalled
1010
1011The TCP stack tries to reclaim memory for a socket. After updates this
1012counter, the TCP stack will try to collapse the out of order queue and
1013the receiving queue. If the memory is still not enough, the TCP stack
1014will try to discard packets from the out of order queue (and update the
1015TcpExtOfoPruned counter)
1016
1017* TcpExtOfoPruned
1018
1019The TCP stack tries to discard packet on the out of order queue.
1020
1021* TcpExtRcvPruned
1022
1023After 'collapse' and discard packets from the out of order queue, if
1024the actually used memory is still larger than the max allowed memory,
1025this counter will be updated. It means the 'prune' fails.
1026
1027* TcpExtTCPRcvCollapsed
1028
1029This counter indicates how many skbs are freed during 'collapse'.
1030
1031examples
1032========
1033
1034ping test
1035---------
1036Run the ping command against the public dns server 8.8.8.8::
1037
1038  nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
1039  PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
1040  64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
1041
1042  --- 8.8.8.8 ping statistics ---
1043  1 packets transmitted, 1 received, 0% packet loss, time 0ms
1044  rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
1045
1046The nstayt result::
1047
1048  nstatuser@nstat-a:~$ nstat
1049  #kernel
1050  IpInReceives                    1                  0.0
1051  IpInDelivers                    1                  0.0
1052  IpOutRequests                   1                  0.0
1053  IcmpInMsgs                      1                  0.0
1054  IcmpInEchoReps                  1                  0.0
1055  IcmpOutMsgs                     1                  0.0
1056  IcmpOutEchos                    1                  0.0
1057  IcmpMsgInType0                  1                  0.0
1058  IcmpMsgOutType8                 1                  0.0
1059  IpExtInOctets                   84                 0.0
1060  IpExtOutOctets                  84                 0.0
1061  IpExtInNoECTPkts                1                  0.0
1062
1063The Linux server sent an ICMP Echo packet, so IpOutRequests,
1064IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
1065server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
1066IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
1067was passed to the ICMP layer via IP layer, so IpInDelivers was
1068increased 1. The default ping data size is 48, so an ICMP Echo packet
1069and its corresponding Echo Reply packet are constructed by:
1070
1071* 14 bytes MAC header
1072* 20 bytes IP header
1073* 16 bytes ICMP header
1074* 48 bytes data (default value of the ping command)
1075
1076So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
1077
1078tcp 3-way handshake
1079-------------------
1080On server side, we run::
1081
1082  nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
1083  Listening on [0.0.0.0] (family 0, port 9000)
1084
1085On client side, we run::
1086
1087  nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
1088  Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
1089
1090The server listened on tcp 9000 port, the client connected to it, they
1091completed the 3-way handshake.
1092
1093On server side, we can find below nstat output::
1094
1095  nstatuser@nstat-b:~$ nstat | grep -i tcp
1096  TcpPassiveOpens                 1                  0.0
1097  TcpInSegs                       2                  0.0
1098  TcpOutSegs                      1                  0.0
1099  TcpExtTCPPureAcks               1                  0.0
1100
1101On client side, we can find below nstat output::
1102
1103  nstatuser@nstat-a:~$ nstat | grep -i tcp
1104  TcpActiveOpens                  1                  0.0
1105  TcpInSegs                       1                  0.0
1106  TcpOutSegs                      2                  0.0
1107
1108When the server received the first SYN, it replied a SYN+ACK, and came into
1109SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
1110SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
1111packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
1112of the 3-way handshake is a pure ACK without data, so
1113TcpExtTCPPureAcks increased 1.
1114
1115When the client sent SYN, the client came into the SYN-SENT state, so
1116TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
1117ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
11181, TcpOutSegs increased 2.
1119
1120TCP normal traffic
1121------------------
1122Run nc on server::
1123
1124  nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1125  Listening on [0.0.0.0] (family 0, port 9000)
1126
1127Run nc on client::
1128
1129  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1130  Connection to nstat-b 9000 port [tcp/*] succeeded!
1131
1132Input a string in the nc client ('hello' in our example)::
1133
1134  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1135  Connection to nstat-b 9000 port [tcp/*] succeeded!
1136  hello
1137
1138The client side nstat output::
1139
1140  nstatuser@nstat-a:~$ nstat
1141  #kernel
1142  IpInReceives                    1                  0.0
1143  IpInDelivers                    1                  0.0
1144  IpOutRequests                   1                  0.0
1145  TcpInSegs                       1                  0.0
1146  TcpOutSegs                      1                  0.0
1147  TcpExtTCPPureAcks               1                  0.0
1148  TcpExtTCPOrigDataSent           1                  0.0
1149  IpExtInOctets                   52                 0.0
1150  IpExtOutOctets                  58                 0.0
1151  IpExtInNoECTPkts                1                  0.0
1152
1153The server side nstat output::
1154
1155  nstatuser@nstat-b:~$ nstat
1156  #kernel
1157  IpInReceives                    1                  0.0
1158  IpInDelivers                    1                  0.0
1159  IpOutRequests                   1                  0.0
1160  TcpInSegs                       1                  0.0
1161  TcpOutSegs                      1                  0.0
1162  IpExtInOctets                   58                 0.0
1163  IpExtOutOctets                  52                 0.0
1164  IpExtInNoECTPkts                1                  0.0
1165
1166Input a string in nc client side again ('world' in our exmaple)::
1167
1168  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1169  Connection to nstat-b 9000 port [tcp/*] succeeded!
1170  hello
1171  world
1172
1173Client side nstat output::
1174
1175  nstatuser@nstat-a:~$ nstat
1176  #kernel
1177  IpInReceives                    1                  0.0
1178  IpInDelivers                    1                  0.0
1179  IpOutRequests                   1                  0.0
1180  TcpInSegs                       1                  0.0
1181  TcpOutSegs                      1                  0.0
1182  TcpExtTCPHPAcks                 1                  0.0
1183  TcpExtTCPOrigDataSent           1                  0.0
1184  IpExtInOctets                   52                 0.0
1185  IpExtOutOctets                  58                 0.0
1186  IpExtInNoECTPkts                1                  0.0
1187
1188
1189Server side nstat output::
1190
1191  nstatuser@nstat-b:~$ nstat
1192  #kernel
1193  IpInReceives                    1                  0.0
1194  IpInDelivers                    1                  0.0
1195  IpOutRequests                   1                  0.0
1196  TcpInSegs                       1                  0.0
1197  TcpOutSegs                      1                  0.0
1198  TcpExtTCPHPHits                 1                  0.0
1199  IpExtInOctets                   58                 0.0
1200  IpExtOutOctets                  52                 0.0
1201  IpExtInNoECTPkts                1                  0.0
1202
1203Compare the first client-side nstat and the second client-side nstat,
1204we could find one difference: the first one had a 'TcpExtTCPPureAcks',
1205but the second one had a 'TcpExtTCPHPAcks'. The first server-side
1206nstat and the second server-side nstat had a difference too: the
1207second server-side nstat had a TcpExtTCPHPHits, but the first
1208server-side nstat didn't have it. The network traffic patterns were
1209exactly the same: the client sent a packet to the server, the server
1210replied an ACK. But kernel handled them in different ways. When the
1211TCP window scale option is not used, kernel will try to enable fast
1212path immediately when the connection comes into the established state,
1213but if the TCP window scale option is used, kernel will disable the
1214fast path at first, and try to enable it after kerenl receives
1215packets. We could use the 'ss' command to verify whether the window
1216scale option is used. e.g. run below command on either server or
1217client::
1218
1219  nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
1220  Netid    Recv-Q     Send-Q            Local Address:Port             Peer Address:Port
1221  tcp      0          0               192.168.122.250:40654         192.168.122.251:9000
1222             ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
1223
1224The 'wscale:7,7' means both server and client set the window scale
1225option to 7. Now we could explain the nstat output in our test:
1226
1227In the first nstat output of client side, the client sent a packet, server
1228reply an ACK, when kernel handled this ACK, the fast path was not
1229enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
1230
1231In the second nstat output of client side, the client sent a packet again,
1232and received another ACK from the server, in this time, the fast path is
1233enabled, and the ACK was qualified for fast path, so it was handled by
1234the fast path, so this ACK was counted into TcpExtTCPHPAcks.
1235
1236In the first nstat output of server side, fast path was not enabled,
1237so there was no 'TcpExtTCPHPHits'.
1238
1239In the second nstat output of server side, the fast path was enabled,
1240and the packet received from client qualified for fast path, so it
1241was counted into 'TcpExtTCPHPHits'.
1242
1243TcpExtTCPAbortOnClose
1244---------------------
1245On the server side, we run below python script::
1246
1247  import socket
1248  import time
1249
1250  port = 9000
1251
1252  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1253  s.bind(('0.0.0.0', port))
1254  s.listen(1)
1255  sock, addr = s.accept()
1256  while True:
1257      time.sleep(9999999)
1258
1259This python script listen on 9000 port, but doesn't read anything from
1260the connection.
1261
1262On the client side, we send the string "hello" by nc::
1263
1264  nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
1265
1266Then, we come back to the server side, the server has received the "hello"
1267packet, and the TCP layer has acked this packet, but the application didn't
1268read it yet. We type Ctrl-C to terminate the server script. Then we
1269could find TcpExtTCPAbortOnClose increased 1 on the server side::
1270
1271  nstatuser@nstat-b:~$ nstat | grep -i abort
1272  TcpExtTCPAbortOnClose           1                  0.0
1273
1274If we run tcpdump on the server side, we could find the server sent a
1275RST after we type Ctrl-C.
1276
1277TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
1278---------------------------------------------------
1279Below is an example which let the orphan socket count be higher than
1280net.ipv4.tcp_max_orphans.
1281Change tcp_max_orphans to a smaller value on client::
1282
1283  sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
1284
1285Client code (create 64 connection to server)::
1286
1287  nstatuser@nstat-a:~$ cat client_orphan.py
1288  import socket
1289  import time
1290
1291  server = 'nstat-b' # server address
1292  port = 9000
1293
1294  count = 64
1295
1296  connection_list = []
1297
1298  for i in range(64):
1299      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1300      s.connect((server, port))
1301      connection_list.append(s)
1302      print("connection_count: %d" % len(connection_list))
1303
1304  while True:
1305      time.sleep(99999)
1306
1307Server code (accept 64 connection from client)::
1308
1309  nstatuser@nstat-b:~$ cat server_orphan.py
1310  import socket
1311  import time
1312
1313  port = 9000
1314  count = 64
1315
1316  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1317  s.bind(('0.0.0.0', port))
1318  s.listen(count)
1319  connection_list = []
1320  while True:
1321      sock, addr = s.accept()
1322      connection_list.append((sock, addr))
1323      print("connection_count: %d" % len(connection_list))
1324
1325Run the python scripts on server and client.
1326
1327On server::
1328
1329  python3 server_orphan.py
1330
1331On client::
1332
1333  python3 client_orphan.py
1334
1335Run iptables on server::
1336
1337  sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
1338
1339Type Ctrl-C on client, stop client_orphan.py.
1340
1341Check TcpExtTCPAbortOnMemory on client::
1342
1343  nstatuser@nstat-a:~$ nstat | grep -i abort
1344  TcpExtTCPAbortOnMemory          54                 0.0
1345
1346Check orphane socket count on client::
1347
1348  nstatuser@nstat-a:~$ ss -s
1349  Total: 131 (kernel 0)
1350  TCP:   14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
1351
1352  Transport Total     IP        IPv6
1353  *         0         -         -
1354  RAW       1         0         1
1355  UDP       1         1         0
1356  TCP       14        13        1
1357  INET      16        14        2
1358  FRAG      0         0         0
1359
1360The explanation of the test: after run server_orphan.py and
1361client_orphan.py, we set up 64 connections between server and
1362client. Run the iptables command, the server will drop all packets from
1363the client, type Ctrl-C on client_orphan.py, the system of the client
1364would try to close these connections, and before they are closed
1365gracefully, these connections became orphan sockets. As the iptables
1366of the server blocked packets from the client, the server won't receive fin
1367from the client, so all connection on clients would be stuck on FIN_WAIT_1
1368stage, so they will keep as orphan sockets until timeout. We have echo
136910 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
1370only keep 10 orphan sockets, for all other orphan sockets, the client
1371system sent RST for them and delete them. We have 64 connections, so
1372the 'ss -s' command shows the system has 10 orphan sockets, and the
1373value of TcpExtTCPAbortOnMemory was 54.
1374
1375An additional explanation about orphan socket count: You could find the
1376exactly orphan socket count by the 'ss -s' command, but when kernel
1377decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
1378doesn't always check the exactly orphan socket count. For increasing
1379performance, kernel checks an approximate count firstly, if the
1380approximate count is more than tcp_max_orphans, kernel checks the
1381exact count again. So if the approximate count is less than
1382tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
1383would find TcpExtTCPAbortOnMemory is not increased at all. If
1384tcp_max_orphans is large enough, it won't occur, but if you decrease
1385tcp_max_orphans to a small value like our test, you might find this
1386issue. So in our test, the client set up 64 connections although the
1387tcp_max_orphans is 10. If the client only set up 11 connections, we
1388can't find the change of TcpExtTCPAbortOnMemory.
1389
1390Continue the previous test, we wait for several minutes. Because of the
1391iptables on the server blocked the traffic, the server wouldn't receive
1392fin, and all the client's orphan sockets would timeout on the
1393FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
139410 timeout on the client::
1395
1396  nstatuser@nstat-a:~$ nstat | grep -i abort
1397  TcpExtTCPAbortOnTimeout         10                 0.0
1398
1399TcpExtTCPAbortOnLinger
1400----------------------
1401The server side code::
1402
1403  nstatuser@nstat-b:~$ cat server_linger.py
1404  import socket
1405  import time
1406
1407  port = 9000
1408
1409  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1410  s.bind(('0.0.0.0', port))
1411  s.listen(1)
1412  sock, addr = s.accept()
1413  while True:
1414      time.sleep(9999999)
1415
1416The client side code::
1417
1418  nstatuser@nstat-a:~$ cat client_linger.py
1419  import socket
1420  import struct
1421
1422  server = 'nstat-b' # server address
1423  port = 9000
1424
1425  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1426  s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
1427  s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
1428  s.connect((server, port))
1429  s.close()
1430
1431Run server_linger.py on server::
1432
1433  nstatuser@nstat-b:~$ python3 server_linger.py
1434
1435Run client_linger.py on client::
1436
1437  nstatuser@nstat-a:~$ python3 client_linger.py
1438
1439After run client_linger.py, check the output of nstat::
1440
1441  nstatuser@nstat-a:~$ nstat | grep -i abort
1442  TcpExtTCPAbortOnLinger          1                  0.0
1443
1444TcpExtTCPRcvCoalesce
1445--------------------
1446On the server, we run a program which listen on TCP port 9000, but
1447doesn't read any data::
1448
1449  import socket
1450  import time
1451  port = 9000
1452  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1453  s.bind(('0.0.0.0', port))
1454  s.listen(1)
1455  sock, addr = s.accept()
1456  while True:
1457      time.sleep(9999999)
1458
1459Save the above code as server_coalesce.py, and run::
1460
1461  python3 server_coalesce.py
1462
1463On the client, save below code as client_coalesce.py::
1464
1465  import socket
1466  server = 'nstat-b'
1467  port = 9000
1468  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
1469  s.connect((server, port))
1470
1471Run::
1472
1473  nstatuser@nstat-a:~$ python3 -i client_coalesce.py
1474
1475We use '-i' to come into the interactive mode, then a packet::
1476
1477  >>> s.send(b'foo')
1478  3
1479
1480Send a packet again::
1481
1482  >>> s.send(b'bar')
1483  3
1484
1485On the server, run nstat::
1486
1487  ubuntu@nstat-b:~$ nstat
1488  #kernel
1489  IpInReceives                    2                  0.0
1490  IpInDelivers                    2                  0.0
1491  IpOutRequests                   2                  0.0
1492  TcpInSegs                       2                  0.0
1493  TcpOutSegs                      2                  0.0
1494  TcpExtTCPRcvCoalesce            1                  0.0
1495  IpExtInOctets                   110                0.0
1496  IpExtOutOctets                  104                0.0
1497  IpExtInNoECTPkts                2                  0.0
1498
1499The client sent two packets, server didn't read any data. When
1500the second packet arrived at server, the first packet was still in
1501the receiving queue. So the TCP layer merged the two packets, and we
1502could find the TcpExtTCPRcvCoalesce increased 1.
1503
1504TcpExtListenOverflows and TcpExtListenDrops
1505-------------------------------------------
1506On server, run the nc command, listen on port 9000::
1507
1508  nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
1509  Listening on [0.0.0.0] (family 0, port 9000)
1510
1511On client, run 3 nc commands in different terminals::
1512
1513  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1514  Connection to nstat-b 9000 port [tcp/*] succeeded!
1515
1516The nc command only accepts 1 connection, and the accept queue length
1517is 1. On current linux implementation, set queue length to n means the
1518actual queue length is n+1. Now we create 3 connections, 1 is accepted
1519by nc, 2 in accepted queue, so the accept queue is full.
1520
1521Before running the 4th nc, we clean the nstat history on the server::
1522
1523  nstatuser@nstat-b:~$ nstat -n
1524
1525Run the 4th nc on the client::
1526
1527  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1528
1529If the nc server is running on kernel 4.10 or higher version, you
1530won't see the "Connection to ... succeeded!" string, because kernel
1531will drop the SYN if the accept queue is full. If the nc client is running
1532on an old kernel, you would see that the connection is succeeded,
1533because kernel would complete the 3 way handshake and keep the socket
1534on half open queue. I did the test on kernel 4.15. Below is the nstat
1535on the server::
1536
1537  nstatuser@nstat-b:~$ nstat
1538  #kernel
1539  IpInReceives                    4                  0.0
1540  IpInDelivers                    4                  0.0
1541  TcpInSegs                       4                  0.0
1542  TcpExtListenOverflows           4                  0.0
1543  TcpExtListenDrops               4                  0.0
1544  IpExtInOctets                   240                0.0
1545  IpExtInNoECTPkts                4                  0.0
1546
1547Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
1548between the 4th nc and the nstat was longer, the value of
1549TcpExtListenOverflows and TcpExtListenDrops would be larger, because
1550the SYN of the 4th nc was dropped, the client was retrying.
1551
1552IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
1553-------------------------------------------------
1554server A IP address: 192.168.122.250
1555server B IP address: 192.168.122.251
1556Prepare on server A, add a route to server B::
1557
1558  $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
1559
1560Prepare on server B, disable send_redirects for all interfaces::
1561
1562  $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
1563  $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
1564  $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
1565  $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
1566
1567We want to let sever A send a packet to 8.8.8.8, and route the packet
1568to server B. When server B receives such packet, it might send a ICMP
1569Redirect message to server A, set send_redirects to 0 will disable
1570this behavior.
1571
1572First, generate InAddrErrors. On server B, we disable IP forwarding::
1573
1574  $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
1575
1576On server A, we send packets to 8.8.8.8::
1577
1578  $ nc -v 8.8.8.8 53
1579
1580On server B, we check the output of nstat::
1581
1582  $ nstat
1583  #kernel
1584  IpInReceives                    3                  0.0
1585  IpInAddrErrors                  3                  0.0
1586  IpExtInOctets                   180                0.0
1587  IpExtInNoECTPkts                3                  0.0
1588
1589As we have let server A route 8.8.8.8 to server B, and we disabled IP
1590forwarding on server B, Server A sent packets to server B, then server B
1591dropped packets and increased IpInAddrErrors. As the nc command would
1592re-send the SYN packet if it didn't receive a SYN+ACK, we could find
1593multiple IpInAddrErrors.
1594
1595Second, generate IpExtInNoRoutes. On server B, we enable IP
1596forwarding::
1597
1598  $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
1599
1600Check the route table of server B and remove the default route::
1601
1602  $ ip route show
1603  default via 192.168.122.1 dev ens3 proto static
1604  192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
1605  $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
1606
1607On server A, we contact 8.8.8.8 again::
1608
1609  $ nc -v 8.8.8.8 53
1610  nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
1611
1612On server B, run nstat::
1613
1614  $ nstat
1615  #kernel
1616  IpInReceives                    1                  0.0
1617  IpOutRequests                   1                  0.0
1618  IcmpOutMsgs                     1                  0.0
1619  IcmpOutDestUnreachs             1                  0.0
1620  IcmpMsgOutType3                 1                  0.0
1621  IpExtInNoRoutes                 1                  0.0
1622  IpExtInOctets                   60                 0.0
1623  IpExtOutOctets                  88                 0.0
1624  IpExtInNoECTPkts                1                  0.0
1625
1626We enabled IP forwarding on server B, when server B received a packet
1627which destination IP address is 8.8.8.8, server B will try to forward
1628this packet. We have deleted the default route, there was no route for
16298.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
1630Destination Unreachable" message to server A.
1631
1632Third, generate IpOutNoRoutes. Run ping command on server B::
1633
1634  $ ping -c 1 8.8.8.8
1635  connect: Network is unreachable
1636
1637Run nstat on server B::
1638
1639  $ nstat
1640  #kernel
1641  IpOutNoRoutes                   1                  0.0
1642
1643We have deleted the default route on server B. Server B couldn't find
1644a route for the 8.8.8.8 IP address, so server B increased
1645IpOutNoRoutes.
1646
1647TcpExtTCPACKSkippedSynRecv
1648--------------------------
1649In this test, we send 3 same SYN packets from client to server. The
1650first SYN will let server create a socket, set it to Syn-Recv status,
1651and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
1652again, and record the reply time (the duplicate ACK reply time). The
1653third SYN will let server check the previous duplicate ACK reply time,
1654and decide to skip the duplicate ACK, then increase the
1655TcpExtTCPACKSkippedSynRecv counter.
1656
1657Run tcpdump to capture a SYN packet::
1658
1659  nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
1660  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1661
1662Open another terminal, run nc command::
1663
1664  nstatuser@nstat-a:~$ nc nstat-b 9000
1665
1666As the nstat-b didn't listen on port 9000, it should reply a RST, and
1667the nc command exited immediately. It was enough for the tcpdump
1668command to capture a SYN packet. A linux server might use hardware
1669offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
1670might be not correct. We call tcprewrite to fix it::
1671
1672  nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
1673
1674On nstat-b, we run nc to listen on port 9000::
1675
1676  nstatuser@nstat-b:~$ nc -lkv 9000
1677  Listening on [0.0.0.0] (family 0, port 9000)
1678
1679On nstat-a, we blocked the packet from port 9000, or nstat-a would send
1680RST to nstat-b::
1681
1682  nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
1683
1684Send 3 SYN repeatly to nstat-b::
1685
1686  nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
1687
1688Check snmp cunter on nstat-b::
1689
1690  nstatuser@nstat-b:~$ nstat | grep -i skip
1691  TcpExtTCPACKSkippedSynRecv      1                  0.0
1692
1693As we expected, TcpExtTCPACKSkippedSynRecv is 1.
1694
1695TcpExtTCPACKSkippedPAWS
1696-----------------------
1697To trigger PAWS, we could send an old SYN.
1698
1699On nstat-b, let nc listen on port 9000::
1700
1701  nstatuser@nstat-b:~$ nc -lkv 9000
1702  Listening on [0.0.0.0] (family 0, port 9000)
1703
1704On nstat-a, run tcpdump to capture a SYN::
1705
1706  nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
1707  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1708
1709On nstat-a, run nc as a client to connect nstat-b::
1710
1711  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1712  Connection to nstat-b 9000 port [tcp/*] succeeded!
1713
1714Now the tcpdump has captured the SYN and exit. We should fix the
1715checksum::
1716
1717  nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
1718
1719Send the SYN packet twice::
1720
1721  nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
1722
1723On nstat-b, check the snmp counter::
1724
1725  nstatuser@nstat-b:~$ nstat | grep -i skip
1726  TcpExtTCPACKSkippedPAWS         1                  0.0
1727
1728We sent two SYN via tcpreplay, both of them would let PAWS check
1729failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
1730for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
1731
1732TcpExtTCPACKSkippedSeq
1733----------------------
1734To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
1735timestamp (to pass PAWS check) but the sequence number is out of
1736window. The linux TCP stack would avoid to skip if the packet has
1737data, so we need a pure ACK packet. To generate such a packet, we
1738could create two sockets: one on port 9000, another on port 9001. Then
1739we capture an ACK on port 9001, change the source/destination port
1740numbers to match the port 9000 socket. Then we could trigger
1741TcpExtTCPACKSkippedSeq via this packet.
1742
1743On nstat-b, open two terminals, run two nc commands to listen on both
1744port 9000 and port 9001::
1745
1746  nstatuser@nstat-b:~$ nc -lkv 9000
1747  Listening on [0.0.0.0] (family 0, port 9000)
1748
1749  nstatuser@nstat-b:~$ nc -lkv 9001
1750  Listening on [0.0.0.0] (family 0, port 9001)
1751
1752On nstat-a, run two nc clients::
1753
1754  nstatuser@nstat-a:~$ nc -v nstat-b 9000
1755  Connection to nstat-b 9000 port [tcp/*] succeeded!
1756
1757  nstatuser@nstat-a:~$ nc -v nstat-b 9001
1758  Connection to nstat-b 9001 port [tcp/*] succeeded!
1759
1760On nstat-a, run tcpdump to capture an ACK::
1761
1762  nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
1763  tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
1764
1765On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
1766string 'foo' in our example::
1767
1768  nstatuser@nstat-b:~$ nc -lkv 9001
1769  Listening on [0.0.0.0] (family 0, port 9001)
1770  Connection from nstat-a 42132 received!
1771  foo
1772
1773On nstat-a, the tcpdump should have caputred the ACK. We should check
1774the source port numbers of the two nc clients::
1775
1776  nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
1777  State  Recv-Q   Send-Q         Local Address:Port           Peer Address:Port
1778  ESTAB  0        0            192.168.122.250:50208       192.168.122.251:9000
1779  ESTAB  0        0            192.168.122.250:42132       192.168.122.251:9001
1780
1781Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to
1782port 50208::
1783
1784  nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
1785
1786Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
1787
1788  nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
1789
1790Check TcpExtTCPACKSkippedSeq on nstat-b::
1791
1792  nstatuser@nstat-b:~$ nstat | grep -i skip
1793  TcpExtTCPACKSkippedSeq          1                  0.0
1794