1============ 2SNMP counter 3============ 4 5This document explains the meaning of SNMP counters. 6 7General IPv4 counters 8===================== 9All layer 4 packets and ICMP packets will change these counters, but 10these counters won't be changed by layer 2 packets (such as STP) or 11ARP packets. 12 13* IpInReceives 14 15Defined in `RFC1213 ipInReceives`_ 16 17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26 18 19The number of packets received by the IP layer. It gets increasing at the 20beginning of ip_rcv function, always be updated together with 21IpExtInOctets. It will be increased even if the packet is dropped 22later (e.g. due to the IP header is invalid or the checksum is wrong 23and so on). It indicates the number of aggregated segments after 24GRO/LRO. 25 26* IpInDelivers 27 28Defined in `RFC1213 ipInDelivers`_ 29 30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28 31 32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP, 33ICMP and so on. If no one listens on a raw socket, only kernel 34supported protocols will be delivered, if someone listens on the raw 35socket, all valid IP packets will be delivered. 36 37* IpOutRequests 38 39Defined in `RFC1213 ipOutRequests`_ 40 41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28 42 43The number of packets sent via IP layer, for both single cast and 44multicast packets, and would always be updated together with 45IpExtOutOctets. 46 47* IpExtInOctets and IpExtOutOctets 48 49They are Linux kernel extensions, no RFC definitions. Please note, 50RFC1213 indeed defines ifInOctets and ifOutOctets, but they 51are different things. The ifInOctets and ifOutOctets include the MAC 52layer header size but IpExtInOctets and IpExtOutOctets don't, they 53only include the IP layer header and the IP layer data. 54 55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts 56 57They indicate the number of four kinds of ECN IP packets, please refer 58`Explicit Congestion Notification`_ for more details. 59 60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6 61 62These 4 counters calculate how many packets received per ECN 63status. They count the real frame number regardless the LRO/GRO. So 64for the same packet, you might find that IpInReceives count 1, but 65IpExtInNoECTPkts counts 2 or more. 66 67* IpInHdrErrors 68 69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is 70dropped due to the IP header error. It might happen in both IP input 71and IP forward paths. 72 73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27 74 75* IpInAddrErrors 76 77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two 78scenarios: (1) The IP address is invalid. (2) The destination IP 79address is not a local address and IP forwarding is not enabled 80 81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27 82 83* IpExtInNoRoutes 84 85This counter means the packet is dropped when the IP stack receives a 86packet and can't find a route for it from the route table. It might 87happen when IP forwarding is enabled and the destination IP address is 88not a local address and there is no route for the destination IP 89address. 90 91* IpInUnknownProtos 92 93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the 94layer 4 protocol is unsupported by kernel. If an application is using 95raw socket, kernel will always deliver the packet to the raw socket 96and this counter won't be increased. 97 98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27 99 100* IpExtInTruncatedPkts 101 102For IPv4 packet, it means the actual data size is smaller than the 103"Total Length" field in the IPv4 header. 104 105* IpInDiscards 106 107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped 108in the IP receiving path and due to kernel internal reasons (e.g. no 109enough memory). 110 111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28 112 113* IpOutDiscards 114 115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is 116dropped in the IP sending path and due to kernel internal reasons. 117 118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28 119 120* IpOutNoRoutes 121 122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is 123dropped in the IP sending path and no route is found for it. 124 125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29 126 127ICMP counters 128============= 129* IcmpInMsgs and IcmpOutMsgs 130 131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_ 132 133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41 134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43 135 136As mentioned in the RFC1213, these two counters include errors, they 137would be increased even if the ICMP packet has an invalid type. The 138ICMP output path will check the header of a raw socket, so the 139IcmpOutMsgs would still be updated if the IP header is constructed by 140a userspace program. 141 142* ICMP named types 143 144| These counters include most of common ICMP types, they are: 145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_ 146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ 147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ 148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_ 149| IcmpInRedirects: `RFC1213 icmpInRedirects`_ 150| IcmpInEchos: `RFC1213 icmpInEchos`_ 151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ 152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_ 153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_ 154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ 155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_ 156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_ 157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_ 158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_ 159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_ 160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_ 161| IcmpOutEchos: `RFC1213 icmpOutEchos`_ 162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ 163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_ 164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_ 165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_ 166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_ 167 168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41 169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41 170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42 171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42 172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42 173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42 174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42 175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42 176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43 177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43 178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43 179 180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44 181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44 182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44 183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44 184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44 185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45 186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45 187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45 188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45 189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45 190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46 191 192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP 193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are 194straightforward. The 'In' counter means kernel receives such a packet 195and the 'Out' counter means kernel sends such a packet. 196 197* ICMP numeric types 198 199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the 200ICMP type number. These counters track all kinds of ICMP packets. The 201ICMP type number definition could be found in the `ICMP parameters`_ 202document. 203 204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml 205 206For example, if the Linux kernel sends an ICMP Echo packet, the 207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply 208packet, IcmpMsgInType0 would increase 1. 209 210* IcmpInCsumErrors 211 212This counter indicates the checksum of the ICMP packet is 213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and 214before updating IcmpMsgInType[N]. If a packet has bad checksum, the 215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated. 216 217* IcmpInErrors and IcmpOutErrors 218 219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_ 220 221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41 222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43 223 224When an error occurs in the ICMP packet handler path, these two 225counters would be updated. The receiving packet path use IcmpInErrors 226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors 227is increased, IcmpInErrors would always be increased too. 228 229relationship of the ICMP counters 230--------------------------------- 231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they 232are updated at the same time. The sum of IcmpMsgInType[N] plus 233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel 234receives an ICMP packet, kernel follows below logic: 235 2361. increase IcmpInMsgs 2372. if has any error, update IcmpInErrors and finish the process 2383. update IcmpMsgOutType[N] 2394. handle the packet depending on the type, if has any error, update 240 IcmpInErrors and finish the process 241 242So if all errors occur in step (2), IcmpInMsgs should be equal to the 243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in 244step (4), IcmpInMsgs should be equal to the sum of 245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4), 246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus 247IcmpInErrors. 248 249General TCP counters 250==================== 251* TcpInSegs 252 253Defined in `RFC1213 tcpInSegs`_ 254 255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48 256 257The number of packets received by the TCP layer. As mentioned in 258RFC1213, it includes the packets received in error, such as checksum 259error, invalid TCP header and so on. Only one error won't be included: 260if the layer 2 destination address is not the NIC's layer 2 261address. It might happen if the packet is a multicast or broadcast 262packet, or the NIC is in promiscuous mode. In these situations, the 263packets would be delivered to the TCP layer, but the TCP layer will discard 264these packets before increasing TcpInSegs. The TcpInSegs counter 265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs 266counter would only increase 1. 267 268* TcpOutSegs 269 270Defined in `RFC1213 tcpOutSegs`_ 271 272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48 273 274The number of packets sent by the TCP layer. As mentioned in RFC1213, 275it excludes the retransmitted packets. But it includes the SYN, ACK 276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of 277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will 278increase 2. 279 280* TcpActiveOpens 281 282Defined in `RFC1213 tcpActiveOpens`_ 283 284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47 285 286It means the TCP layer sends a SYN, and come into the SYN-SENT 287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always 288increase 1. 289 290* TcpPassiveOpens 291 292Defined in `RFC1213 tcpPassiveOpens`_ 293 294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47 295 296It means the TCP layer receives a SYN, replies a SYN+ACK, come into 297the SYN-RCVD state. 298 299* TcpExtTCPRcvCoalesce 300 301When packets are received by the TCP layer and are not be read by the 302application, the TCP layer will try to merge them. This counter 303indicate how many packets are merged in such situation. If GRO is 304enabled, lots of packets would be merged by GRO, these packets 305wouldn't be counted to TcpExtTCPRcvCoalesce. 306 307* TcpExtTCPAutoCorking 308 309When sending packets, the TCP layer will try to merge small packets to 310a bigger one. This counter increase 1 for every packet merged in such 311situation. Please refer to the LWN article for more details: 312https://lwn.net/Articles/576263/ 313 314* TcpExtTCPOrigDataSent 315 316This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 317explaination below:: 318 319 TCPOrigDataSent: number of outgoing packets with original data (excluding 320 retransmission but including data-in-SYN). This counter is different from 321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is 322 more useful to track the TCP retransmission rate. 323 324* TCPSynRetrans 325 326This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 327explaination below:: 328 329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down 330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc. 331 332* TCPFastOpenActiveFail 333 334This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 335explaination below:: 336 337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because 338 the remote does not accept it or the attempts timed out. 339 340.. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd 341 342* TcpExtListenOverflows and TcpExtListenDrops 343 344When kernel receives a SYN from a client, and if the TCP accept queue 345is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. 346At the same time kernel will also add 1 to TcpExtListenDrops. When a 347TCP socket is in LISTEN state, and kernel need to drop a packet, 348kernel would always add 1 to TcpExtListenDrops. So increase 349TcpExtListenOverflows would let TcpExtListenDrops increasing at the 350same time, but TcpExtListenDrops would also increase without 351TcpExtListenOverflows increasing, e.g. a memory allocation fail would 352also let TcpExtListenDrops increase. 353 354Note: The above explanation is based on kernel 4.10 or above version, on 355an old kernel, the TCP stack has different behavior when TCP accept 356queue is full. On the old kernel, TCP stack won't drop the SYN, it 357would complete the 3-way handshake. As the accept queue is full, TCP 358stack will keep the socket in the TCP half-open queue. As it is in the 359half open queue, TCP stack will send SYN+ACK on an exponential backoff 360timer, after client replies ACK, TCP stack checks whether the accept 361queue is still full, if it is not full, moves the socket to the accept 362queue, if it is full, keeps the socket in the half-open queue, at next 363time client replies ACK, this socket will get another chance to move 364to the accept queue. 365 366 367TCP Fast Open 368============= 369* TcpEstabResets 370 371Defined in `RFC1213 tcpEstabResets`_. 372 373.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48 374 375* TcpAttemptFails 376 377Defined in `RFC1213 tcpAttemptFails`_. 378 379.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48 380 381* TcpOutRsts 382 383Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates 384the 'segments sent containing the RST flag', but in linux kernel, this 385couner indicates the segments kerenl tried to send. The sending 386process might be failed due to some errors (e.g. memory alloc failed). 387 388.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52 389 390* TcpExtTCPSpuriousRtxHostQueues 391 392When the TCP stack wants to retransmit a packet, and finds that packet 393is not lost in the network, but the packet is not sent yet, the TCP 394stack would give up the retransmission and update this counter. It 395might happen if a packet stays too long time in a qdisc or driver 396queue. 397 398* TcpEstabResets 399 400The socket receives a RST packet in Establish or CloseWait state. 401 402* TcpExtTCPKeepAlive 403 404This counter indicates many keepalive packets were sent. The keepalive 405won't be enabled by default. A userspace program could enable it by 406setting the SO_KEEPALIVE socket option. 407 408* TcpExtTCPSpuriousRTOs 409 410The spurious retransmission timeout detected by the `F-RTO`_ 411algorithm. 412 413.. _F-RTO: https://tools.ietf.org/html/rfc5682 414 415TCP Fast Path 416============ 417When kernel receives a TCP packet, it has two paths to handler the 418packet, one is fast path, another is slow path. The comment in kernel 419code provides a good explanation of them, I pasted them below:: 420 421 It is split into a fast path and a slow path. The fast path is 422 disabled when: 423 424 - A zero window was announced from us 425 - zero window probing 426 is only handled properly on the slow path. 427 - Out of order segments arrived. 428 - Urgent data is expected. 429 - There is no buffer space left 430 - Unexpected TCP flags/window values/header lengths are received 431 (detected by checking the TCP header against pred_flags) 432 - Data is sent in both directions. The fast path only supports pure senders 433 or pure receivers (this means either the sequence number or the ack 434 value must stay constant) 435 - Unexpected TCP option. 436 437Kernel will try to use fast path unless any of the above conditions 438are satisfied. If the packets are out of order, kernel will handle 439them in slow path, which means the performance might be not very 440good. Kernel would also come into slow path if the "Delayed ack" is 441used, because when using "Delayed ack", the data is sent in both 442directions. When the TCP window scale option is not used, kernel will 443try to enable fast path immediately when the connection comes into the 444established state, but if the TCP window scale option is used, kernel 445will disable the fast path at first, and try to enable it after kernel 446receives packets. 447 448* TcpExtTCPPureAcks and TcpExtTCPHPAcks 449 450If a packet set ACK flag and has no data, it is a pure ACK packet, if 451kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, 452if kernel handles it in the slow path, TcpExtTCPPureAcks will 453increase 1. 454 455* TcpExtTCPHPHits 456 457If a TCP packet has data (which means it is not a pure ACK packet), 458and this packet is handled in the fast path, TcpExtTCPHPHits will 459increase 1. 460 461 462TCP abort 463========= 464* TcpExtTCPAbortOnData 465 466It means TCP layer has data in flight, but need to close the 467connection. So TCP layer sends a RST to the other side, indicate the 468connection is not closed very graceful. An easy way to increase this 469counter is using the SO_LINGER option. Please refer to the SO_LINGER 470section of the `socket man page`_: 471 472.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html 473 474By default, when an application closes a connection, the close function 475will return immediately and kernel will try to send the in-flight data 476async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger 477to a positive number, the close function won't return immediately, but 478wait for the in-flight data are acked by the other side, the max wait 479time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, 480when the application closes a connection, kernel will send a RST 481immediately and increase the TcpExtTCPAbortOnData counter. 482 483* TcpExtTCPAbortOnClose 484 485This counter means the application has unread data in the TCP layer when 486the application wants to close the TCP connection. In such a situation, 487kernel will send a RST to the other side of the TCP connection. 488 489* TcpExtTCPAbortOnMemory 490 491When an application closes a TCP connection, kernel still need to track 492the connection, let it complete the TCP disconnect process. E.g. an 493app calls the close method of a socket, kernel sends fin to the other 494side of the connection, then the app has no relationship with the 495socket any more, but kernel need to keep the socket, this socket 496becomes an orphan socket, kernel waits for the reply of the other side, 497and would come to the TIME_WAIT state finally. When kernel has no 498enough memory to keep the orphan socket, kernel would send an RST to 499the other side, and delete the socket, in such situation, kernel will 500increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger 501TcpExtTCPAbortOnMemory: 502 5031. the memory used by the TCP protocol is higher than the third value of 504the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_: 505 506.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html 507 5082. the orphan socket count is higher than net.ipv4.tcp_max_orphans 509 510 511* TcpExtTCPAbortOnTimeout 512 513This counter will increase when any of the TCP timers expire. In such 514situation, kernel won't send RST, just give up the connection. 515 516* TcpExtTCPAbortOnLinger 517 518When a TCP connection comes into FIN_WAIT_2 state, instead of waiting 519for the fin packet from the other side, kernel could send a RST and 520delete the socket immediately. This is not the default behavior of 521Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, 522you could let kernel follow this behavior. 523 524* TcpExtTCPAbortFailed 525 526The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is 527satisfied. If an internal error occurs during this process, 528TcpExtTCPAbortFailed will be increased. 529 530.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50 531 532TCP Hybrid Slow Start 533===================== 534The Hybrid Slow Start algorithm is an enhancement of the traditional 535TCP congestion window Slow Start algorithm. It uses two pieces of 536information to detect whether the max bandwidth of the TCP path is 537approached. The two pieces of information are ACK train length and 538increase in packet delay. For detail information, please refer the 539`Hybrid Slow Start paper`_. Either ACK train length or packet delay 540hits a specific threshold, the congestion control algorithm will come 541into the Congestion Avoidance state. Until v4.20, two congestion 542control algorithms are using Hybrid Slow Start, they are cubic (the 543default congestion control algorithm) and cdg. Four snmp counters 544relate with the Hybrid Slow Start algorithm. 545 546.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf 547 548* TcpExtTCPHystartTrainDetect 549 550How many times the ACK train length threshold is detected 551 552* TcpExtTCPHystartTrainCwnd 553 554The sum of CWND detected by ACK train length. Dividing this value by 555TcpExtTCPHystartTrainDetect is the average CWND which detected by the 556ACK train length. 557 558* TcpExtTCPHystartDelayDetect 559 560How many times the packet delay threshold is detected. 561 562* TcpExtTCPHystartDelayCwnd 563 564The sum of CWND detected by packet delay. Dividing this value by 565TcpExtTCPHystartDelayDetect is the average CWND which detected by the 566packet delay. 567 568TCP retransmission and congestion control 569========================================= 570The TCP protocol has two retransmission mechanisms: SACK and fast 571recovery. They are exclusive with each other. When SACK is enabled, 572the kernel TCP stack would use SACK, or kernel would use fast 573recovery. The SACK is a TCP option, which is defined in `RFC2018`_, 574the fast recovery is defined in `RFC6582`_, which is also called 575'Reno'. 576 577The TCP congestion control is a big and complex topic. To understand 578the related snmp counter, we need to know the states of the congestion 579control state machine. There are 5 states: Open, Disorder, CWR, 580Recovery and Loss. For details about these states, please refer page 5 581and page 6 of this document: 582https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf 583 584.. _RFC2018: https://tools.ietf.org/html/rfc2018 585.. _RFC6582: https://tools.ietf.org/html/rfc6582 586 587* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery 588 589When the congestion control comes into Recovery state, if sack is 590used, TcpExtTCPSackRecovery increases 1, if sack is not used, 591TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP 592stack begins to retransmit the lost packets. 593 594* TcpExtTCPSACKReneging 595 596A packet was acknowledged by SACK, but the receiver has dropped this 597packet, so the sender needs to retransmit this packet. In this 598situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver 599could drop a packet which has been acknowledged by SACK, although it is 600unusual, it is allowed by the TCP protocol. The sender doesn't really 601know what happened on the receiver side. The sender just waits until 602the RTO expires for this packet, then the sender assumes this packet 603has been dropped by the receiver. 604 605* TcpExtTCPRenoReorder 606 607The reorder packet is detected by fast recovery. It would only be used 608if SACK is disabled. The fast recovery algorithm detects recorder by 609the duplicate ACK number. E.g., if retransmission is triggered, and 610the original retransmitted packet is not lost, it is just out of 611order, the receiver would acknowledge multiple times, one for the 612retransmitted packet, another for the arriving of the original out of 613order packet. Thus the sender would find more ACks than its 614expectation, and the sender knows out of order occurs. 615 616* TcpExtTCPTSReorder 617 618The reorder packet is detected when a hole is filled. E.g., assume the 619sender sends packet 1,2,3,4,5, and the receiving order is 6201,2,4,5,3. When the sender receives the ACK of packet 3 (which will 621fill the hole), two conditions will let TcpExtTCPTSReorder increase 6221: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 6233 is retransmitted but the timestamp of the packet 3's ACK is earlier 624than the retransmission timestamp. 625 626* TcpExtTCPSACKReorder 627 628The reorder packet detected by SACK. The SACK has two methods to 629detect reorder: (1) DSACK is received by the sender. It means the 630sender sends the same packet more than one times. And the only reason 631is the sender believes an out of order packet is lost so it sends the 632packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and 633the sender has received SACKs for packet 2 and 5, now the sender 634receives SACK for packet 4 and the sender doesn't retransmit the 635packet yet, the sender would know packet 4 is out of order. The TCP 636stack of kernel will increase TcpExtTCPSACKReorder for both of the 637above scenarios. 638 639* TcpExtTCPSlowStartRetrans 640 641The TCP stack wants to retransmit a packet and the congestion control 642state is 'Loss'. 643 644* TcpExtTCPFastRetrans 645 646The TCP stack wants to retransmit a packet and the congestion control 647state is not 'Loss'. 648 649* TcpExtTCPLostRetransmit 650 651A SACK points out that a retransmission packet is lost again. 652 653* TcpExtTCPRetransFail 654 655The TCP stack tries to deliver a retransmission packet to lower layers 656but the lower layers return an error. 657 658* TcpExtTCPSynRetrans 659 660The TCP stack retransmits a SYN packet. 661 662DSACK 663===== 664The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report 665duplicate packets to the sender. There are two kinds of 666duplications: (1) a packet which has been acknowledged is 667duplicate. (2) an out of order packet is duplicate. The TCP stack 668counts these two kinds of duplications on both receiver side and 669sender side. 670 671.. _RFC2883 : https://tools.ietf.org/html/rfc2883 672 673* TcpExtTCPDSACKOldSent 674 675The TCP stack receives a duplicate packet which has been acked, so it 676sends a DSACK to the sender. 677 678* TcpExtTCPDSACKOfoSent 679 680The TCP stack receives an out of order duplicate packet, so it sends a 681DSACK to the sender. 682 683* TcpExtTCPDSACKRecv 684The TCP stack receives a DSACK, which indicates an acknowledged 685duplicate packet is received. 686 687* TcpExtTCPDSACKOfoRecv 688 689The TCP stack receives a DSACK, which indicate an out of order 690duplicate packet is received. 691 692invalid SACK and DSACK 693==================== 694When a SACK (or DSACK) block is invalid, a corresponding counter would 695be updated. The validation method is base on the start/end sequence 696number of the SACK block. For more details, please refer the comment 697of the function tcp_is_sackblock_valid in the kernel source code. A 698SACK option could have up to 4 blocks, they are checked 699individually. E.g., if 3 blocks of a SACk is invalid, the 700corresponding counter would be updated 3 times. The comment of the 701`Add counters for discarded SACK blocks`_ patch has additional 702explaination: 703 704.. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32 705 706* TcpExtTCPSACKDiscard 707This counter indicates how many SACK blocks are invalid. If the invalid 708SACK block is caused by ACK recording, the TCP stack will only ignore 709it and won't update this counter. 710 711* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo 712When a DSACK block is invalid, one of these two counters would be 713updated. Which counter will be updated depends on the undo_marker flag 714of the TCP socket. If the undo_marker is not set, the TCP stack isn't 715likely to re-transmit any packets, and we still receive an invalid 716DSACK block, the reason might be that the packet is duplicated in the 717middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo 718will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld 719will be updated. As implied in its name, it might be an old packet. 720 721SACK shift 722========= 723The linux networking stack stores data in sk_buff struct (skb for 724short). If a SACK block acrosses multiple skb, the TCP stack will try 725to re-arrange data in these skb. E.g. if a SACK block acknowledges seq 72610 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and 72715 in skb2 would be moved to skb1. This operation is 'shift'. If a 728SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has 729seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be 730discard, this operation is 'merge'. 731 732* TcpExtTCPSackShifted 733A skb is shifted 734 735* TcpExtTCPSackMerged 736A skb is merged 737 738* TcpExtTCPSackShiftFallback 739A skb should be shifted or merged, but the TCP stack doesn't do it for 740some reasons. 741 742TCP out of order 743================ 744* TcpExtTCPOFOQueue 745 746The TCP layer receives an out of order packet and has enough memory 747to queue it. 748 749* TcpExtTCPOFODrop 750 751The TCP layer receives an out of order packet but doesn't have enough 752memory, so drops it. Such packets won't be counted into 753TcpExtTCPOFOQueue. 754 755* TcpExtTCPOFOMerge 756 757The received out of order packet has an overlay with the previous 758packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge 759packets will also be counted into TcpExtTCPOFOQueue. 760 761TCP PAWS 762======== 763PAWS (Protection Against Wrapped Sequence numbers) is an algorithm 764which is used to drop old packets. It depends on the TCP 765timestamps. For detail information, please refer the `timestamp wiki`_ 766and the `RFC of PAWS`_. 767 768.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17 769.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps 770 771* TcpExtPAWSActive 772 773Packets are dropped by PAWS in Syn-Sent status. 774 775* TcpExtPAWSEstab 776 777Packets are dropped by PAWS in any status other than Syn-Sent. 778 779TCP ACK skip 780============ 781In some scenarios, kernel would avoid sending duplicate ACKs too 782frequently. Please find more details in the tcp_invalid_ratelimit 783section of the `sysctl document`_. When kernel decides to skip an ACK 784due to tcp_invalid_ratelimit, kernel would update one of below 785counters to indicate the ACK is skipped in which scenario. The ACK 786would only be skipped if the received packet is either a SYN packet or 787it has no data. 788 789.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt 790 791* TcpExtTCPACKSkippedSynRecv 792 793The ACK is skipped in Syn-Recv status. The Syn-Recv status means the 794TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is 795waiting for an ACK. Generally, the TCP stack doesn't need to send ACK 796in the Syn-Recv status. But in several scenarios, the TCP stack need 797to send an ACK. E.g., the TCP stack receives the same SYN packet 798repeately, the received packet does not pass the PAWS check, or the 799received packet sequence number is out of window. In these scenarios, 800the TCP stack needs to send ACK. If the ACk sending frequency is higher than 801tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and 802increase TcpExtTCPACKSkippedSynRecv. 803 804 805* TcpExtTCPACKSkippedPAWS 806 807The ACK is skipped due to PAWS (Protect Against Wrapped Sequence 808numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 809or Time-Wait statuses, the skipped ACK would be counted to 810TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or 811TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK 812would be counted to TcpExtTCPACKSkippedPAWS. 813 814* TcpExtTCPACKSkippedSeq 815 816The sequence number is out of window and the timestamp passes the PAWS 817check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait. 818 819* TcpExtTCPACKSkippedFinWait2 820 821The ACK is skipped in Fin-Wait-2 status, the reason would be either 822PAWS check fails or the received sequence number is out of window. 823 824* TcpExtTCPACKSkippedTimeWait 825 826Tha ACK is skipped in Time-Wait status, the reason would be either 827PAWS check failed or the received sequence number is out of window. 828 829* TcpExtTCPACKSkippedChallenge 830 831The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 8323 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, 833`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these 834three scenarios, In some TCP status, the linux TCP stack would also 835send challenge ACKs if the ACK number is before the first 836unacknowledged number (more strict than `RFC 5961 section 5.2`_). 837 838.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7 839.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9 840.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11 841 842TCP receive window 843================== 844* TcpExtTCPWantZeroWindowAdv 845 846Depending on current memory usage, the TCP stack tries to set receive 847window to zero. But the receive window might still be a no-zero 848value. For example, if the previous window size is 10, and the TCP 849stack receives 3 bytes, the current window size would be 7 even if the 850window size calculated by the memory usage is zero. 851 852* TcpExtTCPToZeroWindowAdv 853 854The TCP receive window is set to zero from a no-zero value. 855 856* TcpExtTCPFromZeroWindowAdv 857 858The TCP receive window is set to no-zero value from zero. 859 860 861Delayed ACK 862=========== 863The TCP Delayed ACK is a technique which is used for reducing the 864packet count in the network. For more details, please refer the 865`Delayed ACK wiki`_ 866 867.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment 868 869* TcpExtDelayedACKs 870 871A delayed ACK timer expires. The TCP stack will send a pure ACK packet 872and exit the delayed ACK mode. 873 874* TcpExtDelayedACKLocked 875 876A delayed ACK timer expires, but the TCP stack can't send an ACK 877immediately due to the socket is locked by a userspace program. The 878TCP stack will send a pure ACK later (after the userspace program 879unlock the socket). When the TCP stack sends the pure ACK later, the 880TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK 881mode. 882 883* TcpExtDelayedACKLost 884 885It will be updated when the TCP stack receives a packet which has been 886ACKed. A Delayed ACK loss might cause this issue, but it would also be 887triggered by other reasons, such as a packet is duplicated in the 888network. 889 890Tail Loss Probe (TLP) 891===================== 892TLP is an algorithm which is used to detect TCP packet loss. For more 893details, please refer the `TLP paper`_. 894 895.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01 896 897* TcpExtTCPLossProbes 898 899A TLP probe packet is sent. 900 901* TcpExtTCPLossProbeRecovery 902 903A packet loss is detected and recovered by TLP. 904 905TCP Fast Open 906============= 907TCP Fast Open is a technology which allows data transfer before the 9083-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a 909general description. 910 911.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open 912 913* TcpExtTCPFastOpenActive 914 915When the TCP stack receives an ACK packet in the SYN-SENT status, and 916the ACK packet acknowledges the data in the SYN packet, the TCP stack 917understand the TFO cookie is accepted by the other side, then it 918updates this counter. 919 920* TcpExtTCPFastOpenActiveFail 921 922This counter indicates that the TCP stack initiated a TCP Fast Open, 923but it failed. This counter would be updated in three scenarios: (1) 924the other side doesn't acknowledge the data in the SYN packet. (2) The 925SYN packet which has the TFO cookie is timeout at least once. (3) 926after the 3-way handshake, the retransmission timeout happens 927net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole 928fast open after the handshake. 929 930* TcpExtTCPFastOpenPassive 931 932This counter indicates how many times the TCP stack accepts the fast 933open request. 934 935* TcpExtTCPFastOpenPassiveFail 936 937This counter indicates how many times the TCP stack rejects the fast 938open request. It is caused by either the TFO cookie is invalid or the 939TCP stack finds an error during the socket creating process. 940 941* TcpExtTCPFastOpenListenOverflow 942 943When the pending fast open request number is larger than 944fastopenq->max_qlen, the TCP stack will reject the fast open request 945and update this counter. When this counter is updated, the TCP stack 946won't update TcpExtTCPFastOpenPassive or 947TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the 948TCP_FASTOPEN socket operation and it could not be larger than 949net.core.somaxconn. For example: 950 951setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen)); 952 953* TcpExtTCPFastOpenCookieReqd 954 955This counter indicates how many times a client wants to request a TFO 956cookie. 957 958SYN cookies 959=========== 960SYN cookies are used to mitigate SYN flood, for details, please refer 961the `SYN cookies wiki`_. 962 963.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies 964 965* TcpExtSyncookiesSent 966 967It indicates how many SYN cookies are sent. 968 969* TcpExtSyncookiesRecv 970 971How many reply packets of the SYN cookies the TCP stack receives. 972 973* TcpExtSyncookiesFailed 974 975The MSS decoded from the SYN cookie is invalid. When this counter is 976updated, the received packet won't be treated as a SYN cookie and the 977TcpExtSyncookiesRecv counter wont be updated. 978 979Challenge ACK 980============= 981For details of challenge ACK, please refer the explaination of 982TcpExtTCPACKSkippedChallenge. 983 984* TcpExtTCPChallengeACK 985 986The number of challenge acks sent. 987 988* TcpExtTCPSYNChallenge 989 990The number of challenge acks sent in response to SYN packets. After 991updates this counter, the TCP stack might send a challenge ACK and 992update the TcpExtTCPChallengeACK counter, or it might also skip to 993send the challenge and update the TcpExtTCPACKSkippedChallenge. 994 995prune 996===== 997When a socket is under memory pressure, the TCP stack will try to 998reclaim memory from the receiving queue and out of order queue. One of 999the reclaiming method is 'collapse', which means allocate a big sbk, 1000copy the contiguous skbs to the single big skb, and free these 1001contiguous skbs. 1002 1003* TcpExtPruneCalled 1004 1005The TCP stack tries to reclaim memory for a socket. After updates this 1006counter, the TCP stack will try to collapse the out of order queue and 1007the receiving queue. If the memory is still not enough, the TCP stack 1008will try to discard packets from the out of order queue (and update the 1009TcpExtOfoPruned counter) 1010 1011* TcpExtOfoPruned 1012 1013The TCP stack tries to discard packet on the out of order queue. 1014 1015* TcpExtRcvPruned 1016 1017After 'collapse' and discard packets from the out of order queue, if 1018the actually used memory is still larger than the max allowed memory, 1019this counter will be updated. It means the 'prune' fails. 1020 1021* TcpExtTCPRcvCollapsed 1022 1023This counter indicates how many skbs are freed during 'collapse'. 1024 1025examples 1026======== 1027 1028ping test 1029--------- 1030Run the ping command against the public dns server 8.8.8.8:: 1031 1032 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 1033 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 1034 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms 1035 1036 --- 8.8.8.8 ping statistics --- 1037 1 packets transmitted, 1 received, 0% packet loss, time 0ms 1038 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms 1039 1040The nstayt result:: 1041 1042 nstatuser@nstat-a:~$ nstat 1043 #kernel 1044 IpInReceives 1 0.0 1045 IpInDelivers 1 0.0 1046 IpOutRequests 1 0.0 1047 IcmpInMsgs 1 0.0 1048 IcmpInEchoReps 1 0.0 1049 IcmpOutMsgs 1 0.0 1050 IcmpOutEchos 1 0.0 1051 IcmpMsgInType0 1 0.0 1052 IcmpMsgOutType8 1 0.0 1053 IpExtInOctets 84 0.0 1054 IpExtOutOctets 84 0.0 1055 IpExtInNoECTPkts 1 0.0 1056 1057The Linux server sent an ICMP Echo packet, so IpOutRequests, 1058IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The 1059server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs, 1060IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply 1061was passed to the ICMP layer via IP layer, so IpInDelivers was 1062increased 1. The default ping data size is 48, so an ICMP Echo packet 1063and its corresponding Echo Reply packet are constructed by: 1064 1065* 14 bytes MAC header 1066* 20 bytes IP header 1067* 16 bytes ICMP header 1068* 48 bytes data (default value of the ping command) 1069 1070So the IpExtInOctets and IpExtOutOctets are 20+16+48=84. 1071 1072tcp 3-way handshake 1073------------------- 1074On server side, we run:: 1075 1076 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 1077 Listening on [0.0.0.0] (family 0, port 9000) 1078 1079On client side, we run:: 1080 1081 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000 1082 Connection to 192.168.122.251 9000 port [tcp/*] succeeded! 1083 1084The server listened on tcp 9000 port, the client connected to it, they 1085completed the 3-way handshake. 1086 1087On server side, we can find below nstat output:: 1088 1089 nstatuser@nstat-b:~$ nstat | grep -i tcp 1090 TcpPassiveOpens 1 0.0 1091 TcpInSegs 2 0.0 1092 TcpOutSegs 1 0.0 1093 TcpExtTCPPureAcks 1 0.0 1094 1095On client side, we can find below nstat output:: 1096 1097 nstatuser@nstat-a:~$ nstat | grep -i tcp 1098 TcpActiveOpens 1 0.0 1099 TcpInSegs 1 0.0 1100 TcpOutSegs 2 0.0 1101 1102When the server received the first SYN, it replied a SYN+ACK, and came into 1103SYN-RCVD state, so TcpPassiveOpens increased 1. The server received 1104SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 1105packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK 1106of the 3-way handshake is a pure ACK without data, so 1107TcpExtTCPPureAcks increased 1. 1108 1109When the client sent SYN, the client came into the SYN-SENT state, so 1110TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent 1111ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 11121, TcpOutSegs increased 2. 1113 1114TCP normal traffic 1115------------------ 1116Run nc on server:: 1117 1118 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1119 Listening on [0.0.0.0] (family 0, port 9000) 1120 1121Run nc on client:: 1122 1123 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1124 Connection to nstat-b 9000 port [tcp/*] succeeded! 1125 1126Input a string in the nc client ('hello' in our example):: 1127 1128 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1129 Connection to nstat-b 9000 port [tcp/*] succeeded! 1130 hello 1131 1132The client side nstat output:: 1133 1134 nstatuser@nstat-a:~$ nstat 1135 #kernel 1136 IpInReceives 1 0.0 1137 IpInDelivers 1 0.0 1138 IpOutRequests 1 0.0 1139 TcpInSegs 1 0.0 1140 TcpOutSegs 1 0.0 1141 TcpExtTCPPureAcks 1 0.0 1142 TcpExtTCPOrigDataSent 1 0.0 1143 IpExtInOctets 52 0.0 1144 IpExtOutOctets 58 0.0 1145 IpExtInNoECTPkts 1 0.0 1146 1147The server side nstat output:: 1148 1149 nstatuser@nstat-b:~$ nstat 1150 #kernel 1151 IpInReceives 1 0.0 1152 IpInDelivers 1 0.0 1153 IpOutRequests 1 0.0 1154 TcpInSegs 1 0.0 1155 TcpOutSegs 1 0.0 1156 IpExtInOctets 58 0.0 1157 IpExtOutOctets 52 0.0 1158 IpExtInNoECTPkts 1 0.0 1159 1160Input a string in nc client side again ('world' in our exmaple):: 1161 1162 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1163 Connection to nstat-b 9000 port [tcp/*] succeeded! 1164 hello 1165 world 1166 1167Client side nstat output:: 1168 1169 nstatuser@nstat-a:~$ nstat 1170 #kernel 1171 IpInReceives 1 0.0 1172 IpInDelivers 1 0.0 1173 IpOutRequests 1 0.0 1174 TcpInSegs 1 0.0 1175 TcpOutSegs 1 0.0 1176 TcpExtTCPHPAcks 1 0.0 1177 TcpExtTCPOrigDataSent 1 0.0 1178 IpExtInOctets 52 0.0 1179 IpExtOutOctets 58 0.0 1180 IpExtInNoECTPkts 1 0.0 1181 1182 1183Server side nstat output:: 1184 1185 nstatuser@nstat-b:~$ nstat 1186 #kernel 1187 IpInReceives 1 0.0 1188 IpInDelivers 1 0.0 1189 IpOutRequests 1 0.0 1190 TcpInSegs 1 0.0 1191 TcpOutSegs 1 0.0 1192 TcpExtTCPHPHits 1 0.0 1193 IpExtInOctets 58 0.0 1194 IpExtOutOctets 52 0.0 1195 IpExtInNoECTPkts 1 0.0 1196 1197Compare the first client-side nstat and the second client-side nstat, 1198we could find one difference: the first one had a 'TcpExtTCPPureAcks', 1199but the second one had a 'TcpExtTCPHPAcks'. The first server-side 1200nstat and the second server-side nstat had a difference too: the 1201second server-side nstat had a TcpExtTCPHPHits, but the first 1202server-side nstat didn't have it. The network traffic patterns were 1203exactly the same: the client sent a packet to the server, the server 1204replied an ACK. But kernel handled them in different ways. When the 1205TCP window scale option is not used, kernel will try to enable fast 1206path immediately when the connection comes into the established state, 1207but if the TCP window scale option is used, kernel will disable the 1208fast path at first, and try to enable it after kerenl receives 1209packets. We could use the 'ss' command to verify whether the window 1210scale option is used. e.g. run below command on either server or 1211client:: 1212 1213 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) 1214 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port 1215 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 1216 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98 1217 1218The 'wscale:7,7' means both server and client set the window scale 1219option to 7. Now we could explain the nstat output in our test: 1220 1221In the first nstat output of client side, the client sent a packet, server 1222reply an ACK, when kernel handled this ACK, the fast path was not 1223enabled, so the ACK was counted into 'TcpExtTCPPureAcks'. 1224 1225In the second nstat output of client side, the client sent a packet again, 1226and received another ACK from the server, in this time, the fast path is 1227enabled, and the ACK was qualified for fast path, so it was handled by 1228the fast path, so this ACK was counted into TcpExtTCPHPAcks. 1229 1230In the first nstat output of server side, fast path was not enabled, 1231so there was no 'TcpExtTCPHPHits'. 1232 1233In the second nstat output of server side, the fast path was enabled, 1234and the packet received from client qualified for fast path, so it 1235was counted into 'TcpExtTCPHPHits'. 1236 1237TcpExtTCPAbortOnClose 1238--------------------- 1239On the server side, we run below python script:: 1240 1241 import socket 1242 import time 1243 1244 port = 9000 1245 1246 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1247 s.bind(('0.0.0.0', port)) 1248 s.listen(1) 1249 sock, addr = s.accept() 1250 while True: 1251 time.sleep(9999999) 1252 1253This python script listen on 9000 port, but doesn't read anything from 1254the connection. 1255 1256On the client side, we send the string "hello" by nc:: 1257 1258 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000 1259 1260Then, we come back to the server side, the server has received the "hello" 1261packet, and the TCP layer has acked this packet, but the application didn't 1262read it yet. We type Ctrl-C to terminate the server script. Then we 1263could find TcpExtTCPAbortOnClose increased 1 on the server side:: 1264 1265 nstatuser@nstat-b:~$ nstat | grep -i abort 1266 TcpExtTCPAbortOnClose 1 0.0 1267 1268If we run tcpdump on the server side, we could find the server sent a 1269RST after we type Ctrl-C. 1270 1271TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout 1272--------------------------------------------------- 1273Below is an example which let the orphan socket count be higher than 1274net.ipv4.tcp_max_orphans. 1275Change tcp_max_orphans to a smaller value on client:: 1276 1277 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans" 1278 1279Client code (create 64 connection to server):: 1280 1281 nstatuser@nstat-a:~$ cat client_orphan.py 1282 import socket 1283 import time 1284 1285 server = 'nstat-b' # server address 1286 port = 9000 1287 1288 count = 64 1289 1290 connection_list = [] 1291 1292 for i in range(64): 1293 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1294 s.connect((server, port)) 1295 connection_list.append(s) 1296 print("connection_count: %d" % len(connection_list)) 1297 1298 while True: 1299 time.sleep(99999) 1300 1301Server code (accept 64 connection from client):: 1302 1303 nstatuser@nstat-b:~$ cat server_orphan.py 1304 import socket 1305 import time 1306 1307 port = 9000 1308 count = 64 1309 1310 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1311 s.bind(('0.0.0.0', port)) 1312 s.listen(count) 1313 connection_list = [] 1314 while True: 1315 sock, addr = s.accept() 1316 connection_list.append((sock, addr)) 1317 print("connection_count: %d" % len(connection_list)) 1318 1319Run the python scripts on server and client. 1320 1321On server:: 1322 1323 python3 server_orphan.py 1324 1325On client:: 1326 1327 python3 client_orphan.py 1328 1329Run iptables on server:: 1330 1331 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP 1332 1333Type Ctrl-C on client, stop client_orphan.py. 1334 1335Check TcpExtTCPAbortOnMemory on client:: 1336 1337 nstatuser@nstat-a:~$ nstat | grep -i abort 1338 TcpExtTCPAbortOnMemory 54 0.0 1339 1340Check orphane socket count on client:: 1341 1342 nstatuser@nstat-a:~$ ss -s 1343 Total: 131 (kernel 0) 1344 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 1345 1346 Transport Total IP IPv6 1347 * 0 - - 1348 RAW 1 0 1 1349 UDP 1 1 0 1350 TCP 14 13 1 1351 INET 16 14 2 1352 FRAG 0 0 0 1353 1354The explanation of the test: after run server_orphan.py and 1355client_orphan.py, we set up 64 connections between server and 1356client. Run the iptables command, the server will drop all packets from 1357the client, type Ctrl-C on client_orphan.py, the system of the client 1358would try to close these connections, and before they are closed 1359gracefully, these connections became orphan sockets. As the iptables 1360of the server blocked packets from the client, the server won't receive fin 1361from the client, so all connection on clients would be stuck on FIN_WAIT_1 1362stage, so they will keep as orphan sockets until timeout. We have echo 136310 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would 1364only keep 10 orphan sockets, for all other orphan sockets, the client 1365system sent RST for them and delete them. We have 64 connections, so 1366the 'ss -s' command shows the system has 10 orphan sockets, and the 1367value of TcpExtTCPAbortOnMemory was 54. 1368 1369An additional explanation about orphan socket count: You could find the 1370exactly orphan socket count by the 'ss -s' command, but when kernel 1371decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel 1372doesn't always check the exactly orphan socket count. For increasing 1373performance, kernel checks an approximate count firstly, if the 1374approximate count is more than tcp_max_orphans, kernel checks the 1375exact count again. So if the approximate count is less than 1376tcp_max_orphans, but exactly count is more than tcp_max_orphans, you 1377would find TcpExtTCPAbortOnMemory is not increased at all. If 1378tcp_max_orphans is large enough, it won't occur, but if you decrease 1379tcp_max_orphans to a small value like our test, you might find this 1380issue. So in our test, the client set up 64 connections although the 1381tcp_max_orphans is 10. If the client only set up 11 connections, we 1382can't find the change of TcpExtTCPAbortOnMemory. 1383 1384Continue the previous test, we wait for several minutes. Because of the 1385iptables on the server blocked the traffic, the server wouldn't receive 1386fin, and all the client's orphan sockets would timeout on the 1387FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 138810 timeout on the client:: 1389 1390 nstatuser@nstat-a:~$ nstat | grep -i abort 1391 TcpExtTCPAbortOnTimeout 10 0.0 1392 1393TcpExtTCPAbortOnLinger 1394---------------------- 1395The server side code:: 1396 1397 nstatuser@nstat-b:~$ cat server_linger.py 1398 import socket 1399 import time 1400 1401 port = 9000 1402 1403 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1404 s.bind(('0.0.0.0', port)) 1405 s.listen(1) 1406 sock, addr = s.accept() 1407 while True: 1408 time.sleep(9999999) 1409 1410The client side code:: 1411 1412 nstatuser@nstat-a:~$ cat client_linger.py 1413 import socket 1414 import struct 1415 1416 server = 'nstat-b' # server address 1417 port = 9000 1418 1419 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1420 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) 1421 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) 1422 s.connect((server, port)) 1423 s.close() 1424 1425Run server_linger.py on server:: 1426 1427 nstatuser@nstat-b:~$ python3 server_linger.py 1428 1429Run client_linger.py on client:: 1430 1431 nstatuser@nstat-a:~$ python3 client_linger.py 1432 1433After run client_linger.py, check the output of nstat:: 1434 1435 nstatuser@nstat-a:~$ nstat | grep -i abort 1436 TcpExtTCPAbortOnLinger 1 0.0 1437 1438TcpExtTCPRcvCoalesce 1439-------------------- 1440On the server, we run a program which listen on TCP port 9000, but 1441doesn't read any data:: 1442 1443 import socket 1444 import time 1445 port = 9000 1446 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1447 s.bind(('0.0.0.0', port)) 1448 s.listen(1) 1449 sock, addr = s.accept() 1450 while True: 1451 time.sleep(9999999) 1452 1453Save the above code as server_coalesce.py, and run:: 1454 1455 python3 server_coalesce.py 1456 1457On the client, save below code as client_coalesce.py:: 1458 1459 import socket 1460 server = 'nstat-b' 1461 port = 9000 1462 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1463 s.connect((server, port)) 1464 1465Run:: 1466 1467 nstatuser@nstat-a:~$ python3 -i client_coalesce.py 1468 1469We use '-i' to come into the interactive mode, then a packet:: 1470 1471 >>> s.send(b'foo') 1472 3 1473 1474Send a packet again:: 1475 1476 >>> s.send(b'bar') 1477 3 1478 1479On the server, run nstat:: 1480 1481 ubuntu@nstat-b:~$ nstat 1482 #kernel 1483 IpInReceives 2 0.0 1484 IpInDelivers 2 0.0 1485 IpOutRequests 2 0.0 1486 TcpInSegs 2 0.0 1487 TcpOutSegs 2 0.0 1488 TcpExtTCPRcvCoalesce 1 0.0 1489 IpExtInOctets 110 0.0 1490 IpExtOutOctets 104 0.0 1491 IpExtInNoECTPkts 2 0.0 1492 1493The client sent two packets, server didn't read any data. When 1494the second packet arrived at server, the first packet was still in 1495the receiving queue. So the TCP layer merged the two packets, and we 1496could find the TcpExtTCPRcvCoalesce increased 1. 1497 1498TcpExtListenOverflows and TcpExtListenDrops 1499------------------------------------------- 1500On server, run the nc command, listen on port 9000:: 1501 1502 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1503 Listening on [0.0.0.0] (family 0, port 9000) 1504 1505On client, run 3 nc commands in different terminals:: 1506 1507 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1508 Connection to nstat-b 9000 port [tcp/*] succeeded! 1509 1510The nc command only accepts 1 connection, and the accept queue length 1511is 1. On current linux implementation, set queue length to n means the 1512actual queue length is n+1. Now we create 3 connections, 1 is accepted 1513by nc, 2 in accepted queue, so the accept queue is full. 1514 1515Before running the 4th nc, we clean the nstat history on the server:: 1516 1517 nstatuser@nstat-b:~$ nstat -n 1518 1519Run the 4th nc on the client:: 1520 1521 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1522 1523If the nc server is running on kernel 4.10 or higher version, you 1524won't see the "Connection to ... succeeded!" string, because kernel 1525will drop the SYN if the accept queue is full. If the nc client is running 1526on an old kernel, you would see that the connection is succeeded, 1527because kernel would complete the 3 way handshake and keep the socket 1528on half open queue. I did the test on kernel 4.15. Below is the nstat 1529on the server:: 1530 1531 nstatuser@nstat-b:~$ nstat 1532 #kernel 1533 IpInReceives 4 0.0 1534 IpInDelivers 4 0.0 1535 TcpInSegs 4 0.0 1536 TcpExtListenOverflows 4 0.0 1537 TcpExtListenDrops 4 0.0 1538 IpExtInOctets 240 0.0 1539 IpExtInNoECTPkts 4 0.0 1540 1541Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time 1542between the 4th nc and the nstat was longer, the value of 1543TcpExtListenOverflows and TcpExtListenDrops would be larger, because 1544the SYN of the 4th nc was dropped, the client was retrying. 1545 1546IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes 1547------------------------------------------------- 1548server A IP address: 192.168.122.250 1549server B IP address: 192.168.122.251 1550Prepare on server A, add a route to server B:: 1551 1552 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251 1553 1554Prepare on server B, disable send_redirects for all interfaces:: 1555 1556 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 1557 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 1558 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 1559 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0 1560 1561We want to let sever A send a packet to 8.8.8.8, and route the packet 1562to server B. When server B receives such packet, it might send a ICMP 1563Redirect message to server A, set send_redirects to 0 will disable 1564this behavior. 1565 1566First, generate InAddrErrors. On server B, we disable IP forwarding:: 1567 1568 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0 1569 1570On server A, we send packets to 8.8.8.8:: 1571 1572 $ nc -v 8.8.8.8 53 1573 1574On server B, we check the output of nstat:: 1575 1576 $ nstat 1577 #kernel 1578 IpInReceives 3 0.0 1579 IpInAddrErrors 3 0.0 1580 IpExtInOctets 180 0.0 1581 IpExtInNoECTPkts 3 0.0 1582 1583As we have let server A route 8.8.8.8 to server B, and we disabled IP 1584forwarding on server B, Server A sent packets to server B, then server B 1585dropped packets and increased IpInAddrErrors. As the nc command would 1586re-send the SYN packet if it didn't receive a SYN+ACK, we could find 1587multiple IpInAddrErrors. 1588 1589Second, generate IpExtInNoRoutes. On server B, we enable IP 1590forwarding:: 1591 1592 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1 1593 1594Check the route table of server B and remove the default route:: 1595 1596 $ ip route show 1597 default via 192.168.122.1 dev ens3 proto static 1598 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 1599 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static 1600 1601On server A, we contact 8.8.8.8 again:: 1602 1603 $ nc -v 8.8.8.8 53 1604 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable 1605 1606On server B, run nstat:: 1607 1608 $ nstat 1609 #kernel 1610 IpInReceives 1 0.0 1611 IpOutRequests 1 0.0 1612 IcmpOutMsgs 1 0.0 1613 IcmpOutDestUnreachs 1 0.0 1614 IcmpMsgOutType3 1 0.0 1615 IpExtInNoRoutes 1 0.0 1616 IpExtInOctets 60 0.0 1617 IpExtOutOctets 88 0.0 1618 IpExtInNoECTPkts 1 0.0 1619 1620We enabled IP forwarding on server B, when server B received a packet 1621which destination IP address is 8.8.8.8, server B will try to forward 1622this packet. We have deleted the default route, there was no route for 16238.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP 1624Destination Unreachable" message to server A. 1625 1626Third, generate IpOutNoRoutes. Run ping command on server B:: 1627 1628 $ ping -c 1 8.8.8.8 1629 connect: Network is unreachable 1630 1631Run nstat on server B:: 1632 1633 $ nstat 1634 #kernel 1635 IpOutNoRoutes 1 0.0 1636 1637We have deleted the default route on server B. Server B couldn't find 1638a route for the 8.8.8.8 IP address, so server B increased 1639IpOutNoRoutes. 1640 1641TcpExtTCPACKSkippedSynRecv 1642-------------------------- 1643In this test, we send 3 same SYN packets from client to server. The 1644first SYN will let server create a socket, set it to Syn-Recv status, 1645and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK 1646again, and record the reply time (the duplicate ACK reply time). The 1647third SYN will let server check the previous duplicate ACK reply time, 1648and decide to skip the duplicate ACK, then increase the 1649TcpExtTCPACKSkippedSynRecv counter. 1650 1651Run tcpdump to capture a SYN packet:: 1652 1653 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000 1654 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1655 1656Open another terminal, run nc command:: 1657 1658 nstatuser@nstat-a:~$ nc nstat-b 9000 1659 1660As the nstat-b didn't listen on port 9000, it should reply a RST, and 1661the nc command exited immediately. It was enough for the tcpdump 1662command to capture a SYN packet. A linux server might use hardware 1663offload for the TCP checksum, so the checksum in the /tmp/syn.pcap 1664might be not correct. We call tcprewrite to fix it:: 1665 1666 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum 1667 1668On nstat-b, we run nc to listen on port 9000:: 1669 1670 nstatuser@nstat-b:~$ nc -lkv 9000 1671 Listening on [0.0.0.0] (family 0, port 9000) 1672 1673On nstat-a, we blocked the packet from port 9000, or nstat-a would send 1674RST to nstat-b:: 1675 1676 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP 1677 1678Send 3 SYN repeatly to nstat-b:: 1679 1680 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done 1681 1682Check snmp cunter on nstat-b:: 1683 1684 nstatuser@nstat-b:~$ nstat | grep -i skip 1685 TcpExtTCPACKSkippedSynRecv 1 0.0 1686 1687As we expected, TcpExtTCPACKSkippedSynRecv is 1. 1688 1689TcpExtTCPACKSkippedPAWS 1690----------------------- 1691To trigger PAWS, we could send an old SYN. 1692 1693On nstat-b, let nc listen on port 9000:: 1694 1695 nstatuser@nstat-b:~$ nc -lkv 9000 1696 Listening on [0.0.0.0] (family 0, port 9000) 1697 1698On nstat-a, run tcpdump to capture a SYN:: 1699 1700 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000 1701 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1702 1703On nstat-a, run nc as a client to connect nstat-b:: 1704 1705 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1706 Connection to nstat-b 9000 port [tcp/*] succeeded! 1707 1708Now the tcpdump has captured the SYN and exit. We should fix the 1709checksum:: 1710 1711 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum 1712 1713Send the SYN packet twice:: 1714 1715 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done 1716 1717On nstat-b, check the snmp counter:: 1718 1719 nstatuser@nstat-b:~$ nstat | grep -i skip 1720 TcpExtTCPACKSkippedPAWS 1 0.0 1721 1722We sent two SYN via tcpreplay, both of them would let PAWS check 1723failed, the nstat-b replied an ACK for the first SYN, skipped the ACK 1724for the second SYN, and updated TcpExtTCPACKSkippedPAWS. 1725 1726TcpExtTCPACKSkippedSeq 1727---------------------- 1728To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid 1729timestamp (to pass PAWS check) but the sequence number is out of 1730window. The linux TCP stack would avoid to skip if the packet has 1731data, so we need a pure ACK packet. To generate such a packet, we 1732could create two sockets: one on port 9000, another on port 9001. Then 1733we capture an ACK on port 9001, change the source/destination port 1734numbers to match the port 9000 socket. Then we could trigger 1735TcpExtTCPACKSkippedSeq via this packet. 1736 1737On nstat-b, open two terminals, run two nc commands to listen on both 1738port 9000 and port 9001:: 1739 1740 nstatuser@nstat-b:~$ nc -lkv 9000 1741 Listening on [0.0.0.0] (family 0, port 9000) 1742 1743 nstatuser@nstat-b:~$ nc -lkv 9001 1744 Listening on [0.0.0.0] (family 0, port 9001) 1745 1746On nstat-a, run two nc clients:: 1747 1748 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1749 Connection to nstat-b 9000 port [tcp/*] succeeded! 1750 1751 nstatuser@nstat-a:~$ nc -v nstat-b 9001 1752 Connection to nstat-b 9001 port [tcp/*] succeeded! 1753 1754On nstat-a, run tcpdump to capture an ACK:: 1755 1756 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 1757 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1758 1759On nstat-b, send a packet via the port 9001 socket. E.g. we sent a 1760string 'foo' in our example:: 1761 1762 nstatuser@nstat-b:~$ nc -lkv 9001 1763 Listening on [0.0.0.0] (family 0, port 9001) 1764 Connection from nstat-a 42132 received! 1765 foo 1766 1767On nstat-a, the tcpdump should have caputred the ACK. We should check 1768the source port numbers of the two nc clients:: 1769 1770 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee 1771 State Recv-Q Send-Q Local Address:Port Peer Address:Port 1772 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 1773 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001 1774 1775Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to 1776port 50208:: 1777 1778 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum 1779 1780Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:: 1781 1782 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done 1783 1784Check TcpExtTCPACKSkippedSeq on nstat-b:: 1785 1786 nstatuser@nstat-b:~$ nstat | grep -i skip 1787 TcpExtTCPACKSkippedSeq 1 0.0 1788