1=========== 2SNMP counter 3=========== 4 5This document explains the meaning of SNMP counters. 6 7General IPv4 counters 8==================== 9All layer 4 packets and ICMP packets will change these counters, but 10these counters won't be changed by layer 2 packets (such as STP) or 11ARP packets. 12 13* IpInReceives 14Defined in `RFC1213 ipInReceives`_ 15 16.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26 17 18The number of packets received by the IP layer. It gets increasing at the 19beginning of ip_rcv function, always be updated together with 20IpExtInOctets. It will be increased even if the packet is dropped 21later (e.g. due to the IP header is invalid or the checksum is wrong 22and so on). It indicates the number of aggregated segments after 23GRO/LRO. 24 25* IpInDelivers 26Defined in `RFC1213 ipInDelivers`_ 27 28.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28 29 30The number of packets delivers to the upper layer protocols. E.g. TCP, UDP, 31ICMP and so on. If no one listens on a raw socket, only kernel 32supported protocols will be delivered, if someone listens on the raw 33socket, all valid IP packets will be delivered. 34 35* IpOutRequests 36Defined in `RFC1213 ipOutRequests`_ 37 38.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28 39 40The number of packets sent via IP layer, for both single cast and 41multicast packets, and would always be updated together with 42IpExtOutOctets. 43 44* IpExtInOctets and IpExtOutOctets 45They are Linux kernel extensions, no RFC definitions. Please note, 46RFC1213 indeed defines ifInOctets and ifOutOctets, but they 47are different things. The ifInOctets and ifOutOctets include the MAC 48layer header size but IpExtInOctets and IpExtOutOctets don't, they 49only include the IP layer header and the IP layer data. 50 51* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts 52They indicate the number of four kinds of ECN IP packets, please refer 53`Explicit Congestion Notification`_ for more details. 54 55.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6 56 57These 4 counters calculate how many packets received per ECN 58status. They count the real frame number regardless the LRO/GRO. So 59for the same packet, you might find that IpInReceives count 1, but 60IpExtInNoECTPkts counts 2 or more. 61 62* IpInHdrErrors 63Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is 64dropped due to the IP header error. It might happen in both IP input 65and IP forward paths. 66 67.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27 68 69* IpInAddrErrors 70Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two 71scenarios: (1) The IP address is invalid. (2) The destination IP 72address is not a local address and IP forwarding is not enabled 73 74.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27 75 76* IpExtInNoRoutes 77This counter means the packet is dropped when the IP stack receives a 78packet and can't find a route for it from the route table. It might 79happen when IP forwarding is enabled and the destination IP address is 80not a local address and there is no route for the destination IP 81address. 82 83* IpInUnknownProtos 84Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the 85layer 4 protocol is unsupported by kernel. If an application is using 86raw socket, kernel will always deliver the packet to the raw socket 87and this counter won't be increased. 88 89.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27 90 91* IpExtInTruncatedPkts 92For IPv4 packet, it means the actual data size is smaller than the 93"Total Length" field in the IPv4 header. 94 95* IpInDiscards 96Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped 97in the IP receiving path and due to kernel internal reasons (e.g. no 98enough memory). 99 100.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28 101 102* IpOutDiscards 103Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is 104dropped in the IP sending path and due to kernel internal reasons. 105 106.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28 107 108* IpOutNoRoutes 109Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is 110dropped in the IP sending path and no route is found for it. 111 112.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29 113 114ICMP counters 115============ 116* IcmpInMsgs and IcmpOutMsgs 117Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_ 118 119.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41 120.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43 121 122As mentioned in the RFC1213, these two counters include errors, they 123would be increased even if the ICMP packet has an invalid type. The 124ICMP output path will check the header of a raw socket, so the 125IcmpOutMsgs would still be updated if the IP header is constructed by 126a userspace program. 127 128* ICMP named types 129| These counters include most of common ICMP types, they are: 130| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_ 131| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ 132| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ 133| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_ 134| IcmpInRedirects: `RFC1213 icmpInRedirects`_ 135| IcmpInEchos: `RFC1213 icmpInEchos`_ 136| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ 137| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_ 138| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_ 139| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ 140| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_ 141| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_ 142| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_ 143| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_ 144| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_ 145| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_ 146| IcmpOutEchos: `RFC1213 icmpOutEchos`_ 147| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ 148| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_ 149| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_ 150| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_ 151| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_ 152 153.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41 154.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41 155.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42 156.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42 157.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42 158.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42 159.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42 160.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42 161.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43 162.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43 163.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43 164 165.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44 166.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44 167.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44 168.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44 169.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44 170.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45 171.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45 172.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45 173.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45 174.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45 175.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46 176 177Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP 178Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are 179straightforward. The 'In' counter means kernel receives such a packet 180and the 'Out' counter means kernel sends such a packet. 181 182* ICMP numeric types 183They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the 184ICMP type number. These counters track all kinds of ICMP packets. The 185ICMP type number definition could be found in the `ICMP parameters`_ 186document. 187 188.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml 189 190For example, if the Linux kernel sends an ICMP Echo packet, the 191IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply 192packet, IcmpMsgInType0 would increase 1. 193 194* IcmpInCsumErrors 195This counter indicates the checksum of the ICMP packet is 196wrong. Kernel verifies the checksum after updating the IcmpInMsgs and 197before updating IcmpMsgInType[N]. If a packet has bad checksum, the 198IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated. 199 200* IcmpInErrors and IcmpOutErrors 201Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_ 202 203.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41 204.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43 205 206When an error occurs in the ICMP packet handler path, these two 207counters would be updated. The receiving packet path use IcmpInErrors 208and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors 209is increased, IcmpInErrors would always be increased too. 210 211relationship of the ICMP counters 212------------------------------- 213The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they 214are updated at the same time. The sum of IcmpMsgInType[N] plus 215IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel 216receives an ICMP packet, kernel follows below logic: 217 2181. increase IcmpInMsgs 2192. if has any error, update IcmpInErrors and finish the process 2203. update IcmpMsgOutType[N] 2214. handle the packet depending on the type, if has any error, update 222 IcmpInErrors and finish the process 223 224So if all errors occur in step (2), IcmpInMsgs should be equal to the 225sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in 226step (4), IcmpInMsgs should be equal to the sum of 227IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4), 228IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus 229IcmpInErrors. 230 231General TCP counters 232================== 233* TcpInSegs 234Defined in `RFC1213 tcpInSegs`_ 235 236.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48 237 238The number of packets received by the TCP layer. As mentioned in 239RFC1213, it includes the packets received in error, such as checksum 240error, invalid TCP header and so on. Only one error won't be included: 241if the layer 2 destination address is not the NIC's layer 2 242address. It might happen if the packet is a multicast or broadcast 243packet, or the NIC is in promiscuous mode. In these situations, the 244packets would be delivered to the TCP layer, but the TCP layer will discard 245these packets before increasing TcpInSegs. The TcpInSegs counter 246isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs 247counter would only increase 1. 248 249* TcpOutSegs 250Defined in `RFC1213 tcpOutSegs`_ 251 252.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48 253 254The number of packets sent by the TCP layer. As mentioned in RFC1213, 255it excludes the retransmitted packets. But it includes the SYN, ACK 256and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of 257GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will 258increase 2. 259 260* TcpActiveOpens 261Defined in `RFC1213 tcpActiveOpens`_ 262 263.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47 264 265It means the TCP layer sends a SYN, and come into the SYN-SENT 266state. Every time TcpActiveOpens increases 1, TcpOutSegs should always 267increase 1. 268 269* TcpPassiveOpens 270Defined in `RFC1213 tcpPassiveOpens`_ 271 272.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47 273 274It means the TCP layer receives a SYN, replies a SYN+ACK, come into 275the SYN-RCVD state. 276 277* TcpExtTCPRcvCoalesce 278When packets are received by the TCP layer and are not be read by the 279application, the TCP layer will try to merge them. This counter 280indicate how many packets are merged in such situation. If GRO is 281enabled, lots of packets would be merged by GRO, these packets 282wouldn't be counted to TcpExtTCPRcvCoalesce. 283 284* TcpExtTCPAutoCorking 285When sending packets, the TCP layer will try to merge small packets to 286a bigger one. This counter increase 1 for every packet merged in such 287situation. Please refer to the LWN article for more details: 288https://lwn.net/Articles/576263/ 289 290* TcpExtTCPOrigDataSent 291This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 292explaination below:: 293 294 TCPOrigDataSent: number of outgoing packets with original data (excluding 295 retransmission but including data-in-SYN). This counter is different from 296 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is 297 more useful to track the TCP retransmission rate. 298 299* TCPSynRetrans 300This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 301explaination below:: 302 303 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down 304 retransmissions into SYN, fast-retransmits, timeout retransmits, etc. 305 306* TCPFastOpenActiveFail 307This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 308explaination below:: 309 310 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because 311 the remote does not accept it or the attempts timed out. 312 313.. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd 314 315* TcpExtListenOverflows and TcpExtListenDrops 316When kernel receives a SYN from a client, and if the TCP accept queue 317is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. 318At the same time kernel will also add 1 to TcpExtListenDrops. When a 319TCP socket is in LISTEN state, and kernel need to drop a packet, 320kernel would always add 1 to TcpExtListenDrops. So increase 321TcpExtListenOverflows would let TcpExtListenDrops increasing at the 322same time, but TcpExtListenDrops would also increase without 323TcpExtListenOverflows increasing, e.g. a memory allocation fail would 324also let TcpExtListenDrops increase. 325 326Note: The above explanation is based on kernel 4.10 or above version, on 327an old kernel, the TCP stack has different behavior when TCP accept 328queue is full. On the old kernel, TCP stack won't drop the SYN, it 329would complete the 3-way handshake. As the accept queue is full, TCP 330stack will keep the socket in the TCP half-open queue. As it is in the 331half open queue, TCP stack will send SYN+ACK on an exponential backoff 332timer, after client replies ACK, TCP stack checks whether the accept 333queue is still full, if it is not full, moves the socket to the accept 334queue, if it is full, keeps the socket in the half-open queue, at next 335time client replies ACK, this socket will get another chance to move 336to the accept queue. 337 338 339TCP Fast Open 340============ 341When kernel receives a TCP packet, it has two paths to handler the 342packet, one is fast path, another is slow path. The comment in kernel 343code provides a good explanation of them, I pasted them below:: 344 345 It is split into a fast path and a slow path. The fast path is 346 disabled when: 347 348 - A zero window was announced from us 349 - zero window probing 350 is only handled properly on the slow path. 351 - Out of order segments arrived. 352 - Urgent data is expected. 353 - There is no buffer space left 354 - Unexpected TCP flags/window values/header lengths are received 355 (detected by checking the TCP header against pred_flags) 356 - Data is sent in both directions. The fast path only supports pure senders 357 or pure receivers (this means either the sequence number or the ack 358 value must stay constant) 359 - Unexpected TCP option. 360 361Kernel will try to use fast path unless any of the above conditions 362are satisfied. If the packets are out of order, kernel will handle 363them in slow path, which means the performance might be not very 364good. Kernel would also come into slow path if the "Delayed ack" is 365used, because when using "Delayed ack", the data is sent in both 366directions. When the TCP window scale option is not used, kernel will 367try to enable fast path immediately when the connection comes into the 368established state, but if the TCP window scale option is used, kernel 369will disable the fast path at first, and try to enable it after kernel 370receives packets. 371 372* TcpExtTCPPureAcks and TcpExtTCPHPAcks 373If a packet set ACK flag and has no data, it is a pure ACK packet, if 374kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, 375if kernel handles it in the slow path, TcpExtTCPPureAcks will 376increase 1. 377 378* TcpExtTCPHPHits 379If a TCP packet has data (which means it is not a pure ACK packet), 380and this packet is handled in the fast path, TcpExtTCPHPHits will 381increase 1. 382 383 384TCP abort 385======== 386 387 388* TcpExtTCPAbortOnData 389It means TCP layer has data in flight, but need to close the 390connection. So TCP layer sends a RST to the other side, indicate the 391connection is not closed very graceful. An easy way to increase this 392counter is using the SO_LINGER option. Please refer to the SO_LINGER 393section of the `socket man page`_: 394 395.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html 396 397By default, when an application closes a connection, the close function 398will return immediately and kernel will try to send the in-flight data 399async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger 400to a positive number, the close function won't return immediately, but 401wait for the in-flight data are acked by the other side, the max wait 402time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, 403when the application closes a connection, kernel will send a RST 404immediately and increase the TcpExtTCPAbortOnData counter. 405 406* TcpExtTCPAbortOnClose 407This counter means the application has unread data in the TCP layer when 408the application wants to close the TCP connection. In such a situation, 409kernel will send a RST to the other side of the TCP connection. 410 411* TcpExtTCPAbortOnMemory 412When an application closes a TCP connection, kernel still need to track 413the connection, let it complete the TCP disconnect process. E.g. an 414app calls the close method of a socket, kernel sends fin to the other 415side of the connection, then the app has no relationship with the 416socket any more, but kernel need to keep the socket, this socket 417becomes an orphan socket, kernel waits for the reply of the other side, 418and would come to the TIME_WAIT state finally. When kernel has no 419enough memory to keep the orphan socket, kernel would send an RST to 420the other side, and delete the socket, in such situation, kernel will 421increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger 422TcpExtTCPAbortOnMemory: 423 4241. the memory used by the TCP protocol is higher than the third value of 425the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_: 426 427.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html 428 4292. the orphan socket count is higher than net.ipv4.tcp_max_orphans 430 431 432* TcpExtTCPAbortOnTimeout 433This counter will increase when any of the TCP timers expire. In such 434situation, kernel won't send RST, just give up the connection. 435 436* TcpExtTCPAbortOnLinger 437When a TCP connection comes into FIN_WAIT_2 state, instead of waiting 438for the fin packet from the other side, kernel could send a RST and 439delete the socket immediately. This is not the default behavior of 440Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, 441you could let kernel follow this behavior. 442 443* TcpExtTCPAbortFailed 444The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is 445satisfied. If an internal error occurs during this process, 446TcpExtTCPAbortFailed will be increased. 447 448.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50 449 450TCP Hybrid Slow Start 451==================== 452The Hybrid Slow Start algorithm is an enhancement of the traditional 453TCP congestion window Slow Start algorithm. It uses two pieces of 454information to detect whether the max bandwidth of the TCP path is 455approached. The two pieces of information are ACK train length and 456increase in packet delay. For detail information, please refer the 457`Hybrid Slow Start paper`_. Either ACK train length or packet delay 458hits a specific threshold, the congestion control algorithm will come 459into the Congestion Avoidance state. Until v4.20, two congestion 460control algorithms are using Hybrid Slow Start, they are cubic (the 461default congestion control algorithm) and cdg. Four snmp counters 462relate with the Hybrid Slow Start algorithm. 463 464.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf 465 466* TcpExtTCPHystartTrainDetect 467How many times the ACK train length threshold is detected 468 469* TcpExtTCPHystartTrainCwnd 470The sum of CWND detected by ACK train length. Dividing this value by 471TcpExtTCPHystartTrainDetect is the average CWND which detected by the 472ACK train length. 473 474* TcpExtTCPHystartDelayDetect 475How many times the packet delay threshold is detected. 476 477* TcpExtTCPHystartDelayCwnd 478The sum of CWND detected by packet delay. Dividing this value by 479TcpExtTCPHystartDelayDetect is the average CWND which detected by the 480packet delay. 481 482TCP retransmission and congestion control 483====================================== 484The TCP protocol has two retransmission mechanisms: SACK and fast 485recovery. They are exclusive with each other. When SACK is enabled, 486the kernel TCP stack would use SACK, or kernel would use fast 487recovery. The SACK is a TCP option, which is defined in `RFC2018`_, 488the fast recovery is defined in `RFC6582`_, which is also called 489'Reno'. 490 491The TCP congestion control is a big and complex topic. To understand 492the related snmp counter, we need to know the states of the congestion 493control state machine. There are 5 states: Open, Disorder, CWR, 494Recovery and Loss. For details about these states, please refer page 5 495and page 6 of this document: 496https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf 497 498.. _RFC2018: https://tools.ietf.org/html/rfc2018 499.. _RFC6582: https://tools.ietf.org/html/rfc6582 500 501* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery 502When the congestion control comes into Recovery state, if sack is 503used, TcpExtTCPSackRecovery increases 1, if sack is not used, 504TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP 505stack begins to retransmit the lost packets. 506 507* TcpExtTCPSACKReneging 508A packet was acknowledged by SACK, but the receiver has dropped this 509packet, so the sender needs to retransmit this packet. In this 510situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver 511could drop a packet which has been acknowledged by SACK, although it is 512unusual, it is allowed by the TCP protocol. The sender doesn't really 513know what happened on the receiver side. The sender just waits until 514the RTO expires for this packet, then the sender assumes this packet 515has been dropped by the receiver. 516 517* TcpExtTCPRenoReorder 518The reorder packet is detected by fast recovery. It would only be used 519if SACK is disabled. The fast recovery algorithm detects recorder by 520the duplicate ACK number. E.g., if retransmission is triggered, and 521the original retransmitted packet is not lost, it is just out of 522order, the receiver would acknowledge multiple times, one for the 523retransmitted packet, another for the arriving of the original out of 524order packet. Thus the sender would find more ACks than its 525expectation, and the sender knows out of order occurs. 526 527* TcpExtTCPTSReorder 528The reorder packet is detected when a hole is filled. E.g., assume the 529sender sends packet 1,2,3,4,5, and the receiving order is 5301,2,4,5,3. When the sender receives the ACK of packet 3 (which will 531fill the hole), two conditions will let TcpExtTCPTSReorder increase 5321: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 5333 is retransmitted but the timestamp of the packet 3's ACK is earlier 534than the retransmission timestamp. 535 536* TcpExtTCPSACKReorder 537The reorder packet detected by SACK. The SACK has two methods to 538detect reorder: (1) DSACK is received by the sender. It means the 539sender sends the same packet more than one times. And the only reason 540is the sender believes an out of order packet is lost so it sends the 541packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and 542the sender has received SACKs for packet 2 and 5, now the sender 543receives SACK for packet 4 and the sender doesn't retransmit the 544packet yet, the sender would know packet 4 is out of order. The TCP 545stack of kernel will increase TcpExtTCPSACKReorder for both of the 546above scenarios. 547 548 549DSACK 550===== 551The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report 552duplicate packets to the sender. There are two kinds of 553duplications: (1) a packet which has been acknowledged is 554duplicate. (2) an out of order packet is duplicate. The TCP stack 555counts these two kinds of duplications on both receiver side and 556sender side. 557 558.. _RFC2883 : https://tools.ietf.org/html/rfc2883 559 560* TcpExtTCPDSACKOldSent 561The TCP stack receives a duplicate packet which has been acked, so it 562sends a DSACK to the sender. 563 564* TcpExtTCPDSACKOfoSent 565The TCP stack receives an out of order duplicate packet, so it sends a 566DSACK to the sender. 567 568* TcpExtTCPDSACKRecv 569The TCP stack receives a DSACK, which indicate an acknowledged 570duplicate packet is received. 571 572* TcpExtTCPDSACKOfoRecv 573The TCP stack receives a DSACK, which indicate an out of order 574duplicate packet is received. 575 576TCP out of order 577=============== 578* TcpExtTCPOFOQueue 579The TCP layer receives an out of order packet and has enough memory 580to queue it. 581 582* TcpExtTCPOFODrop 583The TCP layer receives an out of order packet but doesn't have enough 584memory, so drops it. Such packets won't be counted into 585TcpExtTCPOFOQueue. 586 587* TcpExtTCPOFOMerge 588The received out of order packet has an overlay with the previous 589packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge 590packets will also be counted into TcpExtTCPOFOQueue. 591 592TCP PAWS 593======= 594PAWS (Protection Against Wrapped Sequence numbers) is an algorithm 595which is used to drop old packets. It depends on the TCP 596timestamps. For detail information, please refer the `timestamp wiki`_ 597and the `RFC of PAWS`_. 598 599.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17 600.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps 601 602* TcpExtPAWSActive 603Packets are dropped by PAWS in Syn-Sent status. 604 605* TcpExtPAWSEstab 606Packets are dropped by PAWS in any status other than Syn-Sent. 607 608TCP ACK skip 609=========== 610In some scenarios, kernel would avoid sending duplicate ACKs too 611frequently. Please find more details in the tcp_invalid_ratelimit 612section of the `sysctl document`_. When kernel decides to skip an ACK 613due to tcp_invalid_ratelimit, kernel would update one of below 614counters to indicate the ACK is skipped in which scenario. The ACK 615would only be skipped if the received packet is either a SYN packet or 616it has no data. 617 618.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt 619 620* TcpExtTCPACKSkippedSynRecv 621The ACK is skipped in Syn-Recv status. The Syn-Recv status means the 622TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is 623waiting for an ACK. Generally, the TCP stack doesn't need to send ACK 624in the Syn-Recv status. But in several scenarios, the TCP stack need 625to send an ACK. E.g., the TCP stack receives the same SYN packet 626repeately, the received packet does not pass the PAWS check, or the 627received packet sequence number is out of window. In these scenarios, 628the TCP stack needs to send ACK. If the ACk sending frequency is higher than 629tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and 630increase TcpExtTCPACKSkippedSynRecv. 631 632 633* TcpExtTCPACKSkippedPAWS 634The ACK is skipped due to PAWS (Protect Against Wrapped Sequence 635numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 636or Time-Wait statuses, the skipped ACK would be counted to 637TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or 638TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK 639would be counted to TcpExtTCPACKSkippedPAWS. 640 641* TcpExtTCPACKSkippedSeq 642The sequence number is out of window and the timestamp passes the PAWS 643check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait. 644 645* TcpExtTCPACKSkippedFinWait2 646The ACK is skipped in Fin-Wait-2 status, the reason would be either 647PAWS check fails or the received sequence number is out of window. 648 649* TcpExtTCPACKSkippedTimeWait 650Tha ACK is skipped in Time-Wait status, the reason would be either 651PAWS check failed or the received sequence number is out of window. 652 653* TcpExtTCPACKSkippedChallenge 654The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 6553 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, 656`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these 657three scenarios, In some TCP status, the linux TCP stack would also 658send challenge ACKs if the ACK number is before the first 659unacknowledged number (more strict than `RFC 5961 section 5.2`_). 660 661.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7 662.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9 663.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11 664 665 666examples 667======= 668 669ping test 670-------- 671Run the ping command against the public dns server 8.8.8.8:: 672 673 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 674 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 675 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms 676 677 --- 8.8.8.8 ping statistics --- 678 1 packets transmitted, 1 received, 0% packet loss, time 0ms 679 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms 680 681The nstayt result:: 682 683 nstatuser@nstat-a:~$ nstat 684 #kernel 685 IpInReceives 1 0.0 686 IpInDelivers 1 0.0 687 IpOutRequests 1 0.0 688 IcmpInMsgs 1 0.0 689 IcmpInEchoReps 1 0.0 690 IcmpOutMsgs 1 0.0 691 IcmpOutEchos 1 0.0 692 IcmpMsgInType0 1 0.0 693 IcmpMsgOutType8 1 0.0 694 IpExtInOctets 84 0.0 695 IpExtOutOctets 84 0.0 696 IpExtInNoECTPkts 1 0.0 697 698The Linux server sent an ICMP Echo packet, so IpOutRequests, 699IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The 700server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs, 701IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply 702was passed to the ICMP layer via IP layer, so IpInDelivers was 703increased 1. The default ping data size is 48, so an ICMP Echo packet 704and its corresponding Echo Reply packet are constructed by: 705 706* 14 bytes MAC header 707* 20 bytes IP header 708* 16 bytes ICMP header 709* 48 bytes data (default value of the ping command) 710 711So the IpExtInOctets and IpExtOutOctets are 20+16+48=84. 712 713tcp 3-way handshake 714------------------ 715On server side, we run:: 716 717 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 718 Listening on [0.0.0.0] (family 0, port 9000) 719 720On client side, we run:: 721 722 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000 723 Connection to 192.168.122.251 9000 port [tcp/*] succeeded! 724 725The server listened on tcp 9000 port, the client connected to it, they 726completed the 3-way handshake. 727 728On server side, we can find below nstat output:: 729 730 nstatuser@nstat-b:~$ nstat | grep -i tcp 731 TcpPassiveOpens 1 0.0 732 TcpInSegs 2 0.0 733 TcpOutSegs 1 0.0 734 TcpExtTCPPureAcks 1 0.0 735 736On client side, we can find below nstat output:: 737 738 nstatuser@nstat-a:~$ nstat | grep -i tcp 739 TcpActiveOpens 1 0.0 740 TcpInSegs 1 0.0 741 TcpOutSegs 2 0.0 742 743When the server received the first SYN, it replied a SYN+ACK, and came into 744SYN-RCVD state, so TcpPassiveOpens increased 1. The server received 745SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 746packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK 747of the 3-way handshake is a pure ACK without data, so 748TcpExtTCPPureAcks increased 1. 749 750When the client sent SYN, the client came into the SYN-SENT state, so 751TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent 752ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 7531, TcpOutSegs increased 2. 754 755TCP normal traffic 756----------------- 757Run nc on server:: 758 759 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 760 Listening on [0.0.0.0] (family 0, port 9000) 761 762Run nc on client:: 763 764 nstatuser@nstat-a:~$ nc -v nstat-b 9000 765 Connection to nstat-b 9000 port [tcp/*] succeeded! 766 767Input a string in the nc client ('hello' in our example):: 768 769 nstatuser@nstat-a:~$ nc -v nstat-b 9000 770 Connection to nstat-b 9000 port [tcp/*] succeeded! 771 hello 772 773The client side nstat output:: 774 775 nstatuser@nstat-a:~$ nstat 776 #kernel 777 IpInReceives 1 0.0 778 IpInDelivers 1 0.0 779 IpOutRequests 1 0.0 780 TcpInSegs 1 0.0 781 TcpOutSegs 1 0.0 782 TcpExtTCPPureAcks 1 0.0 783 TcpExtTCPOrigDataSent 1 0.0 784 IpExtInOctets 52 0.0 785 IpExtOutOctets 58 0.0 786 IpExtInNoECTPkts 1 0.0 787 788The server side nstat output:: 789 790 nstatuser@nstat-b:~$ nstat 791 #kernel 792 IpInReceives 1 0.0 793 IpInDelivers 1 0.0 794 IpOutRequests 1 0.0 795 TcpInSegs 1 0.0 796 TcpOutSegs 1 0.0 797 IpExtInOctets 58 0.0 798 IpExtOutOctets 52 0.0 799 IpExtInNoECTPkts 1 0.0 800 801Input a string in nc client side again ('world' in our exmaple):: 802 803 nstatuser@nstat-a:~$ nc -v nstat-b 9000 804 Connection to nstat-b 9000 port [tcp/*] succeeded! 805 hello 806 world 807 808Client side nstat output:: 809 810 nstatuser@nstat-a:~$ nstat 811 #kernel 812 IpInReceives 1 0.0 813 IpInDelivers 1 0.0 814 IpOutRequests 1 0.0 815 TcpInSegs 1 0.0 816 TcpOutSegs 1 0.0 817 TcpExtTCPHPAcks 1 0.0 818 TcpExtTCPOrigDataSent 1 0.0 819 IpExtInOctets 52 0.0 820 IpExtOutOctets 58 0.0 821 IpExtInNoECTPkts 1 0.0 822 823 824Server side nstat output:: 825 826 nstatuser@nstat-b:~$ nstat 827 #kernel 828 IpInReceives 1 0.0 829 IpInDelivers 1 0.0 830 IpOutRequests 1 0.0 831 TcpInSegs 1 0.0 832 TcpOutSegs 1 0.0 833 TcpExtTCPHPHits 1 0.0 834 IpExtInOctets 58 0.0 835 IpExtOutOctets 52 0.0 836 IpExtInNoECTPkts 1 0.0 837 838Compare the first client-side nstat and the second client-side nstat, 839we could find one difference: the first one had a 'TcpExtTCPPureAcks', 840but the second one had a 'TcpExtTCPHPAcks'. The first server-side 841nstat and the second server-side nstat had a difference too: the 842second server-side nstat had a TcpExtTCPHPHits, but the first 843server-side nstat didn't have it. The network traffic patterns were 844exactly the same: the client sent a packet to the server, the server 845replied an ACK. But kernel handled them in different ways. When the 846TCP window scale option is not used, kernel will try to enable fast 847path immediately when the connection comes into the established state, 848but if the TCP window scale option is used, kernel will disable the 849fast path at first, and try to enable it after kerenl receives 850packets. We could use the 'ss' command to verify whether the window 851scale option is used. e.g. run below command on either server or 852client:: 853 854 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) 855 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port 856 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 857 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98 858 859The 'wscale:7,7' means both server and client set the window scale 860option to 7. Now we could explain the nstat output in our test: 861 862In the first nstat output of client side, the client sent a packet, server 863reply an ACK, when kernel handled this ACK, the fast path was not 864enabled, so the ACK was counted into 'TcpExtTCPPureAcks'. 865 866In the second nstat output of client side, the client sent a packet again, 867and received another ACK from the server, in this time, the fast path is 868enabled, and the ACK was qualified for fast path, so it was handled by 869the fast path, so this ACK was counted into TcpExtTCPHPAcks. 870 871In the first nstat output of server side, fast path was not enabled, 872so there was no 'TcpExtTCPHPHits'. 873 874In the second nstat output of server side, the fast path was enabled, 875and the packet received from client qualified for fast path, so it 876was counted into 'TcpExtTCPHPHits'. 877 878TcpExtTCPAbortOnClose 879-------------------- 880On the server side, we run below python script:: 881 882 import socket 883 import time 884 885 port = 9000 886 887 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 888 s.bind(('0.0.0.0', port)) 889 s.listen(1) 890 sock, addr = s.accept() 891 while True: 892 time.sleep(9999999) 893 894This python script listen on 9000 port, but doesn't read anything from 895the connection. 896 897On the client side, we send the string "hello" by nc:: 898 899 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000 900 901Then, we come back to the server side, the server has received the "hello" 902packet, and the TCP layer has acked this packet, but the application didn't 903read it yet. We type Ctrl-C to terminate the server script. Then we 904could find TcpExtTCPAbortOnClose increased 1 on the server side:: 905 906 nstatuser@nstat-b:~$ nstat | grep -i abort 907 TcpExtTCPAbortOnClose 1 0.0 908 909If we run tcpdump on the server side, we could find the server sent a 910RST after we type Ctrl-C. 911 912TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout 913----------------------------------------------- 914Below is an example which let the orphan socket count be higher than 915net.ipv4.tcp_max_orphans. 916Change tcp_max_orphans to a smaller value on client:: 917 918 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans" 919 920Client code (create 64 connection to server):: 921 922 nstatuser@nstat-a:~$ cat client_orphan.py 923 import socket 924 import time 925 926 server = 'nstat-b' # server address 927 port = 9000 928 929 count = 64 930 931 connection_list = [] 932 933 for i in range(64): 934 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 935 s.connect((server, port)) 936 connection_list.append(s) 937 print("connection_count: %d" % len(connection_list)) 938 939 while True: 940 time.sleep(99999) 941 942Server code (accept 64 connection from client):: 943 944 nstatuser@nstat-b:~$ cat server_orphan.py 945 import socket 946 import time 947 948 port = 9000 949 count = 64 950 951 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 952 s.bind(('0.0.0.0', port)) 953 s.listen(count) 954 connection_list = [] 955 while True: 956 sock, addr = s.accept() 957 connection_list.append((sock, addr)) 958 print("connection_count: %d" % len(connection_list)) 959 960Run the python scripts on server and client. 961 962On server:: 963 964 python3 server_orphan.py 965 966On client:: 967 968 python3 client_orphan.py 969 970Run iptables on server:: 971 972 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP 973 974Type Ctrl-C on client, stop client_orphan.py. 975 976Check TcpExtTCPAbortOnMemory on client:: 977 978 nstatuser@nstat-a:~$ nstat | grep -i abort 979 TcpExtTCPAbortOnMemory 54 0.0 980 981Check orphane socket count on client:: 982 983 nstatuser@nstat-a:~$ ss -s 984 Total: 131 (kernel 0) 985 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 986 987 Transport Total IP IPv6 988 * 0 - - 989 RAW 1 0 1 990 UDP 1 1 0 991 TCP 14 13 1 992 INET 16 14 2 993 FRAG 0 0 0 994 995The explanation of the test: after run server_orphan.py and 996client_orphan.py, we set up 64 connections between server and 997client. Run the iptables command, the server will drop all packets from 998the client, type Ctrl-C on client_orphan.py, the system of the client 999would try to close these connections, and before they are closed 1000gracefully, these connections became orphan sockets. As the iptables 1001of the server blocked packets from the client, the server won't receive fin 1002from the client, so all connection on clients would be stuck on FIN_WAIT_1 1003stage, so they will keep as orphan sockets until timeout. We have echo 100410 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would 1005only keep 10 orphan sockets, for all other orphan sockets, the client 1006system sent RST for them and delete them. We have 64 connections, so 1007the 'ss -s' command shows the system has 10 orphan sockets, and the 1008value of TcpExtTCPAbortOnMemory was 54. 1009 1010An additional explanation about orphan socket count: You could find the 1011exactly orphan socket count by the 'ss -s' command, but when kernel 1012decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel 1013doesn't always check the exactly orphan socket count. For increasing 1014performance, kernel checks an approximate count firstly, if the 1015approximate count is more than tcp_max_orphans, kernel checks the 1016exact count again. So if the approximate count is less than 1017tcp_max_orphans, but exactly count is more than tcp_max_orphans, you 1018would find TcpExtTCPAbortOnMemory is not increased at all. If 1019tcp_max_orphans is large enough, it won't occur, but if you decrease 1020tcp_max_orphans to a small value like our test, you might find this 1021issue. So in our test, the client set up 64 connections although the 1022tcp_max_orphans is 10. If the client only set up 11 connections, we 1023can't find the change of TcpExtTCPAbortOnMemory. 1024 1025Continue the previous test, we wait for several minutes. Because of the 1026iptables on the server blocked the traffic, the server wouldn't receive 1027fin, and all the client's orphan sockets would timeout on the 1028FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 102910 timeout on the client:: 1030 1031 nstatuser@nstat-a:~$ nstat | grep -i abort 1032 TcpExtTCPAbortOnTimeout 10 0.0 1033 1034TcpExtTCPAbortOnLinger 1035--------------------- 1036The server side code:: 1037 1038 nstatuser@nstat-b:~$ cat server_linger.py 1039 import socket 1040 import time 1041 1042 port = 9000 1043 1044 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1045 s.bind(('0.0.0.0', port)) 1046 s.listen(1) 1047 sock, addr = s.accept() 1048 while True: 1049 time.sleep(9999999) 1050 1051The client side code:: 1052 1053 nstatuser@nstat-a:~$ cat client_linger.py 1054 import socket 1055 import struct 1056 1057 server = 'nstat-b' # server address 1058 port = 9000 1059 1060 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1061 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) 1062 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) 1063 s.connect((server, port)) 1064 s.close() 1065 1066Run server_linger.py on server:: 1067 1068 nstatuser@nstat-b:~$ python3 server_linger.py 1069 1070Run client_linger.py on client:: 1071 1072 nstatuser@nstat-a:~$ python3 client_linger.py 1073 1074After run client_linger.py, check the output of nstat:: 1075 1076 nstatuser@nstat-a:~$ nstat | grep -i abort 1077 TcpExtTCPAbortOnLinger 1 0.0 1078 1079TcpExtTCPRcvCoalesce 1080------------------- 1081On the server, we run a program which listen on TCP port 9000, but 1082doesn't read any data:: 1083 1084 import socket 1085 import time 1086 port = 9000 1087 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1088 s.bind(('0.0.0.0', port)) 1089 s.listen(1) 1090 sock, addr = s.accept() 1091 while True: 1092 time.sleep(9999999) 1093 1094Save the above code as server_coalesce.py, and run:: 1095 1096 python3 server_coalesce.py 1097 1098On the client, save below code as client_coalesce.py:: 1099 1100 import socket 1101 server = 'nstat-b' 1102 port = 9000 1103 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1104 s.connect((server, port)) 1105 1106Run:: 1107 1108 nstatuser@nstat-a:~$ python3 -i client_coalesce.py 1109 1110We use '-i' to come into the interactive mode, then a packet:: 1111 1112 >>> s.send(b'foo') 1113 3 1114 1115Send a packet again:: 1116 1117 >>> s.send(b'bar') 1118 3 1119 1120On the server, run nstat:: 1121 1122 ubuntu@nstat-b:~$ nstat 1123 #kernel 1124 IpInReceives 2 0.0 1125 IpInDelivers 2 0.0 1126 IpOutRequests 2 0.0 1127 TcpInSegs 2 0.0 1128 TcpOutSegs 2 0.0 1129 TcpExtTCPRcvCoalesce 1 0.0 1130 IpExtInOctets 110 0.0 1131 IpExtOutOctets 104 0.0 1132 IpExtInNoECTPkts 2 0.0 1133 1134The client sent two packets, server didn't read any data. When 1135the second packet arrived at server, the first packet was still in 1136the receiving queue. So the TCP layer merged the two packets, and we 1137could find the TcpExtTCPRcvCoalesce increased 1. 1138 1139TcpExtListenOverflows and TcpExtListenDrops 1140---------------------------------------- 1141On server, run the nc command, listen on port 9000:: 1142 1143 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1144 Listening on [0.0.0.0] (family 0, port 9000) 1145 1146On client, run 3 nc commands in different terminals:: 1147 1148 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1149 Connection to nstat-b 9000 port [tcp/*] succeeded! 1150 1151The nc command only accepts 1 connection, and the accept queue length 1152is 1. On current linux implementation, set queue length to n means the 1153actual queue length is n+1. Now we create 3 connections, 1 is accepted 1154by nc, 2 in accepted queue, so the accept queue is full. 1155 1156Before running the 4th nc, we clean the nstat history on the server:: 1157 1158 nstatuser@nstat-b:~$ nstat -n 1159 1160Run the 4th nc on the client:: 1161 1162 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1163 1164If the nc server is running on kernel 4.10 or higher version, you 1165won't see the "Connection to ... succeeded!" string, because kernel 1166will drop the SYN if the accept queue is full. If the nc client is running 1167on an old kernel, you would see that the connection is succeeded, 1168because kernel would complete the 3 way handshake and keep the socket 1169on half open queue. I did the test on kernel 4.15. Below is the nstat 1170on the server:: 1171 1172 nstatuser@nstat-b:~$ nstat 1173 #kernel 1174 IpInReceives 4 0.0 1175 IpInDelivers 4 0.0 1176 TcpInSegs 4 0.0 1177 TcpExtListenOverflows 4 0.0 1178 TcpExtListenDrops 4 0.0 1179 IpExtInOctets 240 0.0 1180 IpExtInNoECTPkts 4 0.0 1181 1182Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time 1183between the 4th nc and the nstat was longer, the value of 1184TcpExtListenOverflows and TcpExtListenDrops would be larger, because 1185the SYN of the 4th nc was dropped, the client was retrying. 1186 1187IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes 1188---------------------------------------------- 1189server A IP address: 192.168.122.250 1190server B IP address: 192.168.122.251 1191Prepare on server A, add a route to server B:: 1192 1193 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251 1194 1195Prepare on server B, disable send_redirects for all interfaces:: 1196 1197 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 1198 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 1199 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 1200 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0 1201 1202We want to let sever A send a packet to 8.8.8.8, and route the packet 1203to server B. When server B receives such packet, it might send a ICMP 1204Redirect message to server A, set send_redirects to 0 will disable 1205this behavior. 1206 1207First, generate InAddrErrors. On server B, we disable IP forwarding:: 1208 1209 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0 1210 1211On server A, we send packets to 8.8.8.8:: 1212 1213 $ nc -v 8.8.8.8 53 1214 1215On server B, we check the output of nstat:: 1216 1217 $ nstat 1218 #kernel 1219 IpInReceives 3 0.0 1220 IpInAddrErrors 3 0.0 1221 IpExtInOctets 180 0.0 1222 IpExtInNoECTPkts 3 0.0 1223 1224As we have let server A route 8.8.8.8 to server B, and we disabled IP 1225forwarding on server B, Server A sent packets to server B, then server B 1226dropped packets and increased IpInAddrErrors. As the nc command would 1227re-send the SYN packet if it didn't receive a SYN+ACK, we could find 1228multiple IpInAddrErrors. 1229 1230Second, generate IpExtInNoRoutes. On server B, we enable IP 1231forwarding:: 1232 1233 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1 1234 1235Check the route table of server B and remove the default route:: 1236 1237 $ ip route show 1238 default via 192.168.122.1 dev ens3 proto static 1239 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 1240 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static 1241 1242On server A, we contact 8.8.8.8 again:: 1243 1244 $ nc -v 8.8.8.8 53 1245 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable 1246 1247On server B, run nstat:: 1248 1249 $ nstat 1250 #kernel 1251 IpInReceives 1 0.0 1252 IpOutRequests 1 0.0 1253 IcmpOutMsgs 1 0.0 1254 IcmpOutDestUnreachs 1 0.0 1255 IcmpMsgOutType3 1 0.0 1256 IpExtInNoRoutes 1 0.0 1257 IpExtInOctets 60 0.0 1258 IpExtOutOctets 88 0.0 1259 IpExtInNoECTPkts 1 0.0 1260 1261We enabled IP forwarding on server B, when server B received a packet 1262which destination IP address is 8.8.8.8, server B will try to forward 1263this packet. We have deleted the default route, there was no route for 12648.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP 1265Destination Unreachable" message to server A. 1266 1267Third, generate IpOutNoRoutes. Run ping command on server B:: 1268 1269 $ ping -c 1 8.8.8.8 1270 connect: Network is unreachable 1271 1272Run nstat on server B:: 1273 1274 $ nstat 1275 #kernel 1276 IpOutNoRoutes 1 0.0 1277 1278We have deleted the default route on server B. Server B couldn't find 1279a route for the 8.8.8.8 IP address, so server B increased 1280IpOutNoRoutes. 1281 1282TcpExtTCPACKSkippedSynRecv 1283------------------------ 1284In this test, we send 3 same SYN packets from client to server. The 1285first SYN will let server create a socket, set it to Syn-Recv status, 1286and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK 1287again, and record the reply time (the duplicate ACK reply time). The 1288third SYN will let server check the previous duplicate ACK reply time, 1289and decide to skip the duplicate ACK, then increase the 1290TcpExtTCPACKSkippedSynRecv counter. 1291 1292Run tcpdump to capture a SYN packet:: 1293 1294 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000 1295 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1296 1297Open another terminal, run nc command:: 1298 1299 nstatuser@nstat-a:~$ nc nstat-b 9000 1300 1301As the nstat-b didn't listen on port 9000, it should reply a RST, and 1302the nc command exited immediately. It was enough for the tcpdump 1303command to capture a SYN packet. A linux server might use hardware 1304offload for the TCP checksum, so the checksum in the /tmp/syn.pcap 1305might be not correct. We call tcprewrite to fix it:: 1306 1307 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum 1308 1309On nstat-b, we run nc to listen on port 9000:: 1310 1311 nstatuser@nstat-b:~$ nc -lkv 9000 1312 Listening on [0.0.0.0] (family 0, port 9000) 1313 1314On nstat-a, we blocked the packet from port 9000, or nstat-a would send 1315RST to nstat-b:: 1316 1317 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP 1318 1319Send 3 SYN repeatly to nstat-b:: 1320 1321 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done 1322 1323Check snmp cunter on nstat-b:: 1324 1325 nstatuser@nstat-b:~$ nstat | grep -i skip 1326 TcpExtTCPACKSkippedSynRecv 1 0.0 1327 1328As we expected, TcpExtTCPACKSkippedSynRecv is 1. 1329 1330TcpExtTCPACKSkippedPAWS 1331---------------------- 1332To trigger PAWS, we could send an old SYN. 1333 1334On nstat-b, let nc listen on port 9000:: 1335 1336 nstatuser@nstat-b:~$ nc -lkv 9000 1337 Listening on [0.0.0.0] (family 0, port 9000) 1338 1339On nstat-a, run tcpdump to capture a SYN:: 1340 1341 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000 1342 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1343 1344On nstat-a, run nc as a client to connect nstat-b:: 1345 1346 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1347 Connection to nstat-b 9000 port [tcp/*] succeeded! 1348 1349Now the tcpdump has captured the SYN and exit. We should fix the 1350checksum:: 1351 1352 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum 1353 1354Send the SYN packet twice:: 1355 1356 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done 1357 1358On nstat-b, check the snmp counter:: 1359 1360 nstatuser@nstat-b:~$ nstat | grep -i skip 1361 TcpExtTCPACKSkippedPAWS 1 0.0 1362 1363We sent two SYN via tcpreplay, both of them would let PAWS check 1364failed, the nstat-b replied an ACK for the first SYN, skipped the ACK 1365for the second SYN, and updated TcpExtTCPACKSkippedPAWS. 1366 1367TcpExtTCPACKSkippedSeq 1368-------------------- 1369To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid 1370timestamp (to pass PAWS check) but the sequence number is out of 1371window. The linux TCP stack would avoid to skip if the packet has 1372data, so we need a pure ACK packet. To generate such a packet, we 1373could create two sockets: one on port 9000, another on port 9001. Then 1374we capture an ACK on port 9001, change the source/destination port 1375numbers to match the port 9000 socket. Then we could trigger 1376TcpExtTCPACKSkippedSeq via this packet. 1377 1378On nstat-b, open two terminals, run two nc commands to listen on both 1379port 9000 and port 9001:: 1380 1381 nstatuser@nstat-b:~$ nc -lkv 9000 1382 Listening on [0.0.0.0] (family 0, port 9000) 1383 1384 nstatuser@nstat-b:~$ nc -lkv 9001 1385 Listening on [0.0.0.0] (family 0, port 9001) 1386 1387On nstat-a, run two nc clients:: 1388 1389 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1390 Connection to nstat-b 9000 port [tcp/*] succeeded! 1391 1392 nstatuser@nstat-a:~$ nc -v nstat-b 9001 1393 Connection to nstat-b 9001 port [tcp/*] succeeded! 1394 1395On nstat-a, run tcpdump to capture an ACK:: 1396 1397 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 1398 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1399 1400On nstat-b, send a packet via the port 9001 socket. E.g. we sent a 1401string 'foo' in our example:: 1402 1403 nstatuser@nstat-b:~$ nc -lkv 9001 1404 Listening on [0.0.0.0] (family 0, port 9001) 1405 Connection from nstat-a 42132 received! 1406 foo 1407 1408On nstat-a, the tcpdump should have caputred the ACK. We should check 1409the source port numbers of the two nc clients:: 1410 1411 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee 1412 State Recv-Q Send-Q Local Address:Port Peer Address:Port 1413 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 1414 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001 1415 1416Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to 1417port 50208:: 1418 1419 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum 1420 1421Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:: 1422 1423 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done 1424 1425Check TcpExtTCPACKSkippedSeq on nstat-b:: 1426 1427 nstatuser@nstat-b:~$ nstat | grep -i skip 1428 TcpExtTCPACKSkippedSeq 1 0.0 1429