1.. SPDX-License-Identifier: GPL-2.0
2
3============
4Timestamping
5============
6
7
81. Control Interfaces
9=====================
10
11The interfaces for receiving network packages timestamps are:
12
13SO_TIMESTAMP
14  Generates a timestamp for each incoming packet in (not necessarily
15  monotonic) system time. Reports the timestamp via recvmsg() in a
16  control message in usec resolution.
17  SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD
18  based on the architecture type and time_t representation of libc.
19  Control message format is in struct __kernel_old_timeval for
20  SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for
21  SO_TIMESTAMP_NEW options respectively.
22
23SO_TIMESTAMPNS
24  Same timestamping mechanism as SO_TIMESTAMP, but reports the
25  timestamp as struct timespec in nsec resolution.
26  SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD
27  based on the architecture type and time_t representation of libc.
28  Control message format is in struct timespec for SO_TIMESTAMPNS_OLD
29  and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options
30  respectively.
31
32IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
33  Only for multicast:approximate transmit timestamp obtained by
34  reading the looped packet receive timestamp.
35
36SO_TIMESTAMPING
37  Generates timestamps on reception, transmission or both. Supports
38  multiple timestamp sources, including hardware. Supports generating
39  timestamps for stream sockets.
40
41
421.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW)
43-------------------------------------------------------------
44
45This socket option enables timestamping of datagrams on the reception
46path. Because the destination socket, if any, is not known early in
47the network stack, the feature has to be enabled for all packets. The
48same is true for all early receive timestamp options.
49
50For interface details, see `man 7 socket`.
51
52Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in
53struct __kernel_sock_timeval format.
54
55SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038
56on 32 bit machines.
57
581.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW)
59-------------------------------------------------------------------
60
61This option is identical to SO_TIMESTAMP except for the returned data type.
62Its struct timespec allows for higher resolution (ns) timestamps than the
63timeval of SO_TIMESTAMP (ms).
64
65Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in
66struct __kernel_timespec format.
67
68SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038
69on 32 bit machines.
70
711.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW)
72----------------------------------------------------------------------
73
74Supports multiple types of timestamp requests. As a result, this
75socket option takes a bitmap of flags, not a boolean. In::
76
77  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
78
79val is an integer with any of the following bits set. Setting other
80bit returns EINVAL and does not change the current state.
81
82The socket option configures timestamp generation for individual
83sk_buffs (1.3.1), timestamp reporting to the socket's error
84queue (1.3.2) and options (1.3.3). Timestamp generation can also
85be enabled for individual sendmsg calls using cmsg (1.3.4).
86
87
881.3.1 Timestamp Generation
89^^^^^^^^^^^^^^^^^^^^^^^^^^
90
91Some bits are requests to the stack to try to generate timestamps. Any
92combination of them is valid. Changes to these bits apply to newly
93created packets, not to packets already in the stack. As a result, it
94is possible to selectively request timestamps for a subset of packets
95(e.g., for sampling) by embedding an send() call within two setsockopt
96calls, one to enable timestamp generation and one to disable it.
97Timestamps may also be generated for reasons other than being
98requested by a particular socket, such as when receive timestamping is
99enabled system wide, as explained earlier.
100
101SOF_TIMESTAMPING_RX_HARDWARE:
102  Request rx timestamps generated by the network adapter.
103
104SOF_TIMESTAMPING_RX_SOFTWARE:
105  Request rx timestamps when data enters the kernel. These timestamps
106  are generated just after a device driver hands a packet to the
107  kernel receive stack.
108
109SOF_TIMESTAMPING_TX_HARDWARE:
110  Request tx timestamps generated by the network adapter. This flag
111  can be enabled via both socket options and control messages.
112
113SOF_TIMESTAMPING_TX_SOFTWARE:
114  Request tx timestamps when data leaves the kernel. These timestamps
115  are generated in the device driver as close as possible, but always
116  prior to, passing the packet to the network interface. Hence, they
117  require driver support and may not be available for all devices.
118  This flag can be enabled via both socket options and control messages.
119
120SOF_TIMESTAMPING_TX_SCHED:
121  Request tx timestamps prior to entering the packet scheduler. Kernel
122  transmit latency is, if long, often dominated by queuing delay. The
123  difference between this timestamp and one taken at
124  SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
125  of protocol processing. The latency incurred in protocol
126  processing, if any, can be computed by subtracting a userspace
127  timestamp taken immediately before send() from this timestamp. On
128  machines with virtual devices where a transmitted packet travels
129  through multiple devices and, hence, multiple packet schedulers,
130  a timestamp is generated at each layer. This allows for fine
131  grained measurement of queuing delay. This flag can be enabled
132  via both socket options and control messages.
133
134SOF_TIMESTAMPING_TX_ACK:
135  Request tx timestamps when all data in the send buffer has been
136  acknowledged. This only makes sense for reliable protocols. It is
137  currently only implemented for TCP. For that protocol, it may
138  over-report measurement, because the timestamp is generated when all
139  data up to and including the buffer at send() was acknowledged: the
140  cumulative acknowledgment. The mechanism ignores SACK and FACK.
141  This flag can be enabled via both socket options and control messages.
142
143
1441.3.2 Timestamp Reporting
145^^^^^^^^^^^^^^^^^^^^^^^^^
146
147The other three bits control which timestamps will be reported in a
148generated control message. Changes to the bits take immediate
149effect at the timestamp reporting locations in the stack. Timestamps
150are only reported for packets that also have the relevant timestamp
151generation request set.
152
153SOF_TIMESTAMPING_SOFTWARE:
154  Report any software timestamps when available.
155
156SOF_TIMESTAMPING_SYS_HARDWARE:
157  This option is deprecated and ignored.
158
159SOF_TIMESTAMPING_RAW_HARDWARE:
160  Report hardware timestamps as generated by
161  SOF_TIMESTAMPING_TX_HARDWARE when available.
162
163
1641.3.3 Timestamp Options
165^^^^^^^^^^^^^^^^^^^^^^^
166
167The interface supports the options
168
169SOF_TIMESTAMPING_OPT_ID:
170  Generate a unique identifier along with each packet. A process can
171  have multiple concurrent timestamping requests outstanding. Packets
172  can be reordered in the transmit path, for instance in the packet
173  scheduler. In that case timestamps will be queued onto the error
174  queue out of order from the original send() calls. It is not always
175  possible to uniquely match timestamps to the original send() calls
176  based on timestamp order or payload inspection alone, then.
177
178  This option associates each packet at send() with a unique
179  identifier and returns that along with the timestamp. The identifier
180  is derived from a per-socket u32 counter (that wraps). For datagram
181  sockets, the counter increments with each sent packet. For stream
182  sockets, it increments with every byte.
183
184  The counter starts at zero. It is initialized the first time that
185  the socket option is enabled. It is reset each time the option is
186  enabled after having been disabled. Resetting the counter does not
187  change the identifiers of existing packets in the system.
188
189  This option is implemented only for transmit timestamps. There, the
190  timestamp is always looped along with a struct sock_extended_err.
191  The option modifies field ee_data to pass an id that is unique
192  among all possibly concurrently outstanding timestamp requests for
193  that socket.
194
195
196SOF_TIMESTAMPING_OPT_CMSG:
197  Support recv() cmsg for all timestamped packets. Control messages
198  are already supported unconditionally on all packets with receive
199  timestamps and on IPv6 packets with transmit timestamp. This option
200  extends them to IPv4 packets with transmit timestamp. One use case
201  is to correlate packets with their egress device, by enabling socket
202  option IP_PKTINFO simultaneously.
203
204
205SOF_TIMESTAMPING_OPT_TSONLY:
206  Applies to transmit timestamps only. Makes the kernel return the
207  timestamp as a cmsg alongside an empty packet, as opposed to
208  alongside the original packet. This reduces the amount of memory
209  charged to the socket's receive budget (SO_RCVBUF) and delivers
210  the timestamp even if sysctl net.core.tstamp_allow_data is 0.
211  This option disables SOF_TIMESTAMPING_OPT_CMSG.
212
213SOF_TIMESTAMPING_OPT_STATS:
214  Optional stats that are obtained along with the transmit timestamps.
215  It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the
216  transmit timestamp is available, the stats are available in a
217  separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a
218  list of TLVs (struct nlattr) of types. These stats allow the
219  application to associate various transport layer stats with
220  the transmit timestamps, such as how long a certain block of
221  data was limited by peer's receiver window.
222
223SOF_TIMESTAMPING_OPT_PKTINFO:
224  Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming
225  packets with hardware timestamps. The message contains struct
226  scm_ts_pktinfo, which supplies the index of the real interface which
227  received the packet and its length at layer 2. A valid (non-zero)
228  interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is
229  enabled and the driver is using NAPI. The struct contains also two
230  other fields, but they are reserved and undefined.
231
232SOF_TIMESTAMPING_OPT_TX_SWHW:
233  Request both hardware and software timestamps for outgoing packets
234  when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE
235  are enabled at the same time. If both timestamps are generated,
236  two separate messages will be looped to the socket's error queue,
237  each containing just one timestamp.
238
239New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
240disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
241regardless of the setting of sysctl net.core.tstamp_allow_data.
242
243An exception is when a process needs additional cmsg data, for
244instance SOL_IP/IP_PKTINFO to detect the egress network interface.
245Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
246having access to the contents of the original packet, so cannot be
247combined with SOF_TIMESTAMPING_OPT_TSONLY.
248
249
2501.3.4. Enabling timestamps via control messages
251^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
252
253In addition to socket options, timestamp generation can be requested
254per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
255Using this feature, applications can sample timestamps per sendmsg()
256without paying the overhead of enabling and disabling timestamps via
257setsockopt::
258
259  struct msghdr *msg;
260  ...
261  cmsg			       = CMSG_FIRSTHDR(msg);
262  cmsg->cmsg_level	       = SOL_SOCKET;
263  cmsg->cmsg_type	       = SO_TIMESTAMPING;
264  cmsg->cmsg_len	       = CMSG_LEN(sizeof(__u32));
265  *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
266				 SOF_TIMESTAMPING_TX_SOFTWARE |
267				 SOF_TIMESTAMPING_TX_ACK;
268  err = sendmsg(fd, msg, 0);
269
270The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
271the SOF_TIMESTAMPING_TX_* flags set via setsockopt.
272
273Moreover, applications must still enable timestamp reporting via
274setsockopt to receive timestamps::
275
276  __u32 val = SOF_TIMESTAMPING_SOFTWARE |
277	      SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
278  err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
279
280
2811.4 Bytestream Timestamps
282-------------------------
283
284The SO_TIMESTAMPING interface supports timestamping of bytes in a
285bytestream. Each request is interpreted as a request for when the
286entire contents of the buffer has passed a timestamping point. That
287is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
288when all bytes have reached the device driver, regardless of how
289many packets the data has been converted into.
290
291In general, bytestreams have no natural delimiters and therefore
292correlating a timestamp with data is non-trivial. A range of bytes
293may be split across segments, any segments may be merged (possibly
294coalescing sections of previously segmented buffers associated with
295independent send() calls). Segments can be reordered and the same
296byte range can coexist in multiple segments for protocols that
297implement retransmissions.
298
299It is essential that all timestamps implement the same semantics,
300regardless of these possible transformations, as otherwise they are
301incomparable. Handling "rare" corner cases differently from the
302simple case (a 1:1 mapping from buffer to skb) is insufficient
303because performance debugging often needs to focus on such outliers.
304
305In practice, timestamps can be correlated with segments of a
306bytestream consistently, if both semantics of the timestamp and the
307timing of measurement are chosen correctly. This challenge is no
308different from deciding on a strategy for IP fragmentation. There, the
309definition is that only the first fragment is timestamped. For
310bytestreams, we chose that a timestamp is generated only when all
311bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
312implement and reason about. An implementation that has to take into
313account SACK would be more complex due to possible transmission holes
314and out of order arrival.
315
316On the host, TCP can also break the simple 1:1 mapping from buffer to
317skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
318implementation ensures correctness in all cases by tracking the
319individual last byte passed to send(), even if it is no longer the
320last byte after an skbuff extend or merge operation. It stores the
321relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
322has only one such field, only one timestamp can be generated.
323
324In rare cases, a timestamp request can be missed if two requests are
325collapsed onto the same skb. A process can detect this situation by
326enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
327send time with the value returned for each timestamp. It can prevent
328the situation by always flushing the TCP stack in between requests,
329for instance by enabling TCP_NODELAY and disabling TCP_CORK and
330autocork.
331
332These precautions ensure that the timestamp is generated only when all
333bytes have passed a timestamp point, assuming that the network stack
334itself does not reorder the segments. The stack indeed tries to avoid
335reordering. The one exception is under administrator control: it is
336possible to construct a packet scheduler configuration that delays
337segments from the same stream differently. Such a setup would be
338unusual.
339
340
3412 Data Interfaces
342==================
343
344Timestamps are read using the ancillary data feature of recvmsg().
345See `man 3 cmsg` for details of this interface. The socket manual
346page (`man 7 socket`) describes how timestamps generated with
347SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
348
349
3502.1 SCM_TIMESTAMPING records
351----------------------------
352
353These timestamps are returned in a control message with cmsg_level
354SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
355
356For SO_TIMESTAMPING_OLD::
357
358	struct scm_timestamping {
359		struct timespec ts[3];
360	};
361
362For SO_TIMESTAMPING_NEW::
363
364	struct scm_timestamping64 {
365		struct __kernel_timespec ts[3];
366
367Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in
368struct scm_timestamping64 format.
369
370SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038
371on 32 bit machines.
372
373The structure can return up to three timestamps. This is a legacy
374feature. At least one field is non-zero at any time. Most timestamps
375are passed in ts[0]. Hardware timestamps are passed in ts[2].
376
377ts[1] used to hold hardware timestamps converted to system time.
378Instead, expose the hardware clock device on the NIC directly as
379a HW PTP clock source, to allow time conversion in userspace and
380optionally synchronize system time with a userspace PTP stack such
381as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.
382
383Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled
384together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false
385software timestamp will be generated in the recvmsg() call and passed
386in ts[0] when a real software timestamp is missing. This happens also
387on hardware transmit timestamps.
388
3892.1.1 Transmit timestamps with MSG_ERRQUEUE
390^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
391
392For transmit timestamps the outgoing packet is looped back to the
393socket's error queue with the send timestamp(s) attached. A process
394receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
395set and with a msg_control buffer sufficiently large to receive the
396relevant metadata structures. The recvmsg call returns the original
397outgoing data packet with two ancillary messages attached.
398
399A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
400embeds a struct sock_extended_err. This defines the error type. For
401timestamps, the ee_errno field is ENOMSG. The other ancillary message
402will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
403embeds the struct scm_timestamping.
404
405
4062.1.1.2 Timestamp types
407~~~~~~~~~~~~~~~~~~~~~~~
408
409The semantics of the three struct timespec are defined by field
410ee_info in the extended error structure. It contains a value of
411type SCM_TSTAMP_* to define the actual timestamp passed in
412scm_timestamping.
413
414The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
415control fields discussed previously, with one exception. For legacy
416reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
417SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
418is the first if ts[2] is non-zero, the second otherwise, in which
419case the timestamp is stored in ts[0].
420
421
4222.1.1.3 Fragmentation
423~~~~~~~~~~~~~~~~~~~~~
424
425Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
426explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
427then only the first fragment is timestamped and returned to the sending
428socket.
429
430
4312.1.1.4 Packet Payload
432~~~~~~~~~~~~~~~~~~~~~~
433
434The calling application is often not interested in receiving the whole
435packet payload that it passed to the stack originally: the socket
436error queue mechanism is just a method to piggyback the timestamp on.
437In this case, the application can choose to read datagrams with a
438smaller buffer, possibly even of length 0. The payload is truncated
439accordingly. Until the process calls recvmsg() on the error queue,
440however, the full packet is queued, taking up budget from SO_RCVBUF.
441
442
4432.1.1.5 Blocking Read
444~~~~~~~~~~~~~~~~~~~~~
445
446Reading from the error queue is always a non-blocking operation. To
447block waiting on a timestamp, use poll or select. poll() will return
448POLLERR in pollfd.revents if any data is ready on the error queue.
449There is no need to pass this flag in pollfd.events. This flag is
450ignored on request. See also `man 2 poll`.
451
452
4532.1.2 Receive timestamps
454^^^^^^^^^^^^^^^^^^^^^^^^
455
456On reception, there is no reason to read from the socket error queue.
457The SCM_TIMESTAMPING ancillary data is sent along with the packet data
458on a normal recvmsg(). Since this is not a socket error, it is not
459accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
460the meaning of the three fields in struct scm_timestamping is
461implicitly defined. ts[0] holds a software timestamp if set, ts[1]
462is again deprecated and ts[2] holds a hardware timestamp if set.
463
464
4653. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP
466=======================================================================
467
468Hardware time stamping must also be initialized for each device driver
469that is expected to do hardware time stamping. The parameter is defined in
470include/uapi/linux/net_tstamp.h as::
471
472	struct hwtstamp_config {
473		int flags;	/* no flags defined right now, must be zero */
474		int tx_type;	/* HWTSTAMP_TX_* */
475		int rx_filter;	/* HWTSTAMP_FILTER_* */
476	};
477
478Desired behavior is passed into the kernel and to a specific device by
479calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose
480ifr_data points to a struct hwtstamp_config. The tx_type and
481rx_filter are hints to the driver what it is expected to do. If
482the requested fine-grained filtering for incoming packets is not
483supported, the driver may time stamp more than just the requested types
484of packets.
485
486Drivers are free to use a more permissive configuration than the requested
487configuration. It is expected that drivers should only implement directly the
488most generic mode that can be supported. For example if the hardware can
489support HWTSTAMP_FILTER_V2_EVENT, then it should generally always upscale
490HWTSTAMP_FILTER_V2_L2_SYNC_MESSAGE, and so forth, as HWTSTAMP_FILTER_V2_EVENT
491is more generic (and more useful to applications).
492
493A driver which supports hardware time stamping shall update the struct
494with the actual, possibly more permissive configuration. If the
495requested packets cannot be time stamped, then nothing should be
496changed and ERANGE shall be returned (in contrast to EINVAL, which
497indicates that SIOCSHWTSTAMP is not supported at all).
498
499Only a processes with admin rights may change the configuration. User
500space is responsible to ensure that multiple processes don't interfere
501with each other and that the settings are reset.
502
503Any process can read the actual configuration by passing this
504structure to ioctl(SIOCGHWTSTAMP) in the same way.  However, this has
505not been implemented in all drivers.
506
507::
508
509    /* possible values for hwtstamp_config->tx_type */
510    enum {
511	    /*
512	    * no outgoing packet will need hardware time stamping;
513	    * should a packet arrive which asks for it, no hardware
514	    * time stamping will be done
515	    */
516	    HWTSTAMP_TX_OFF,
517
518	    /*
519	    * enables hardware time stamping for outgoing packets;
520	    * the sender of the packet decides which are to be
521	    * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
522	    * before sending the packet
523	    */
524	    HWTSTAMP_TX_ON,
525    };
526
527    /* possible values for hwtstamp_config->rx_filter */
528    enum {
529	    /* time stamp no incoming packet at all */
530	    HWTSTAMP_FILTER_NONE,
531
532	    /* time stamp any incoming packet */
533	    HWTSTAMP_FILTER_ALL,
534
535	    /* return value: time stamp all packets requested plus some others */
536	    HWTSTAMP_FILTER_SOME,
537
538	    /* PTP v1, UDP, any kind of event packet */
539	    HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
540
541	    /* for the complete list of values, please check
542	    * the include file include/uapi/linux/net_tstamp.h
543	    */
544    };
545
5463.1 Hardware Timestamping Implementation: Device Drivers
547--------------------------------------------------------
548
549A driver which supports hardware time stamping must support the
550SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with
551the actual values as described in the section on SIOCSHWTSTAMP.  It
552should also support SIOCGHWTSTAMP.
553
554Time stamps for received packets must be stored in the skb. To get a pointer
555to the shared time stamp structure of the skb call skb_hwtstamps(). Then
556set the time stamps in the structure::
557
558    struct skb_shared_hwtstamps {
559	    /* hardware time stamp transformed into duration
560	    * since arbitrary point in time
561	    */
562	    ktime_t	hwtstamp;
563    };
564
565Time stamps for outgoing packets are to be generated as follows:
566
567- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
568  is set no-zero. If yes, then the driver is expected to do hardware time
569  stamping.
570- If this is possible for the skb and requested, then declare
571  that the driver is doing the time stamping by setting the flag
572  SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with::
573
574      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
575
576  You might want to keep a pointer to the associated skb for the next step
577  and not free the skb. A driver not supporting hardware time stamping doesn't
578  do that. A driver must never touch sk_buff::tstamp! It is used to store
579  software generated time stamps by the network subsystem.
580- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
581  as possible. skb_tx_timestamp() provides a software time stamp if requested
582  and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
583- As soon as the driver has sent the packet and/or obtained a
584  hardware time stamp for it, it passes the time stamp back by
585  calling skb_hwtstamp_tx() with the original skb, the raw
586  hardware time stamp. skb_hwtstamp_tx() clones the original skb and
587  adds the timestamps, therefore the original skb has to be freed now.
588  If obtaining the hardware time stamp somehow fails, then the driver
589  should not fall back to software time stamping. The rationale is that
590  this would occur at a later time in the processing pipeline than other
591  software time stamping and therefore could lead to unexpected deltas
592  between time stamps.
593
5943.2 Special considerations for stacked PTP Hardware Clocks
595----------------------------------------------------------
596
597There are situations when there may be more than one PHC (PTP Hardware Clock)
598in the data path of a packet. The kernel has no explicit mechanism to allow the
599user to select which PHC to use for timestamping Ethernet frames. Instead, the
600assumption is that the outermost PHC is always the most preferable, and that
601kernel drivers collaborate towards achieving that goal. Currently there are 3
602cases of stacked PHCs, detailed below:
603
6043.2.1 DSA (Distributed Switch Architecture) switches
605^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
606
607These are Ethernet switches which have one of their ports connected to an
608(otherwise completely unaware) host Ethernet interface, and perform the role of
609a port multiplier with optional forwarding acceleration features.  Each DSA
610switch port is visible to the user as a standalone (virtual) network interface,
611and its network I/O is performed, under the hood, indirectly through the host
612interface (redirecting to the host port on TX, and intercepting frames on RX).
613
614When a DSA switch is attached to a host port, PTP synchronization has to
615suffer, since the switch's variable queuing delay introduces a path delay
616jitter between the host port and its PTP partner. For this reason, some DSA
617switches include a timestamping clock of their own, and have the ability to
618perform network timestamping on their own MAC, such that path delays only
619measure wire and PHY propagation latencies. Timestamping DSA switches are
620supported in Linux and expose the same ABI as any other network interface (save
621for the fact that the DSA interfaces are in fact virtual in terms of network
622I/O, they do have their own PHC).  It is typical, but not mandatory, for all
623interfaces of a DSA switch to share the same PHC.
624
625By design, PTP timestamping with a DSA switch does not need any special
626handling in the driver for the host port it is attached to.  However, when the
627host port also supports PTP timestamping, DSA will take care of intercepting
628the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable
629hardware timestamping on it. This is because the SO_TIMESTAMPING API does not
630allow the delivery of multiple hardware timestamps for the same packet, so
631anybody else except for the DSA switch port must be prevented from doing so.
632
633In the generic layer, DSA provides the following infrastructure for PTP
634timestamping:
635
636- ``.port_txtstamp()``: a hook called prior to the transmission of
637  packets with a hardware TX timestamping request from user space.
638  This is required for two-step timestamping, since the hardware
639  timestamp becomes available after the actual MAC transmission, so the
640  driver must be prepared to correlate the timestamp with the original
641  packet so that it can re-enqueue the packet back into the socket's
642  error queue. To save the packet for when the timestamp becomes
643  available, the driver can call ``skb_clone_sk`` , save the clone pointer
644  in skb->cb and enqueue a tx skb queue. Typically, a switch will have a
645  PTP TX timestamp register (or sometimes a FIFO) where the timestamp
646  becomes available. In case of a FIFO, the hardware might store
647  key-value pairs of PTP sequence ID/message type/domain number and the
648  actual timestamp. To perform the correlation correctly between the
649  packets in a queue waiting for timestamping and the actual timestamps,
650  drivers can use a BPF classifier (``ptp_classify_raw``) to identify
651  the PTP transport type, and ``ptp_parse_header`` to interpret the PTP
652  header fields. There may be an IRQ that is raised upon this
653  timestamp's availability, or the driver might have to poll after
654  invoking ``dev_queue_xmit()`` towards the host interface.
655  One-step TX timestamping do not require packet cloning, since there is
656  no follow-up message required by the PTP protocol (because the
657  TX timestamp is embedded into the packet by the MAC), and therefore
658  user space does not expect the packet annotated with the TX timestamp
659  to be re-enqueued into its socket's error queue.
660
661- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to
662  identify PTP event messages (any other packets, including PTP general
663  messages, are not timestamped). The original (and only) timestampable
664  skb is provided to the driver, for it to annotate it with a timestamp,
665  if that is immediately available, or defer to later. On reception,
666  timestamps might either be available in-band (through metadata in the
667  DSA header, or attached in other ways to the packet), or out-of-band
668  (through another RX timestamping FIFO). Deferral on RX is typically
669  necessary when retrieving the timestamp needs a sleepable context. In
670  that case, it is the responsibility of the DSA driver to call
671  ``netif_rx_ni()`` on the freshly timestamped skb.
672
6733.2.2 Ethernet PHYs
674^^^^^^^^^^^^^^^^^^^
675
676These are devices that typically fulfill a Layer 1 role in the network stack,
677hence they do not have a representation in terms of a network interface as DSA
678switches do. However, PHYs may be able to detect and timestamp PTP packets, for
679performance reasons: timestamps taken as close as possible to the wire have the
680potential to yield a more stable and precise synchronization.
681
682A PHY driver that supports PTP timestamping must create a ``struct
683mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence
684of this pointer will be checked by the networking stack.
685
686Since PHYs do not have network interface representations, the timestamping and
687ethtool ioctl operations for them need to be mediated by their respective MAC
688driver.  Therefore, as opposed to DSA switches, modifications need to be done
689to each individual MAC driver for PHY timestamping support. This entails:
690
691- Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)``
692  is true or not. If it is, then the MAC driver should not process this request
693  but instead pass it on to the PHY using ``phy_mii_ioctl()``.
694
695- On RX, special intervention may or may not be needed, depending on the
696  function used to deliver skb's up the network stack. In the case of plain
697  ``netif_rx()`` and similar, MAC drivers must check whether
698  ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't
699  call ``netif_rx()`` at all.  If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is
700  enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook
701  will be called now, to determine, using logic very similar to DSA, whether
702  deferral for RX timestamping is necessary.  Again like DSA, it becomes the
703  responsibility of the PHY driver to send the packet up the stack when the
704  timestamp is available.
705
706  For other skb receive functions, such as ``napi_gro_receive`` and
707  ``netif_receive_skb``, the stack automatically checks whether
708  ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside
709  the driver.
710
711- On TX, again, special intervention might or might not be needed.  The
712  function that calls the ``mii_ts->txtstamp()`` hook is named
713  ``skb_clone_tx_timestamp()``. This function can either be called directly
714  (case in which explicit MAC driver support is indeed needed), but the
715  function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC
716  drivers already perform for software timestamping purposes. Therefore, if a
717  MAC supports software timestamping, it does not need to do anything further
718  at this stage.
719
7203.2.3 MII bus snooping devices
721^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
722
723These perform the same role as timestamping Ethernet PHYs, save for the fact
724that they are discrete devices and can therefore be used in conjunction with
725any PHY even if it doesn't support timestamping. In Linux, they are
726discoverable and attachable to a ``struct phy_device`` through Device Tree, and
727for the rest, they use the same mii_ts infrastructure as those. See
728Documentation/devicetree/bindings/ptp/timestamper.txt for more details.
729
7303.2.4 Other caveats for MAC drivers
731^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
732
733Stacked PHCs, especially DSA (but not only) - since that doesn't require any
734modification to MAC drivers, so it is more difficult to ensure correctness of
735all possible code paths - is that they uncover bugs which were impossible to
736trigger before the existence of stacked PTP clocks.  One example has to do with
737this line of code, already presented earlier::
738
739      skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
740
741Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY
742driver or a MII bus snooping device driver, should set this flag.
743But a MAC driver that is unaware of PHC stacking might get tripped up by
744somebody other than itself setting this flag, and deliver a duplicate
745timestamp.
746For example, a typical driver design for TX timestamping might be to split the
747transmission part into 2 portions:
748
7491. "TX": checks whether PTP timestamping has been previously enabled through
750   the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the
751   current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags &
752   SKBTX_HW_TSTAMP``"). If this is true, it sets the
753   "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as
754   described above, in the case of a stacked PHC system, this condition should
755   never trigger, as this MAC is certainly not the outermost PHC. But this is
756   not where the typical issue is.  Transmission proceeds with this packet.
757
7582. "TX confirmation": Transmission has finished. The driver checks whether it
759   is necessary to collect any TX timestamp for it. Here is where the typical
760   issues are: the MAC driver takes a shortcut and only checks whether
761   "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked
762   PHC system, this is incorrect because this MAC driver is not the only entity
763   in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first
764   place.
765
766The correct solution for this problem is for MAC drivers to have a compound
767check in their "TX confirmation" portion, not only for
768"``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for
769"``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures
770that PTP timestamping is not enabled for anything other than the outermost PHC,
771this enhanced check will avoid delivering a duplicated TX timestamp to user
772space.
773