1.. SPDX-License-Identifier: GPL-2.0
2
3===
4RDS
5===
6
7Overview
8========
9
10This readme tries to provide some background on the hows and whys of RDS,
11and will hopefully help you find your way around the code.
12
13In addition, please see this email about RDS origins:
14http://oss.oracle.com/pipermail/rds-devel/2007-November/000228.html
15
16RDS Architecture
17================
18
19RDS provides reliable, ordered datagram delivery by using a single
20reliable connection between any two nodes in the cluster. This allows
21applications to use a single socket to talk to any other process in the
22cluster - so in a cluster with N processes you need N sockets, in contrast
23to N*N if you use a connection-oriented socket transport like TCP.
24
25RDS is not Infiniband-specific; it was designed to support different
26transports.  The current implementation used to support RDS over TCP as well
27as IB.
28
29The high-level semantics of RDS from the application's point of view are
30
31 *	Addressing
32
33	RDS uses IPv4 addresses and 16bit port numbers to identify
34	the end point of a connection. All socket operations that involve
35	passing addresses between kernel and user space generally
36	use a struct sockaddr_in.
37
38	The fact that IPv4 addresses are used does not mean the underlying
39	transport has to be IP-based. In fact, RDS over IB uses a
40	reliable IB connection; the IP address is used exclusively to
41	locate the remote node's GID (by ARPing for the given IP).
42
43	The port space is entirely independent of UDP, TCP or any other
44	protocol.
45
46 *	Socket interface
47
48	RDS sockets work *mostly* as you would expect from a BSD
49	socket. The next section will cover the details. At any rate,
50	all I/O is performed through the standard BSD socket API.
51	Some additions like zerocopy support are implemented through
52	control messages, while other extensions use the getsockopt/
53	setsockopt calls.
54
55	Sockets must be bound before you can send or receive data.
56	This is needed because binding also selects a transport and
57	attaches it to the socket. Once bound, the transport assignment
58	does not change. RDS will tolerate IPs moving around (eg in
59	a active-active HA scenario), but only as long as the address
60	doesn't move to a different transport.
61
62 *	sysctls
63
64	RDS supports a number of sysctls in /proc/sys/net/rds
65
66
67Socket Interface
68================
69
70  AF_RDS, PF_RDS, SOL_RDS
71	AF_RDS and PF_RDS are the domain type to be used with socket(2)
72	to create RDS sockets. SOL_RDS is the socket-level to be used
73	with setsockopt(2) and getsockopt(2) for RDS specific socket
74	options.
75
76  fd = socket(PF_RDS, SOCK_SEQPACKET, 0);
77	This creates a new, unbound RDS socket.
78
79  setsockopt(SOL_SOCKET): send and receive buffer size
80	RDS honors the send and receive buffer size socket options.
81	You are not allowed to queue more than SO_SNDSIZE bytes to
82	a socket. A message is queued when sendmsg is called, and
83	it leaves the queue when the remote system acknowledges
84	its arrival.
85
86	The SO_RCVSIZE option controls the maximum receive queue length.
87	This is a soft limit rather than a hard limit - RDS will
88	continue to accept and queue incoming messages, even if that
89	takes the queue length over the limit. However, it will also
90	mark the port as "congested" and send a congestion update to
91	the source node. The source node is supposed to throttle any
92	processes sending to this congested port.
93
94  bind(fd, &sockaddr_in, ...)
95	This binds the socket to a local IP address and port, and a
96	transport, if one has not already been selected via the
97	SO_RDS_TRANSPORT socket option
98
99  sendmsg(fd, ...)
100	Sends a message to the indicated recipient. The kernel will
101	transparently establish the underlying reliable connection
102	if it isn't up yet.
103
104	An attempt to send a message that exceeds SO_SNDSIZE will
105	return with -EMSGSIZE
106
107	An attempt to send a message that would take the total number
108	of queued bytes over the SO_SNDSIZE threshold will return
109	EAGAIN.
110
111	An attempt to send a message to a destination that is marked
112	as "congested" will return ENOBUFS.
113
114  recvmsg(fd, ...)
115	Receives a message that was queued to this socket. The sockets
116	recv queue accounting is adjusted, and if the queue length
117	drops below SO_SNDSIZE, the port is marked uncongested, and
118	a congestion update is sent to all peers.
119
120	Applications can ask the RDS kernel module to receive
121	notifications via control messages (for instance, there is a
122	notification when a congestion update arrived, or when a RDMA
123	operation completes). These notifications are received through
124	the msg.msg_control buffer of struct msghdr. The format of the
125	messages is described in manpages.
126
127  poll(fd)
128	RDS supports the poll interface to allow the application
129	to implement async I/O.
130
131	POLLIN handling is pretty straightforward. When there's an
132	incoming message queued to the socket, or a pending notification,
133	we signal POLLIN.
134
135	POLLOUT is a little harder. Since you can essentially send
136	to any destination, RDS will always signal POLLOUT as long as
137	there's room on the send queue (ie the number of bytes queued
138	is less than the sendbuf size).
139
140	However, the kernel will refuse to accept messages to
141	a destination marked congested - in this case you will loop
142	forever if you rely on poll to tell you what to do.
143	This isn't a trivial problem, but applications can deal with
144	this - by using congestion notifications, and by checking for
145	ENOBUFS errors returned by sendmsg.
146
147  setsockopt(SOL_RDS, RDS_CANCEL_SENT_TO, &sockaddr_in)
148	This allows the application to discard all messages queued to a
149	specific destination on this particular socket.
150
151	This allows the application to cancel outstanding messages if
152	it detects a timeout. For instance, if it tried to send a message,
153	and the remote host is unreachable, RDS will keep trying forever.
154	The application may decide it's not worth it, and cancel the
155	operation. In this case, it would use RDS_CANCEL_SENT_TO to
156	nuke any pending messages.
157
158  ``setsockopt(fd, SOL_RDS, SO_RDS_TRANSPORT, (int *)&transport ..), getsockopt(fd, SOL_RDS, SO_RDS_TRANSPORT, (int *)&transport ..)``
159	Set or read an integer defining  the underlying
160	encapsulating transport to be used for RDS packets on the
161	socket. When setting the option, integer argument may be
162	one of RDS_TRANS_TCP or RDS_TRANS_IB. When retrieving the
163	value, RDS_TRANS_NONE will be returned on an unbound socket.
164	This socket option may only be set exactly once on the socket,
165	prior to binding it via the bind(2) system call. Attempts to
166	set SO_RDS_TRANSPORT on a socket for which the transport has
167	been previously attached explicitly (by SO_RDS_TRANSPORT) or
168	implicitly (via bind(2)) will return an error of EOPNOTSUPP.
169	An attempt to set SO_RDS_TRANSPORT to RDS_TRANS_NONE will
170	always return EINVAL.
171
172RDMA for RDS
173============
174
175  see rds-rdma(7) manpage (available in rds-tools)
176
177
178Congestion Notifications
179========================
180
181  see rds(7) manpage
182
183
184RDS Protocol
185============
186
187  Message header
188
189    The message header is a 'struct rds_header' (see rds.h):
190
191    Fields:
192
193      h_sequence:
194	  per-packet sequence number
195      h_ack:
196	  piggybacked acknowledgment of last packet received
197      h_len:
198	  length of data, not including header
199      h_sport:
200	  source port
201      h_dport:
202	  destination port
203      h_flags:
204	  Can be:
205
206	  =============  ==================================
207	  CONG_BITMAP    this is a congestion update bitmap
208	  ACK_REQUIRED   receiver must ack this packet
209	  RETRANSMITTED  packet has previously been sent
210	  =============  ==================================
211
212      h_credit:
213	  indicate to other end of connection that
214	  it has more credits available (i.e. there is
215	  more send room)
216      h_padding[4]:
217	  unused, for future use
218      h_csum:
219	  header checksum
220      h_exthdr:
221	  optional data can be passed here. This is currently used for
222	  passing RDMA-related information.
223
224  ACK and retransmit handling
225
226      One might think that with reliable IB connections you wouldn't need
227      to ack messages that have been received.  The problem is that IB
228      hardware generates an ack message before it has DMAed the message
229      into memory.  This creates a potential message loss if the HCA is
230      disabled for any reason between when it sends the ack and before
231      the message is DMAed and processed.  This is only a potential issue
232      if another HCA is available for fail-over.
233
234      Sending an ack immediately would allow the sender to free the sent
235      message from their send queue quickly, but could cause excessive
236      traffic to be used for acks. RDS piggybacks acks on sent data
237      packets.  Ack-only packets are reduced by only allowing one to be
238      in flight at a time, and by the sender only asking for acks when
239      its send buffers start to fill up. All retransmissions are also
240      acked.
241
242  Flow Control
243
244      RDS's IB transport uses a credit-based mechanism to verify that
245      there is space in the peer's receive buffers for more data. This
246      eliminates the need for hardware retries on the connection.
247
248  Congestion
249
250      Messages waiting in the receive queue on the receiving socket
251      are accounted against the sockets SO_RCVBUF option value.  Only
252      the payload bytes in the message are accounted for.  If the
253      number of bytes queued equals or exceeds rcvbuf then the socket
254      is congested.  All sends attempted to this socket's address
255      should return block or return -EWOULDBLOCK.
256
257      Applications are expected to be reasonably tuned such that this
258      situation very rarely occurs.  An application encountering this
259      "back-pressure" is considered a bug.
260
261      This is implemented by having each node maintain bitmaps which
262      indicate which ports on bound addresses are congested.  As the
263      bitmap changes it is sent through all the connections which
264      terminate in the local address of the bitmap which changed.
265
266      The bitmaps are allocated as connections are brought up.  This
267      avoids allocation in the interrupt handling path which queues
268      sages on sockets.  The dense bitmaps let transports send the
269      entire bitmap on any bitmap change reasonably efficiently.  This
270      is much easier to implement than some finer-grained
271      communication of per-port congestion.  The sender does a very
272      inexpensive bit test to test if the port it's about to send to
273      is congested or not.
274
275
276RDS Transport Layer
277===================
278
279  As mentioned above, RDS is not IB-specific. Its code is divided
280  into a general RDS layer and a transport layer.
281
282  The general layer handles the socket API, congestion handling,
283  loopback, stats, usermem pinning, and the connection state machine.
284
285  The transport layer handles the details of the transport. The IB
286  transport, for example, handles all the queue pairs, work requests,
287  CM event handlers, and other Infiniband details.
288
289
290RDS Kernel Structures
291=====================
292
293  struct rds_message
294    aka possibly "rds_outgoing", the generic RDS layer copies data to
295    be sent and sets header fields as needed, based on the socket API.
296    This is then queued for the individual connection and sent by the
297    connection's transport.
298
299  struct rds_incoming
300    a generic struct referring to incoming data that can be handed from
301    the transport to the general code and queued by the general code
302    while the socket is awoken. It is then passed back to the transport
303    code to handle the actual copy-to-user.
304
305  struct rds_socket
306    per-socket information
307
308  struct rds_connection
309    per-connection information
310
311  struct rds_transport
312    pointers to transport-specific functions
313
314  struct rds_statistics
315    non-transport-specific statistics
316
317  struct rds_cong_map
318    wraps the raw congestion bitmap, contains rbnode, waitq, etc.
319
320Connection management
321=====================
322
323  Connections may be in UP, DOWN, CONNECTING, DISCONNECTING, and
324  ERROR states.
325
326  The first time an attempt is made by an RDS socket to send data to
327  a node, a connection is allocated and connected. That connection is
328  then maintained forever -- if there are transport errors, the
329  connection will be dropped and re-established.
330
331  Dropping a connection while packets are queued will cause queued or
332  partially-sent datagrams to be retransmitted when the connection is
333  re-established.
334
335
336The send path
337=============
338
339  rds_sendmsg()
340    - struct rds_message built from incoming data
341    - CMSGs parsed (e.g. RDMA ops)
342    - transport connection alloced and connected if not already
343    - rds_message placed on send queue
344    - send worker awoken
345
346  rds_send_worker()
347    - calls rds_send_xmit() until queue is empty
348
349  rds_send_xmit()
350    - transmits congestion map if one is pending
351    - may set ACK_REQUIRED
352    - calls transport to send either non-RDMA or RDMA message
353      (RDMA ops never retransmitted)
354
355  rds_ib_xmit()
356    - allocs work requests from send ring
357    - adds any new send credits available to peer (h_credits)
358    - maps the rds_message's sg list
359    - piggybacks ack
360    - populates work requests
361    - post send to connection's queue pair
362
363The recv path
364=============
365
366  rds_ib_recv_cq_comp_handler()
367    - looks at write completions
368    - unmaps recv buffer from device
369    - no errors, call rds_ib_process_recv()
370    - refill recv ring
371
372  rds_ib_process_recv()
373    - validate header checksum
374    - copy header to rds_ib_incoming struct if start of a new datagram
375    - add to ibinc's fraglist
376    - if competed datagram:
377	 - update cong map if datagram was cong update
378	 - call rds_recv_incoming() otherwise
379	 - note if ack is required
380
381  rds_recv_incoming()
382    - drop duplicate packets
383    - respond to pings
384    - find the sock associated with this datagram
385    - add to sock queue
386    - wake up sock
387    - do some congestion calculations
388  rds_recvmsg
389    - copy data into user iovec
390    - handle CMSGs
391    - return to application
392
393Multipath RDS (mprds)
394=====================
395  Mprds is multipathed-RDS, primarily intended for RDS-over-TCP
396  (though the concept can be extended to other transports). The classical
397  implementation of RDS-over-TCP is implemented by demultiplexing multiple
398  PF_RDS sockets between any 2 endpoints (where endpoint == [IP address,
399  port]) over a single TCP socket between the 2 IP addresses involved. This
400  has the limitation that it ends up funneling multiple RDS flows over a
401  single TCP flow, thus it is
402  (a) upper-bounded to the single-flow bandwidth,
403  (b) suffers from head-of-line blocking for all the RDS sockets.
404
405  Better throughput (for a fixed small packet size, MTU) can be achieved
406  by having multiple TCP/IP flows per rds/tcp connection, i.e., multipathed
407  RDS (mprds).  Each such TCP/IP flow constitutes a path for the rds/tcp
408  connection. RDS sockets will be attached to a path based on some hash
409  (e.g., of local address and RDS port number) and packets for that RDS
410  socket will be sent over the attached path using TCP to segment/reassemble
411  RDS datagrams on that path.
412
413  Multipathed RDS is implemented by splitting the struct rds_connection into
414  a common (to all paths) part, and a per-path struct rds_conn_path. All
415  I/O workqs and reconnect threads are driven from the rds_conn_path.
416  Transports such as TCP that are multipath capable may then set up a
417  TCP socket per rds_conn_path, and this is managed by the transport via
418  the transport privatee cp_transport_data pointer.
419
420  Transports announce themselves as multipath capable by setting the
421  t_mp_capable bit during registration with the rds core module. When the
422  transport is multipath-capable, rds_sendmsg() hashes outgoing traffic
423  across multiple paths. The outgoing hash is computed based on the
424  local address and port that the PF_RDS socket is bound to.
425
426  Additionally, even if the transport is MP capable, we may be
427  peering with some node that does not support mprds, or supports
428  a different number of paths. As a result, the peering nodes need
429  to agree on the number of paths to be used for the connection.
430  This is done by sending out a control packet exchange before the
431  first data packet. The control packet exchange must have completed
432  prior to outgoing hash completion in rds_sendmsg() when the transport
433  is mutlipath capable.
434
435  The control packet is an RDS ping packet (i.e., packet to rds dest
436  port 0) with the ping packet having a rds extension header option  of
437  type RDS_EXTHDR_NPATHS, length 2 bytes, and the value is the
438  number of paths supported by the sender. The "probe" ping packet will
439  get sent from some reserved port, RDS_FLAG_PROBE_PORT (in <linux/rds.h>)
440  The receiver of a ping from RDS_FLAG_PROBE_PORT will thus immediately
441  be able to compute the min(sender_paths, rcvr_paths). The pong
442  sent in response to a probe-ping should contain the rcvr's npaths
443  when the rcvr is mprds-capable.
444
445  If the rcvr is not mprds-capable, the exthdr in the ping will be
446  ignored.  In this case the pong will not have any exthdrs, so the sender
447  of the probe-ping can default to single-path mprds.
448
449