Documentation/networking/rds.rst

1 .. SPDX-License-Identifier: GPL-2.0
14 http://oss.oracle.com/pipermail/rds-devel/2007-November/000228.html
22 cluster - so in a cluster with N processes you need N sockets, in contrast
23 to N*N if you use a connection-oriented socket transport like TCP.
25 RDS is not Infiniband-specific; it was designed to support different
29 The high-level semantics of RDS from the application's point of view are
39 	transport has to be IP-based. In fact, RDS over IB uses a
59 	a active-active HA scenario), but only as long as the address
72 	to create RDS sockets. SOL_RDS is the socket-level to be used
87 	This is a soft limit rather than a hard limit - RDS will
105 	return with -EMSGSIZE
116 	recv queue accounting is adjusted, and if the queue length
141 	a destination marked congested - in this case you will loop
144 	this - by using congestion notifications, and by checking for
175   see rds-rdma(7) manpage (available in rds-tools)
194 	  per-packet sequence number
222 	  passing RDMA-related information.
232       if another HCA is available for fail-over.
237       packets.  Ack-only packets are reduced by only allowing one to be
244       RDS's IB transport uses a credit-based mechanism to verify that
255       should return block or return -EWOULDBLOCK.
259       "back-pressure" is considered a bug.
270       is much easier to implement than some finer-grained
271       communication of per-port congestion.  The sender does a very
279   As mentioned above, RDS is not IB-specific. Its code is divided
303     code to handle the actual copy-to-user.
306     per-socket information
309     per-connection information
312     pointers to transport-specific functions
315     non-transport-specific statistics
328   then maintained forever -- if there are transport errors, the
329   connection will be dropped and re-established.
332   partially-sent datagrams to be retransmitted when the connection is
333   re-established.
340     - struct rds_message built from incoming data
341     - CMSGs parsed (e.g. RDMA ops)
342     - transport connection alloced and connected if not already
343     - rds_message placed on send queue
344     - send worker awoken
347     - calls rds_send_xmit() until queue is empty
350     - transmits congestion map if one is pending
351     - may set ACK_REQUIRED
352     - calls transport to send either non-RDMA or RDMA message
356     - allocs work requests from send ring
357     - adds any new send credits available to peer (h_credits)
358     - maps the rds_message's sg list
359     - piggybacks ack
360     - populates work requests
361     - post send to connection's queue pair
363 The recv path
367     - looks at write completions
368     - unmaps recv buffer from device
369     - no errors, call rds_ib_process_recv()
370     - refill recv ring
373     - validate header checksum
374     - copy header to rds_ib_incoming struct if start of a new datagram
375     - add to ibinc's fraglist
376     - if competed datagram:
377 	 - update cong map if datagram was cong update
378 	 - call rds_recv_incoming() otherwise
379 	 - note if ack is required
382     - drop duplicate packets
383     - respond to pings
384     - find the sock associated with this datagram
385     - add to sock queue
386     - wake up sock
387     - do some congestion calculations
389     - copy data into user iovec
390     - handle CMSGs
391     - return to application
395   Mprds is multipathed-RDS, primarily intended for RDS-over-TCP
397   implementation of RDS-over-TCP is implemented by demultiplexing multiple
402   (a) upper-bounded to the single-flow bandwidth,
403   (b) suffers from head-of-line blocking for all the RDS sockets.
414   a common (to all paths) part, and a per-path struct rds_conn_path. All
422   transport is multipath-capable, rds_sendmsg() hashes outgoing traffic
442   sent in response to a probe-ping should contain the rcvr's npaths
443   when the rcvr is mprds-capable.
445   If the rcvr is not mprds-capable, the exthdr in the ping will be
447   of the probe-ping can default to single-path mprds.