qemu/docs/rdma.txt

1 (RDMA: Remote Direct Memory Access)
2 RDMA Live Migration Specification, Version # 1
5 Github: git@github.com:hinesmr/qemu.git, 'rdma' branch
18 * RDMA Migration Protocol Description
28 RDMA helps make your migration more deterministic under heavy load because
30 because the RDMA I/O architecture reduces the number of interrupts and
37 RDMA currently comes in two flavors: both Ethernet based (RoCE, or RDMA
39 migration using RDMA is capable of using both technologies because of
43 Refer to openfabrics.org or your respective RDMA hardware vendor for
47 for a working build of QEMU to run successfully using RDMA Migration.
52 Use of RDMA during migration requires pinning and registering memory
56 of RDMA migration may in fact be harmful to co-located VMs or other
59 use of RDMA is discouraged and it is recommended to use standard TCP migration.
66 high-performance RDMA hardware using the following command:
69 $ migrate_set_capability rdma-pin-all on # disabled by default
84 still gain from the benefits of advanced pinning with RDMA.
92 $ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your RDMA device
96 qemu ..... -incoming rdma:host:port
101 $ migrate -d rdma:host:port
106 Here is a brief summary of total migration time and downtime using RDMA:
123 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
124 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
127 you have to migrate using RDMA.
135 RDMA Protocol Description:
138 Migration with RDMA is separated into two parts:
140 1. The transmission of the pages using RDMA
148 The only difference between a SEND message and an RDMA
151 infiniband receiver side, whereas RDMA messages (used
161 RDMA messages are much easier to deal with. Once the memory
167 completes, given that RDMA migrations are very fast.)
176 as follows (migration-rdma.c):
259    hold the rkey need to perform RDMA. Note that the virtual address
271    before performing the RDMA operations.
273    be sent with RDMA, the registration commands are used to ask the
361 At the beginning of the migration, (migration-rdma.c),
368 addresses and possibly includes pre-registered RDMA keys in case dynamic
372 but is instead migrated with normal RDMA Write operations.
381 After pinning, an RDMA Write is generated and transmitted
390 and helps keep the hardware busy performing RDMA operations.
397 we use for RDMA migration.
401 cleanup all the RDMA descriptors and unregister all
406 socket is broken during a non-RDMA based migration.
414    the use of KSM and ballooning while using RDMA.