qemu/docs/rdma.txt

2 RDMA Live Migration Specification, Version # 1
18 * RDMA Migration Protocol Description
21 * Migration of VM's ram
28 RDMA helps make your migration more deterministic under heavy load because
31 data copies by bypassing the host networking stack. In particular, a TCP-based
32 migration, under certain types of memory-bound workloads, may take a more
33 unpredictable amount of time to complete the migration if the amount of
34 memory tracked during each live migration iteration round cannot keep pace
38 over Converged Ethernet) as well as Infiniband-based. This implementation of
39 migration using RDMA is capable of using both technologies because of
47 for a working build of QEMU to run successfully using RDMA Migration.
52 Use of RDMA during migration requires pinning and registering memory
56 of RDMA migration may in fact be harmful to co-located VMs or other
59 use of RDMA is discouraged and it is recommended to use standard TCP migration.
65 bulk-phase round of the migration and can be enabled for extremely
66 high-performance RDMA hardware using the following command:
69 $ migrate_set_capability rdma-pin-all on # disabled by default
75 of the migration, which can greatly reduce the "total" time of your migration.
82 and thus extending the total migration time. However, this will not
83 affect the determinism or predictability of your migration you will
89 First, set the migration speed to match your hardware's capabilities:
92 $ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your RDMA device
96 qemu ..... -incoming rdma:host:port
98 Finally, perform the actual migration on the source machine:
101 $ migrate -d rdma:host:port
106 Here is a brief summary of total migration time and downtime using RDMA:
107 Using a 40gbps infiniband link performing a worst-case stress test,
111 $ apt-get install stress
112 $ stress --vm-bytes 7500M --vm 1 --vm-keep
114 1. Migration throughput: 26 gigabits/second.
123 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
124 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
130 migration *downtime*. This is because, without this feature, all of the
132 the bulk round and does not need to be re-registered during the successive
138 Migration with RDMA is separated into two parts:
166 (Memory is not released from pinning until the migration
173 a control transport for migration of device state.
175 To begin the migration, the initial connection setup is
176 as follows (migration-rdma.c):
201 The maximum number of repeats is hard-coded to 4096. This is a conservative
208      3. Ready                      (control-channel is available)
209      4. QEMU File                  (for sending non-live device state)
223 After connection setup, message 5 & 6 are used to exchange ram block
226 After ram block exchange is completed, we have two protocol-level
227 functions, responsible for communicating control-channel commands
238 3. Block on a CQ event channel and wait for the SEND to arrive.
240 5. Verify that the command-type and version received matches the one we expected.
244 1. Block on the CQ event channel waiting for a READY command
256    we block again and wait for that response using the additional
267    this protocol before the actual migration begins. This information includes
277    when transmitting non-live state, such as devices or to send
278    its own protocol information during the migration process.
295 at connection-setup time before any infiniband traffic is generated.
322 Finally: Negotiation happens with the Flags field: If the primary-VM
324 will return a zero-bit for that flag and the primary-VM will understand
325 that as not being an available capability and will thus disable that
326 capability on the primary-VM side.
337 describe above to deliver bytes without changing the upper-level
344 to hold on to the bytes received from control-channel's SEND
347 Each time we receive a complete "QEMU File" control-channel
356 asking for a new SEND message to re-fill the buffer.
358 Migration of VM's ram:
361 At the beginning of the migration, (migration-rdma.c),
368 addresses and possibly includes pre-registered RDMA keys in case dynamic
369 page registration was disabled on the server-side, otherwise not.
374 Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now).
392 Error-handling:
397 we use for RDMA migration.
400 the decision is to abort the migration entirely and
406 socket is broken during a non-RDMA based migration.
410 1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
412    an aborted migration (but with the source VM left unaffected).
415 3. Also, some form of balloon-device usage tracking would also
417 4. Use LRU to provide more fine-grained direction of UNREGISTER
419 5. Expose UNREGISTER support to the user by way of workload-specific