qemu/docs/rdma.txt

31 data copies by bypassing the host networking stack. In particular, a TCP-based
32 migration, under certain types of memory-bound workloads, may take a more
38 over Converged Ethernet) as well as Infiniband-based. This implementation of
55 If this is not acceptable for your application or product, then the use
56 of RDMA migration may in fact be harmful to co-located VMs or other
57 software on the machine if there is not sufficient memory available to
65 bulk-phase round of the migration and can be enabled for extremely
66 high-performance RDMA hardware using the following command:
69 $ migrate_set_capability rdma-pin-all on # disabled by default
72 not what you want, then please ignore this step altogether.
82 and thus extending the total migration time. However, this will not
92 $ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your RDMA device
96 qemu ..... -incoming rdma:host:port
101 $ migrate -d rdma:host:port
107 Using a 40gbps infiniband link performing a worst-case stress test,
111 $ apt-get install stress
112 $ stress --vm-bytes 7500M --vm 1 --vm-keep
123 1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
124 2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
129 Enabling this feature does *not* have any measurable affect on
132 the bulk round and does not need to be re-registered during the successive
144 protocol now, consisting of infiniband SEND messages.
146 An infiniband SEND message is the standard ibverbs
148 The only difference between a SEND message and an RDMA
149 message is that SEND messages cause notifications
152 for VM's ram) do not (to behave like an actual DMA).
157 2. (SEND only) work requests to be posted on both
166 (Memory is not released from pinning until the migration
169 SEND messages require more coordination because the
176 as follows (migration-rdma.c):
185 At this point, we define a control channel on top of SEND messages
186 which is described by a formal protocol. Each SEND message has a
188 as a single SEND message).
201 The maximum number of repeats is hard-coded to 4096. This is a conservative
202 limit based on the maximum size of a SEND message along with empirical
208      3. Ready                      (control-channel is available)
209      4. QEMU File                  (for sending non-live device state)
226 After ram block exchange is completed, we have two protocol-level
227 functions, responsible for communicating control-channel commands
238 3. Block on a CQ event channel and wait for the SEND to arrive.
239 4. When the send arrives, librdmacm will unblock us.
240 5. Verify that the command-type and version received matches the one we expected.
248    (that we have not yet transmitted), let's post an RQ
253 4. Now, we can actually post the work request to SEND
263 All of the remaining command types (not including 'ready')
277    when transmitting non-live state, such as devices or to send
279 4. Finally, zero pages are only checked if a page has not yet been registered
280    using chunk registration (or not checked at all and unconditionally
284    zero, then we send a compress command to zap the page on the other side.
295 at connection-setup time before any infiniband traffic is generated.
298     * Version (protocol version validated before send/recv occurs),
305 is only 192 bytes per the Infiniband specification, so it's not
309 versioning because the user does not need to register memory to
322 Finally: Negotiation happens with the Flags field: If the primary-VM
323 sets a flag, but the destination does not support this capability, it
324 will return a zero-bit for that flag and the primary-VM will understand
325 that as not being an available capability and will thus disable that
326 capability on the primary-VM side.
337 describe above to deliver bytes without changing the upper-level
343 using an analogy not unlike individual UDP frames, we have
344 to hold on to the bytes received from control-channel's SEND
347 Each time we receive a complete "QEMU File" control-channel
348 message, the bytes from SEND are copied into a small local holding area.
354 If the buffer is empty, then we follow the same steps
356 asking for a new SEND message to re-fill the buffer.
361 At the beginning of the migration, (migration-rdma.c),
368 addresses and possibly includes pre-registered RDMA keys in case dynamic
369 page registration was disabled on the server-side, otherwise not.
371 Main memory is not migrated with the aforementioned protocol,
374 Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now).
375 Chunk size is not dynamic, but it could be in a future implementation.
385 do not request that the hardware signal the completion queue
392 Error-handling:
406 socket is broken during a non-RDMA based migration.
410 1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits
411    are not compatible with infiniband memory pinning and will result in
415 3. Also, some form of balloon-device usage tracking would also
417 4. Use LRU to provide more fine-grained direction of UNREGISTER
419 5. Expose UNREGISTER support to the user by way of workload-specific