1Mapped-ram 2========== 3 4Mapped-ram is a new stream format for the RAM section designed to 5supplement the existing ``file:`` migration and make it compatible 6with ``multifd``. This enables parallel migration of a guest's RAM to 7a file. 8 9The core of the feature is to ensure that RAM pages are mapped 10directly to offsets in the resulting migration file. This enables the 11``multifd`` threads to write exclusively to those offsets even if the 12guest is constantly dirtying pages (i.e. live migration). Another 13benefit is that the resulting file will have a bounded size, since 14pages which are dirtied multiple times will always go to a fixed 15location in the file, rather than constantly being added to a 16sequential stream. Having the pages at fixed offsets also allows the 17usage of O_DIRECT for save/restore of the migration stream as the 18pages are ensured to be written respecting O_DIRECT alignment 19restrictions. 20 21Usage 22----- 23 24On both source and destination, enable the ``multifd`` and 25``mapped-ram`` capabilities: 26 27 ``migrate_set_capability multifd on`` 28 29 ``migrate_set_capability mapped-ram on`` 30 31Use a ``file:`` URL for migration: 32 33 ``migrate file:/path/to/migration/file`` 34 35Mapped-ram migration is best done non-live, i.e. by stopping the VM on 36the source side before migrating. 37 38For best performance enable the ``direct-io`` parameter as well: 39 40 ``migrate_set_parameter direct-io on`` 41 42Use-cases 43--------- 44 45The mapped-ram feature was designed for use cases where the migration 46stream will be directed to a file in the filesystem and not 47immediately restored on the destination VM [#]_. These could be 48thought of as snapshots. We can further categorize them into live and 49non-live. 50 51- Non-live snapshot 52 53If the use case requires a VM to be stopped before taking a snapshot, 54that's the ideal scenario for mapped-ram migration. Not having to 55track dirty pages, the migration will write the RAM pages to the disk 56as fast as it can. 57 58Note: if a snapshot is taken of a running VM, but the VM will be 59stopped after the snapshot by the admin, then consider stopping it 60right before the snapshot to take benefit of the performance gains 61mentioned above. 62 63- Live snapshot 64 65If the use case requires that the VM keeps running during and after 66the snapshot operation, then mapped-ram migration can still be used, 67but will be less performant. Other strategies such as 68background-snapshot should be evaluated as well. One benefit of 69mapped-ram in this scenario is portability since background-snapshot 70depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not 71supported outside of Linux. 72 73.. [#] While this same effect could be obtained with the usage of 74 snapshots or the ``file:`` migration alone, mapped-ram provides 75 a performance increase for VMs with larger RAM sizes (10s to 76 100s of GiBs), specially if the VM has been stopped beforehand. 77 78RAM section format 79------------------ 80 81Instead of having a sequential stream of pages that follow the 82RAMBlock headers, the dirty pages for a RAMBlock follow its header 83instead. This ensures that each RAM page has a fixed offset in the 84resulting migration file. 85 86A bitmap is introduced to track which pages have been written in the 87migration file. Pages are written at a fixed location for every 88ramblock. Zero pages are ignored as they'd be zero in the destination 89migration as well. 90 91:: 92 93 Without mapped-ram: With mapped-ram: 94 95 --------------------- -------------------------------- 96 | ramblock 1 header | | ramblock 1 header | 97 --------------------- -------------------------------- 98 | ramblock 2 header | | ramblock 1 mapped-ram header | 99 --------------------- -------------------------------- 100 | ... | | padding to next 1MB boundary | 101 --------------------- | ... | 102 | ramblock n header | -------------------------------- 103 --------------------- | ramblock 1 pages | 104 | RAM_SAVE_FLAG_EOS | | ... | 105 --------------------- -------------------------------- 106 | stream of pages | | ramblock 2 header | 107 | (iter 1) | -------------------------------- 108 | ... | | ramblock 2 mapped-ram header | 109 --------------------- -------------------------------- 110 | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary | 111 --------------------- | ... | 112 | stream of pages | -------------------------------- 113 | (iter 2) | | ramblock 2 pages | 114 | ... | | ... | 115 --------------------- -------------------------------- 116 | ... | | ... | 117 --------------------- -------------------------------- 118 | RAM_SAVE_FLAG_EOS | 119 -------------------------------- 120 | ... | 121 -------------------------------- 122 123where: 124 - ramblock header: the generic information for a ramblock, such as 125 idstr, used_len, etc. 126 127 - ramblock mapped-ram header: the information added by this feature: 128 bitmap of pages written, bitmap size and offset of pages in the 129 migration file. 130 131Restrictions 132------------ 133 134Since pages are written to their relative offsets and out of order 135(due to the memory dirtying patterns), streaming channels such as 136sockets are not supported. A seekable channel such as a file is 137required. This can be verified in the QIOChannel by the presence of 138the QIO_CHANNEL_FEATURE_SEEKABLE. 139 140The improvements brought by this feature apply only to guest physical 141RAM. Other types of memory such as VRAM are migrated as part of device 142states. 143