1Mapped-ram
2==========
3
4Mapped-ram is a new stream format for the RAM section designed to
5supplement the existing ``file:`` migration and make it compatible
6with ``multifd``. This enables parallel migration of a guest's RAM to
7a file.
8
9The core of the feature is to ensure that RAM pages are mapped
10directly to offsets in the resulting migration file. This enables the
11``multifd`` threads to write exclusively to those offsets even if the
12guest is constantly dirtying pages (i.e. live migration). Another
13benefit is that the resulting file will have a bounded size, since
14pages which are dirtied multiple times will always go to a fixed
15location in the file, rather than constantly being added to a
16sequential stream. Having the pages at fixed offsets also allows the
17usage of O_DIRECT for save/restore of the migration stream as the
18pages are ensured to be written respecting O_DIRECT alignment
19restrictions.
20
21Usage
22-----
23
24On both source and destination, enable the ``multifd`` and
25``mapped-ram`` capabilities:
26
27    ``migrate_set_capability multifd on``
28
29    ``migrate_set_capability mapped-ram on``
30
31Use a ``file:`` URL for migration:
32
33    ``migrate file:/path/to/migration/file``
34
35Mapped-ram migration is best done non-live, i.e. by stopping the VM on
36the source side before migrating.
37
38For best performance enable the ``direct-io`` parameter as well:
39
40    ``migrate_set_parameter direct-io on``
41
42Use-cases
43---------
44
45The mapped-ram feature was designed for use cases where the migration
46stream will be directed to a file in the filesystem and not
47immediately restored on the destination VM\ [#alternatives]_. These could be
48thought of as snapshots. We can further categorize them into live and
49non-live.
50
51- Non-live snapshot
52
53If the use case requires a VM to be stopped before taking a snapshot,
54that's the ideal scenario for mapped-ram migration. Not having to
55track dirty pages, the migration will write the RAM pages to the disk
56as fast as it can.
57
58Note: if a snapshot is taken of a running VM, but the VM will be
59stopped after the snapshot by the admin, then consider stopping it
60right before the snapshot to take benefit of the performance gains
61mentioned above.
62
63- Live snapshot
64
65If the use case requires that the VM keeps running during and after
66the snapshot operation, then mapped-ram migration can still be used,
67but will be less performant. Other strategies such as
68background-snapshot should be evaluated as well. One benefit of
69mapped-ram in this scenario is portability since background-snapshot
70depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not
71supported outside of Linux.
72
73.. [#alternatives] While this same effect could be obtained with the usage of
74       snapshots or the ``file:`` migration alone, mapped-ram provides
75       a performance increase for VMs with larger RAM sizes (10s to
76       100s of GiBs), specially if the VM has been stopped beforehand.
77
78RAM section format
79------------------
80
81Instead of having a sequential stream of pages that follow the
82RAMBlock headers, the dirty pages for a RAMBlock follow its header
83instead. This ensures that each RAM page has a fixed offset in the
84resulting migration file.
85
86A bitmap is introduced to track which pages have been written in the
87migration file. Pages are written at a fixed location for every
88ramblock. Zero pages are ignored as they'd be zero in the destination
89migration as well.
90
91::
92
93 Without mapped-ram:                  With mapped-ram:
94
95 ---------------------               --------------------------------
96 | ramblock 1 header |               | ramblock 1 header            |
97 ---------------------               --------------------------------
98 | ramblock 2 header |               | ramblock 1 mapped-ram header |
99 ---------------------               --------------------------------
100 | ...               |               | padding to next 1MB boundary |
101 ---------------------               | ...                          |
102 | ramblock n header |               --------------------------------
103 ---------------------               | ramblock 1 pages             |
104 | RAM_SAVE_FLAG_EOS |               | ...                          |
105 ---------------------               --------------------------------
106 | stream of pages   |               | ramblock 2 header            |
107 | (iter 1)          |               --------------------------------
108 | ...               |               | ramblock 2 mapped-ram header |
109 ---------------------               --------------------------------
110 | RAM_SAVE_FLAG_EOS |               | padding to next 1MB boundary |
111 ---------------------               | ...                          |
112 | stream of pages   |               --------------------------------
113 | (iter 2)          |               | ramblock 2 pages             |
114 | ...               |               | ...                          |
115 ---------------------               --------------------------------
116 | ...               |               | ...                          |
117 ---------------------               --------------------------------
118                                     | RAM_SAVE_FLAG_EOS            |
119                                     --------------------------------
120                                     | ...                          |
121                                     --------------------------------
122
123where:
124 - ramblock header: the generic information for a ramblock, such as
125   idstr, used_len, etc.
126
127 - ramblock mapped-ram header: the information added by this feature:
128   bitmap of pages written, bitmap size and offset of pages in the
129   migration file.
130
131Restrictions
132------------
133
134Since pages are written to their relative offsets and out of order
135(due to the memory dirtying patterns), streaming channels such as
136sockets are not supported. A seekable channel such as a file is
137required. This can be verified in the QIOChannel by the presence of
138the QIO_CHANNEL_FEATURE_SEEKABLE.
139
140The improvements brought by this feature apply only to guest physical
141RAM. Other types of memory such as VRAM are migrated as part of device
142states.
143