1Mapped-ram
2==========
3
4Mapped-ram is a new stream format for the RAM section designed to
5supplement the existing ``file:`` migration and make it compatible
6with ``multifd``. This enables parallel migration of a guest's RAM to
7a file.
8
9The core of the feature is to ensure that RAM pages are mapped
10directly to offsets in the resulting migration file. This enables the
11``multifd`` threads to write exclusively to those offsets even if the
12guest is constantly dirtying pages (i.e. live migration). Another
13benefit is that the resulting file will have a bounded size, since
14pages which are dirtied multiple times will always go to a fixed
15location in the file, rather than constantly being added to a
16sequential stream. Having the pages at fixed offsets also allows the
17usage of O_DIRECT for save/restore of the migration stream as the
18pages are ensured to be written respecting O_DIRECT alignment
19restrictions (direct-io support not yet implemented).
20
21Usage
22-----
23
24On both source and destination, enable the ``multifd`` and
25``mapped-ram`` capabilities:
26
27    ``migrate_set_capability multifd on``
28
29    ``migrate_set_capability mapped-ram on``
30
31Use a ``file:`` URL for migration:
32
33    ``migrate file:/path/to/migration/file``
34
35Mapped-ram migration is best done non-live, i.e. by stopping the VM on
36the source side before migrating.
37
38Use-cases
39---------
40
41The mapped-ram feature was designed for use cases where the migration
42stream will be directed to a file in the filesystem and not
43immediately restored on the destination VM [#]_. These could be
44thought of as snapshots. We can further categorize them into live and
45non-live.
46
47- Non-live snapshot
48
49If the use case requires a VM to be stopped before taking a snapshot,
50that's the ideal scenario for mapped-ram migration. Not having to
51track dirty pages, the migration will write the RAM pages to the disk
52as fast as it can.
53
54Note: if a snapshot is taken of a running VM, but the VM will be
55stopped after the snapshot by the admin, then consider stopping it
56right before the snapshot to take benefit of the performance gains
57mentioned above.
58
59- Live snapshot
60
61If the use case requires that the VM keeps running during and after
62the snapshot operation, then mapped-ram migration can still be used,
63but will be less performant. Other strategies such as
64background-snapshot should be evaluated as well. One benefit of
65mapped-ram in this scenario is portability since background-snapshot
66depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not
67supported outside of Linux.
68
69.. [#] While this same effect could be obtained with the usage of
70       snapshots or the ``file:`` migration alone, mapped-ram provides
71       a performance increase for VMs with larger RAM sizes (10s to
72       100s of GiBs), specially if the VM has been stopped beforehand.
73
74RAM section format
75------------------
76
77Instead of having a sequential stream of pages that follow the
78RAMBlock headers, the dirty pages for a RAMBlock follow its header
79instead. This ensures that each RAM page has a fixed offset in the
80resulting migration file.
81
82A bitmap is introduced to track which pages have been written in the
83migration file. Pages are written at a fixed location for every
84ramblock. Zero pages are ignored as they'd be zero in the destination
85migration as well.
86
87::
88
89 Without mapped-ram:                  With mapped-ram:
90
91 ---------------------               --------------------------------
92 | ramblock 1 header |               | ramblock 1 header            |
93 ---------------------               --------------------------------
94 | ramblock 2 header |               | ramblock 1 mapped-ram header |
95 ---------------------               --------------------------------
96 | ...               |               | padding to next 1MB boundary |
97 ---------------------               | ...                          |
98 | ramblock n header |               --------------------------------
99 ---------------------               | ramblock 1 pages             |
100 | RAM_SAVE_FLAG_EOS |               | ...                          |
101 ---------------------               --------------------------------
102 | stream of pages   |               | ramblock 2 header            |
103 | (iter 1)          |               --------------------------------
104 | ...               |               | ramblock 2 mapped-ram header |
105 ---------------------               --------------------------------
106 | RAM_SAVE_FLAG_EOS |               | padding to next 1MB boundary |
107 ---------------------               | ...                          |
108 | stream of pages   |               --------------------------------
109 | (iter 2)          |               | ramblock 2 pages             |
110 | ...               |               | ...                          |
111 ---------------------               --------------------------------
112 | ...               |               | ...                          |
113 ---------------------               --------------------------------
114                                     | RAM_SAVE_FLAG_EOS            |
115                                     --------------------------------
116                                     | ...                          |
117                                     --------------------------------
118
119where:
120 - ramblock header: the generic information for a ramblock, such as
121   idstr, used_len, etc.
122
123 - ramblock mapped-ram header: the information added by this feature:
124   bitmap of pages written, bitmap size and offset of pages in the
125   migration file.
126
127Restrictions
128------------
129
130Since pages are written to their relative offsets and out of order
131(due to the memory dirtying patterns), streaming channels such as
132sockets are not supported. A seekable channel such as a file is
133required. This can be verified in the QIOChannel by the presence of
134the QIO_CHANNEL_FEATURE_SEEKABLE.
135
136The improvements brought by this feature apply only to guest physical
137RAM. Other types of memory such as VRAM are migrated as part of device
138states.
139