xref: /openbmc/qemu/docs/devel/migration/main.rst (revision 3e964275)
1===================
2Migration framework
3===================
4
5QEMU has code to load/save the state of the guest that it is running.
6These are two complementary operations.  Saving the state just does
7that, saves the state for each device that the guest is running.
8Restoring a guest is just the opposite operation: we need to load the
9state of each device.
10
11For this to work, QEMU has to be launched with the same arguments the
12two times.  I.e. it can only restore the state in one guest that has
13the same devices that the one it was saved (this last requirement can
14be relaxed a bit, but for now we can consider that configuration has
15to be exactly the same).
16
17Once that we are able to save/restore a guest, a new functionality is
18requested: migration.  This means that QEMU is able to start in one
19machine and being "migrated" to another machine.  I.e. being moved to
20another machine.
21
22Next was the "live migration" functionality.  This is important
23because some guests run with a lot of state (specially RAM), and it
24can take a while to move all state from one machine to another.  Live
25migration allows the guest to continue running while the state is
26transferred.  Only while the last part of the state is transferred has
27the guest to be stopped.  Typically the time that the guest is
28unresponsive during live migration is the low hundred of milliseconds
29(notice that this depends on a lot of things).
30
31.. contents::
32
33Transports
34==========
35
36The migration stream is normally just a byte stream that can be passed
37over any transport.
38
39- tcp migration: do the migration using tcp sockets
40- unix migration: do the migration using unix sockets
41- exec migration: do the migration using the stdin/stdout through a process.
42- fd migration: do the migration using a file descriptor that is
43  passed to QEMU.  QEMU doesn't care how this file descriptor is opened.
44- file migration: do the migration using a file that is passed to QEMU
45  by path. A file offset option is supported to allow a management
46  application to add its own metadata to the start of the file without
47  QEMU interference. Note that QEMU does not flush cached file
48  data/metadata at the end of migration.
49
50In addition, support is included for migration using RDMA, which
51transports the page data using ``RDMA``, where the hardware takes care of
52transporting the pages, and the load on the CPU is much lower.  While the
53internals of RDMA migration are a bit different, this isn't really visible
54outside the RAM migration code.
55
56All these migration protocols use the same infrastructure to
57save/restore state devices.  This infrastructure is shared with the
58savevm/loadvm functionality.
59
60Common infrastructure
61=====================
62
63The files, sockets or fd's that carry the migration stream are abstracted by
64the  ``QEMUFile`` type (see ``migration/qemu-file.h``).  In most cases this
65is connected to a subtype of ``QIOChannel`` (see ``io/``).
66
67
68Saving the state of one device
69==============================
70
71For most devices, the state is saved in a single call to the migration
72infrastructure; these are *non-iterative* devices.  The data for these
73devices is sent at the end of precopy migration, when the CPUs are paused.
74There are also *iterative* devices, which contain a very large amount of
75data (e.g. RAM or large tables).  See the iterative device section below.
76
77General advice for device developers
78------------------------------------
79
80- The migration state saved should reflect the device being modelled rather
81  than the way your implementation works.  That way if you change the implementation
82  later the migration stream will stay compatible.  That model may include
83  internal state that's not directly visible in a register.
84
85- When saving a migration stream the device code may walk and check
86  the state of the device.  These checks might fail in various ways (e.g.
87  discovering internal state is corrupt or that the guest has done something bad).
88  Consider carefully before asserting/aborting at this point, since the
89  normal response from users is that *migration broke their VM* since it had
90  apparently been running fine until then.  In these error cases, the device
91  should log a message indicating the cause of error, and should consider
92  putting the device into an error state, allowing the rest of the VM to
93  continue execution.
94
95- The migration might happen at an inconvenient point,
96  e.g. right in the middle of the guest reprogramming the device, during
97  guest reboot or shutdown or while the device is waiting for external IO.
98  It's strongly preferred that migrations do not fail in this situation,
99  since in the cloud environment migrations might happen automatically to
100  VMs that the administrator doesn't directly control.
101
102- If you do need to fail a migration, ensure that sufficient information
103  is logged to identify what went wrong.
104
105- The destination should treat an incoming migration stream as hostile
106  (which we do to varying degrees in the existing code).  Check that offsets
107  into buffers and the like can't cause overruns.  Fail the incoming migration
108  in the case of a corrupted stream like this.
109
110- Take care with internal device state or behaviour that might become
111  migration version dependent.  For example, the order of PCI capabilities
112  is required to stay constant across migration.  Another example would
113  be that a special case handled by subsections (see below) might become
114  much more common if a default behaviour is changed.
115
116- The state of the source should not be changed or destroyed by the
117  outgoing migration.  Migrations timing out or being failed by
118  higher levels of management, or failures of the destination host are
119  not unusual, and in that case the VM is restarted on the source.
120  Note that the management layer can validly revert the migration
121  even though the QEMU level of migration has succeeded as long as it
122  does it before starting execution on the destination.
123
124- Buses and devices should be able to explicitly specify addresses when
125  instantiated, and management tools should use those.  For example,
126  when hot adding USB devices it's important to specify the ports
127  and addresses, since implicit ordering based on the command line order
128  may be different on the destination.  This can result in the
129  device state being loaded into the wrong device.
130
131VMState
132-------
133
134Most device data can be described using the ``VMSTATE`` macros (mostly defined
135in ``include/migration/vmstate.h``).
136
137An example (from hw/input/pckbd.c)
138
139.. code:: c
140
141  static const VMStateDescription vmstate_kbd = {
142      .name = "pckbd",
143      .version_id = 3,
144      .minimum_version_id = 3,
145      .fields = (const VMStateField[]) {
146          VMSTATE_UINT8(write_cmd, KBDState),
147          VMSTATE_UINT8(status, KBDState),
148          VMSTATE_UINT8(mode, KBDState),
149          VMSTATE_UINT8(pending, KBDState),
150          VMSTATE_END_OF_LIST()
151      }
152  };
153
154We are declaring the state with name "pckbd".  The ``version_id`` is
1553, and there are 4 uint8_t fields in the KBDState structure.  We
156registered this ``VMSTATEDescription`` with one of the following
157functions.  The first one will generate a device ``instance_id``
158different for each registration.  Use the second one if you already
159have an id that is different for each instance of the device:
160
161.. code:: c
162
163    vmstate_register_any(NULL, &vmstate_kbd, s);
164    vmstate_register(NULL, instance_id, &vmstate_kbd, s);
165
166For devices that are ``qdev`` based, we can register the device in the class
167init function:
168
169.. code:: c
170
171    dc->vmsd = &vmstate_kbd_isa;
172
173The VMState macros take care of ensuring that the device data section
174is formatted portably (normally big endian) and make some compile time checks
175against the types of the fields in the structures.
176
177VMState macros can include other VMStateDescriptions to store substructures
178(see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length
179arrays (``VMSTATE_VARRAY_``).  Various other macros exist for special
180cases.
181
182Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32
183ends up with a 4 byte bigendian representation on the wire; in the future
184it might be possible to use a more structured format.
185
186Legacy way
187----------
188
189This way is going to disappear as soon as all current users are ported to VMSTATE;
190although converting existing code can be tricky, and thus 'soon' is relative.
191
192Each device has to register two functions, one to save the state and
193another to load the state back.
194
195.. code:: c
196
197  int register_savevm_live(const char *idstr,
198                           int instance_id,
199                           int version_id,
200                           SaveVMHandlers *ops,
201                           void *opaque);
202
203Two functions in the ``ops`` structure are the ``save_state``
204and ``load_state`` functions.  Notice that ``load_state`` receives a version_id
205parameter to know what state format is receiving.  ``save_state`` doesn't
206have a version_id parameter because it always uses the latest version.
207
208Note that because the VMState macros still save the data in a raw
209format, in many cases it's possible to replace legacy code
210with a carefully constructed VMState description that matches the
211byte layout of the existing code.
212
213Changing migration data structures
214----------------------------------
215
216When we migrate a device, we save/load the state as a series
217of fields.  Sometimes, due to bugs or new functionality, we need to
218change the state to store more/different information.  Changing the migration
219state saved for a device can break migration compatibility unless
220care is taken to use the appropriate techniques.  In general QEMU tries
221to maintain forward migration compatibility (i.e. migrating from
222QEMU n->n+1) and there are users who benefit from backward compatibility
223as well.
224
225Subsections
226-----------
227
228The most common structure change is adding new data, e.g. when adding
229a newer form of device, or adding that state that you previously
230forgot to migrate.  This is best solved using a subsection.
231
232A subsection is "like" a device vmstate, but with a particularity, it
233has a Boolean function that tells if that values are needed to be sent
234or not.  If this functions returns false, the subsection is not sent.
235Subsections have a unique name, that is looked for on the receiving
236side.
237
238On the receiving side, if we found a subsection for a device that we
239don't understand, we just fail the migration.  If we understand all
240the subsections, then we load the state with success.  There's no check
241that a subsection is loaded, so a newer QEMU that knows about a subsection
242can (with care) load a stream from an older QEMU that didn't send
243the subsection.
244
245If the new data is only needed in a rare case, then the subsection
246can be made conditional on that case and the migration will still
247succeed to older QEMUs in most cases.  This is OK for data that's
248critical, but in some use cases it's preferred that the migration
249should succeed even with the data missing.  To support this the
250subsection can be connected to a device property and from there
251to a versioned machine type.
252
253The 'pre_load' and 'post_load' functions on subsections are only
254called if the subsection is loaded.
255
256One important note is that the outer post_load() function is called "after"
257loading all subsections, because a newer subsection could change the same
258value that it uses.  A flag, and the combination of outer pre_load and
259post_load can be used to detect whether a subsection was loaded, and to
260fall back on default behaviour when the subsection isn't present.
261
262Example:
263
264.. code:: c
265
266  static bool ide_drive_pio_state_needed(void *opaque)
267  {
268      IDEState *s = opaque;
269
270      return ((s->status & DRQ_STAT) != 0)
271          || (s->bus->error_status & BM_STATUS_PIO_RETRY);
272  }
273
274  const VMStateDescription vmstate_ide_drive_pio_state = {
275      .name = "ide_drive/pio_state",
276      .version_id = 1,
277      .minimum_version_id = 1,
278      .pre_save = ide_drive_pio_pre_save,
279      .post_load = ide_drive_pio_post_load,
280      .needed = ide_drive_pio_state_needed,
281      .fields = (const VMStateField[]) {
282          VMSTATE_INT32(req_nb_sectors, IDEState),
283          VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1,
284                               vmstate_info_uint8, uint8_t),
285          VMSTATE_INT32(cur_io_buffer_offset, IDEState),
286          VMSTATE_INT32(cur_io_buffer_len, IDEState),
287          VMSTATE_UINT8(end_transfer_fn_idx, IDEState),
288          VMSTATE_INT32(elementary_transfer_size, IDEState),
289          VMSTATE_INT32(packet_transfer_size, IDEState),
290          VMSTATE_END_OF_LIST()
291      }
292  };
293
294  const VMStateDescription vmstate_ide_drive = {
295      .name = "ide_drive",
296      .version_id = 3,
297      .minimum_version_id = 0,
298      .post_load = ide_drive_post_load,
299      .fields = (const VMStateField[]) {
300          .... several fields ....
301          VMSTATE_END_OF_LIST()
302      },
303      .subsections = (const VMStateDescription * const []) {
304          &vmstate_ide_drive_pio_state,
305          NULL
306      }
307  };
308
309Here we have a subsection for the pio state.  We only need to
310save/send this state when we are in the middle of a pio operation
311(that is what ``ide_drive_pio_state_needed()`` checks).  If DRQ_STAT is
312not enabled, the values on that fields are garbage and don't need to
313be sent.
314
315Connecting subsections to properties
316------------------------------------
317
318Using a condition function that checks a 'property' to determine whether
319to send a subsection allows backward migration compatibility when
320new subsections are added, especially when combined with versioned
321machine types.
322
323For example:
324
325   a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and
326      default it to true.
327   b) Add an entry to the ``hw_compat_`` for the previous version that sets
328      the property to false.
329   c) Add a static bool  support_foo function that tests the property.
330   d) Add a subsection with a .needed set to the support_foo function
331   e) (potentially) Add an outer pre_load that sets up a default value
332      for 'foo' to be used if the subsection isn't loaded.
333
334Now that subsection will not be generated when using an older
335machine type and the migration stream will be accepted by older
336QEMU versions.
337
338Not sending existing elements
339-----------------------------
340
341Sometimes members of the VMState are no longer needed:
342
343  - removing them will break migration compatibility
344
345  - making them version dependent and bumping the version will break backward migration
346    compatibility.
347
348Adding a dummy field into the migration stream is normally the best way to preserve
349compatibility.
350
351If the field really does need to be removed then:
352
353  a) Add a new property/compatibility/function in the same way for subsections above.
354  b) replace the VMSTATE macro with the _TEST version of the macro, e.g.:
355
356   ``VMSTATE_UINT32(foo, barstruct)``
357
358   becomes
359
360   ``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)``
361
362   Sometime in the future when we no longer care about the ancient versions these can be killed off.
363   Note that for backward compatibility it's important to fill in the structure with
364   data that the destination will understand.
365
366Any difference in the predicates on the source and destination will end up
367with different fields being enabled and data being loaded into the wrong
368fields; for this reason conditional fields like this are very fragile.
369
370Versions
371--------
372
373Version numbers are intended for major incompatible changes to the
374migration of a device, and using them breaks backward-migration
375compatibility; in general most changes can be made by adding Subsections
376(see above) or _TEST macros (see above) which won't break compatibility.
377
378Each version is associated with a series of fields saved.  The ``save_state`` always saves
379the state as the newer version.  But ``load_state`` sometimes is able to
380load state from an older version.
381
382You can see that there are two version fields:
383
384- ``version_id``: the maximum version_id supported by VMState for that device.
385- ``minimum_version_id``: the minimum version_id that VMState is able to understand
386  for that device.
387
388VMState is able to read versions from minimum_version_id to version_id.
389
390There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields,
391e.g.
392
393.. code:: c
394
395   VMSTATE_UINT16_V(ip_id, Slirp, 2),
396
397only loads that field for versions 2 and newer.
398
399Saving state will always create a section with the 'version_id' value
400and thus can't be loaded by any older QEMU.
401
402Massaging functions
403-------------------
404
405Sometimes, it is not enough to be able to save the state directly
406from one structure, we need to fill the correct values there.  One
407example is when we are using kvm.  Before saving the cpu state, we
408need to ask kvm to copy to QEMU the state that it is using.  And the
409opposite when we are loading the state, we need a way to tell kvm to
410load the state for the cpu that we have just loaded from the QEMUFile.
411
412The functions to do that are inside a vmstate definition, and are called:
413
414- ``int (*pre_load)(void *opaque);``
415
416  This function is called before we load the state of one device.
417
418- ``int (*post_load)(void *opaque, int version_id);``
419
420  This function is called after we load the state of one device.
421
422- ``int (*pre_save)(void *opaque);``
423
424  This function is called before we save the state of one device.
425
426- ``int (*post_save)(void *opaque);``
427
428  This function is called after we save the state of one device
429  (even upon failure, unless the call to pre_save returned an error).
430
431Example: You can look at hpet.c, that uses the first three functions
432to massage the state that is transferred.
433
434The ``VMSTATE_WITH_TMP`` macro may be useful when the migration
435data doesn't match the stored device data well; it allows an
436intermediate temporary structure to be populated with migration
437data and then transferred to the main structure.
438
439If you use memory or portio_list API functions that update memory layout outside
440initialization (i.e., in response to a guest action), this is a strong
441indication that you need to call these functions in a ``post_load`` callback.
442Examples of such API functions are:
443
444  - memory_region_add_subregion()
445  - memory_region_del_subregion()
446  - memory_region_set_readonly()
447  - memory_region_set_nonvolatile()
448  - memory_region_set_enabled()
449  - memory_region_set_address()
450  - memory_region_set_alias_offset()
451  - portio_list_set_address()
452  - portio_list_set_enabled()
453
454Iterative device migration
455--------------------------
456
457Some devices, such as RAM or certain platform devices,
458have large amounts of data that would mean that the CPUs would be
459paused for too long if they were sent in one section.  For these
460devices an *iterative* approach is taken.
461
462The iterative devices generally don't use VMState macros
463(although it may be possible in some cases) and instead use
464qemu_put_*/qemu_get_* macros to read/write data to the stream.  Specialist
465versions exist for high bandwidth IO.
466
467
468An iterative device must provide:
469
470  - A ``save_setup`` function that initialises the data structures and
471    transmits a first section containing information on the device.  In the
472    case of RAM this transmits a list of RAMBlocks and sizes.
473
474  - A ``load_setup`` function that initialises the data structures on the
475    destination.
476
477  - A ``state_pending_exact`` function that indicates how much more
478    data we must save.  The core migration code will use this to
479    determine when to pause the CPUs and complete the migration.
480
481  - A ``state_pending_estimate`` function that indicates how much more
482    data we must save.  When the estimated amount is smaller than the
483    threshold, we call ``state_pending_exact``.
484
485  - A ``save_live_iterate`` function should send a chunk of data until
486    the point that stream bandwidth limits tell it to stop.  Each call
487    generates one section.
488
489  - A ``save_live_complete_precopy`` function that must transmit the
490    last section for the device containing any remaining data.
491
492  - A ``load_state`` function used to load sections generated by
493    any of the save functions that generate sections.
494
495  - ``cleanup`` functions for both save and load that are called
496    at the end of migration.
497
498Note that the contents of the sections for iterative migration tend
499to be open-coded by the devices; care should be taken in parsing
500the results and structuring the stream to make them easy to validate.
501
502Device ordering
503---------------
504
505There are cases in which the ordering of device loading matters; for
506example in some systems where a device may assert an interrupt during loading,
507if the interrupt controller is loaded later then it might lose the state.
508
509Some ordering is implicitly provided by the order in which the machine
510definition creates devices, however this is somewhat fragile.
511
512The ``MigrationPriority`` enum provides a means of explicitly enforcing
513ordering.  Numerically higher priorities are loaded earlier.
514The priority is set by setting the ``priority`` field of the top level
515``VMStateDescription`` for the device.
516
517Stream structure
518================
519
520The stream tries to be word and endian agnostic, allowing migration between hosts
521of different characteristics running the same VM.
522
523  - Header
524
525    - Magic
526    - Version
527    - VM configuration section
528
529       - Machine type
530       - Target page bits
531  - List of sections
532    Each section contains a device, or one iteration of a device save.
533
534    - section type
535    - section id
536    - ID string (First section of each device)
537    - instance id (First section of each device)
538    - version id (First section of each device)
539    - <device data>
540    - Footer mark
541  - EOF mark
542  - VM Description structure
543    Consisting of a JSON description of the contents for analysis only
544
545The ``device data`` in each section consists of the data produced
546by the code described above.  For non-iterative devices they have a single
547section; iterative devices have an initial and last section and a set
548of parts in between.
549Note that there is very little checking by the common code of the integrity
550of the ``device data`` contents, that's up to the devices themselves.
551The ``footer mark`` provides a little bit of protection for the case where
552the receiving side reads more or less data than expected.
553
554The ``ID string`` is normally unique, having been formed from a bus name
555and device address, PCI devices and storage devices hung off PCI controllers
556fit this pattern well.  Some devices are fixed single instances (e.g. "pc-ram").
557Others (especially either older devices or system devices which for
558some reason don't have a bus concept) make use of the ``instance id``
559for otherwise identically named devices.
560
561Return path
562-----------
563
564Only a unidirectional stream is required for normal migration, however a
565``return path`` can be created when bidirectional communication is desired.
566This is primarily used by postcopy, but is also used to return a success
567flag to the source at the end of migration.
568
569``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return
570path.
571
572  Source side
573
574     Forward path - written by migration thread
575     Return path  - opened by main thread, read by return-path thread
576
577  Destination side
578
579     Forward path - read by main thread
580     Return path  - opened by main thread, written by main thread AND postcopy
581     thread (protected by rp_mutex)
582
583