1===================== 2VFIO device Migration 3===================== 4 5Migration of virtual machine involves saving the state for each device that 6the guest is running on source host and restoring this saved state on the 7destination host. This document details how saving and restoring of VFIO 8devices is done in QEMU. 9 10Migration of VFIO devices currently consists of a single stop-and-copy phase. 11During the stop-and-copy phase the guest is stopped and the entire VFIO device 12data is transferred to the destination. 13 14The pre-copy phase of migration is currently not supported for VFIO devices. 15Support for VFIO pre-copy will be added later on. 16 17Note that currently VFIO migration is supported only for a single device. This 18is due to VFIO migration's lack of P2P support. However, P2P support is planned 19to be added later on. 20 21A detailed description of the UAPI for VFIO device migration can be found in 22the comment for the ``vfio_device_mig_state`` structure in the header file 23linux-headers/linux/vfio.h. 24 25VFIO implements the device hooks for the iterative approach as follows: 26 27* A ``save_setup`` function that sets up migration on the source. 28 29* A ``load_setup`` function that sets the VFIO device on the destination in 30 _RESUMING state. 31 32* A ``state_pending_exact`` function that reads pending_bytes from the vendor 33 driver, which indicates the amount of data that the vendor driver has yet to 34 save for the VFIO device. 35 36* A ``save_state`` function to save the device config space if it is present. 37 38* A ``save_live_complete_precopy`` function that sets the VFIO device in 39 _STOP_COPY state and iteratively copies the data for the VFIO device until 40 the vendor driver indicates that no data remains. 41 42* A ``load_state`` function that loads the config section and the data 43 sections that are generated by the save functions above. 44 45* ``cleanup`` functions for both save and load that perform any migration 46 related cleanup. 47 48 49The VFIO migration code uses a VM state change handler to change the VFIO 50device state when the VM state changes from running to not-running, and 51vice versa. 52 53Similarly, a migration state change handler is used to trigger a transition of 54the VFIO device state when certain changes of the migration state occur. For 55example, the VFIO device state is transitioned back to _RUNNING in case a 56migration failed or was canceled. 57 58System memory dirty pages tracking 59---------------------------------- 60 61A ``log_global_start`` and ``log_global_stop`` memory listener callback informs 62the VFIO dirty tracking module to start and stop dirty page tracking. A 63``log_sync`` memory listener callback queries the dirty page bitmap from the 64dirty tracking module and marks system memory pages which were DMA-ed by the 65VFIO device as dirty. The dirty page bitmap is queried per container. 66 67Currently there are two ways dirty page tracking can be done: 68(1) Device dirty tracking: 69In this method the device is responsible to log and report its DMAs. This 70method can be used only if the device is capable of tracking its DMAs. 71Discovering device capability, starting and stopping dirty tracking, and 72syncing the dirty bitmaps from the device are done using the DMA logging uAPI. 73More info about the uAPI can be found in the comments of the 74``vfio_device_feature_dma_logging_control`` and 75``vfio_device_feature_dma_logging_report`` structures in the header file 76linux-headers/linux/vfio.h. 77 78(2) VFIO IOMMU module: 79In this method dirty tracking is done by IOMMU. However, there is currently no 80IOMMU support for dirty page tracking. For this reason, all pages are 81perpetually marked dirty, unless the device driver pins pages through external 82APIs in which case only those pinned pages are perpetually marked dirty. 83 84If the above two methods are not supported, all pages are perpetually marked 85dirty by QEMU. 86 87By default, dirty pages are tracked during pre-copy as well as stop-and-copy 88phase. So, a page marked as dirty will be copied to the destination in both 89phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can 90achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding 91dirty pages continuously, then it understands that even in stop-and-copy phase, 92it is likely to find dirty pages and can predict the downtime accordingly. 93 94QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` 95which disables querying the dirty bitmap during pre-copy phase. If it is set to 96off, all dirty pages will be copied to the destination in stop-and-copy phase 97only. 98 99System memory dirty pages tracking when vIOMMU is enabled 100--------------------------------------------------------- 101 102With vIOMMU, an IO virtual address range can get unmapped while in pre-copy 103phase of migration. In that case, the unmap ioctl returns any dirty pages in 104that range and QEMU reports corresponding guest physical pages dirty. During 105stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped 106pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those 107mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration 108will be blocked. 109 110Flow of state changes during Live migration 111=========================================== 112 113Below is the flow of state change during live migration. 114The values in the brackets represent the VM state, the migration state, and 115the VFIO device state, respectively. 116 117Live migration save path 118------------------------ 119 120:: 121 122 QEMU normal running state 123 (RUNNING, _NONE, _RUNNING) 124 | 125 migrate_init spawns migration_thread 126 Migration thread then calls each device's .save_setup() 127 (RUNNING, _SETUP, _RUNNING) 128 | 129 (RUNNING, _ACTIVE, _RUNNING) 130 If device is active, get pending_bytes by .state_pending_exact() 131 If total pending_bytes >= threshold_size, call .save_live_iterate() 132 Iterate till total pending bytes converge and are less than threshold 133 | 134 On migration completion, vCPU stops and calls .save_live_complete_precopy for 135 each active device. The VFIO device is then transitioned into _STOP_COPY state 136 (FINISH_MIGRATE, _DEVICE, _STOP_COPY) 137 | 138 For the VFIO device, iterate in .save_live_complete_precopy until 139 pending data is 0 140 (FINISH_MIGRATE, _DEVICE, _STOP) 141 | 142 (FINISH_MIGRATE, _COMPLETED, _STOP) 143 Migraton thread schedules cleanup bottom half and exits 144 145Live migration resume path 146-------------------------- 147 148:: 149 150 Incoming migration calls .load_setup for each device 151 (RESTORE_VM, _ACTIVE, _STOP) 152 | 153 For each device, .load_state is called for that device section data 154 (RESTORE_VM, _ACTIVE, _RESUMING) 155 | 156 At the end, .load_cleanup is called for each device and vCPUs are started 157 (RUNNING, _NONE, _RUNNING) 158 159Postcopy 160======== 161 162Postcopy migration is currently not supported for VFIO devices. 163