1QEMU VM templating 2================== 3 4This document explains how to use VM templating in QEMU. 5 6For now, the focus is on VM memory aspects, and not about how to save and 7restore other VM state (i.e., migrate-to-file with ``x-ignore-shared``). 8 9Overview 10-------- 11 12With VM templating, a single template VM serves as the starting point for 13new VMs. This allows for fast and efficient replication of VMs, resulting 14in fast startup times and reduced memory consumption. 15 16Conceptually, the VM state is frozen, to then be used as a basis for new 17VMs. The Copy-On-Write mechanism in the operating systems makes sure that 18new VMs are able to read template VM memory; however, any modifications 19stay private and don't modify the original template VM or any other 20created VM. 21 22!!! Security Alert !!! 23---------------------- 24 25When effectively cloning VMs by VM templating, hardware identifiers 26(such as UUIDs and NIC MAC addresses), and similar data in the guest OS 27(such as machine IDs, SSH keys, certificates) that are supposed to be 28*unique* are no longer unique, which can be a security concern. 29 30Please be aware of these implications and how to mitigate them for your 31use case, which might involve vmgenid, hot(un)plug of NIC, etc.. 32 33Memory configuration 34-------------------- 35 36In order to create the template VM, we have to make sure that VM memory 37ends up in a file, from where it can be reused for the new VMs: 38 39Supply VM RAM via memory-backend-file, with ``share=on`` (modifications go 40to the file) and ``readonly=off`` (open the file writable). Note that 41``readonly=off`` is implicit. 42 43In the following command-line example, a 2GB VM is created, whereby VM RAM 44is to be stored in the ``template`` file. 45 46.. parsed-literal:: 47 48 |qemu_system| [...] -m 2g \\ 49 -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \\ 50 -machine q35,memory-backend=pc.ram 51 52If multiple memory backends are used (vNUMA, DIMMs), configure all 53memory backends accordingly. 54 55Once the VM is in the desired state, stop the VM and save other VM state, 56leaving the current state of VM RAM reside in the file. 57 58In order to have a new VM be based on a template VM, we have to 59configure VM RAM to be based on a template VM RAM file; however, the VM 60should not be able to modify file content. 61 62Supply VM RAM via memory-backend-file, with ``share=off`` (modifications 63stay private), ``readonly=on`` (open the file readonly) and ``rom=off`` 64(don't make the memory readonly for the VM). Note that ``share=off`` is 65implicit and that other VM state has to be restored separately. 66 67In the following command-line example, a 2GB VM is created based on the 68existing 2GB file ``template``. 69 70.. parsed-literal:: 71 72 |qemu_system| [...] -m 2g \\ 73 -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \\ 74 -machine q35,memory-backend=pc.ram 75 76If multiple memory backends are used (vNUMA, DIMMs), configure all 77memory backends accordingly. 78 79Note that ``-mem-path`` cannot be used for VM templating when creating the 80template VM or when starting new VMs based on a template VM. 81 82Incompatible features 83--------------------- 84 85Some features are incompatible with VM templating, as the underlying file 86cannot be modified to discard VM RAM, or to actually share memory with 87another process. 88 89vhost-user and multi-process QEMU 90~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 91 92vhost-user and multi-process QEMU are incompatible with VM templating. 93These technologies rely on shared memory, however, the template VMs 94don't actually share memory (``share=off``), even though they are 95file-based. 96 97virtio-balloon 98~~~~~~~~~~~~~~ 99 100virtio-balloon inflation and "free page reporting" cannot discard VM RAM 101and will repeatedly report errors. While virtio-balloon can be used 102for template VMs (e.g., report VM RAM stats), "free page reporting" 103should be disabled and the balloon should not be inflated. 104 105virtio-mem 106~~~~~~~~~~ 107 108virtio-mem cannot discard VM RAM that is managed by the virtio-mem 109device. virtio-mem will fail early when realizing the device. To use 110VM templating with virtio-mem, either hotplug virtio-mem devices to the 111new VM, or don't supply any memory to the template VM using virtio-mem 112(requested-size=0), not using a template VM file as memory backend for the 113virtio-mem device. 114 115VM migration 116~~~~~~~~~~~~ 117 118For VM migration, "x-release-ram" similarly relies on discarding of VM 119RAM on the migration source to free up migrated RAM, and will 120repeatedly report errors. 121 122Postcopy live migration fails discarding VM RAM on the migration 123destination early and refuses to activate postcopy live migration. Note 124that postcopy live migration usually only works on selected filesystems 125(shmem/tmpfs, hugetlbfs) either way. 126