Lines Matching +full:data +full:- +full:mirror
5 later. See the COPYING file in the top-level directory.
14 live block device jobs -- stream, commit, mirror, and backup. These can
16 namely: live copy data from backing files into overlays; shorten long
17 disk image chains by merging data from overlays into backing files; live
18 synchronize data from a disk image chain (including current active disk)
19 to another target image; and point-in-time (and incremental) backups of
21 primitives, and some (non-exhaustive list of) examples to illustrate
25 The file ``qapi/block-core.json`` in the QEMU source tree has the
35 ---------------------------------
38 ``blockdev-snapshot-sync``, or offline via ``qemu-img``)::
45 [A] <----- [B]
59 block-core.json.)
74 -------------------------------------------
79 (1) ``block-stream``: Live copy of data from backing files into overlay
86 reference to the now-streamed and redundant backing
92 (c) the streamed file remains valid -- i.e. further
94 ``block-stream`` section further below for more
97 (2) ``block-commit``: Live merge of data from overlay files into backing
99 chain). Since QEMU 2.0, this includes "active ``block-commit``"
106 to now-redundant overlay images that have been
110 -- it ought to be manually removed;
112 (c) however, unlike in the case of ``block-stream``, the
113 intermediate images will be rendered invalid -- i.e.
115 them. Refer the ``block-commit`` section further
118 (3) ``drive-mirror`` (and ``blockdev-mirror``): Synchronize a running
121 (4) ``blockdev-backup`` (and the deprecated ``drive-backup``):
122 Point-in-time (live) copy of a block device to a destination.
128 --------------------------------
130 To show some example invocations of command-line, we will use the
134 .. parsed-literal::
136 $ |qemu_system| -display none -no-user-config -nodefaults \\
137 -m 512 -blockdev \\
138 node-name=node-A,driver=qcow2,file.driver=file,file.node-name=file,file.filename=./a.qcow2 \\
139 -device virtio-blk,drive=node-A,id=virtio0 \\
140 -monitor stdio -qmp unix:/tmp/qmp-sock,server=on,wait=off
142 The ``-blockdev`` command-line option, used above, is available from
143 QEMU 2.9 onwards. In the above invocation, notice the ``node-name``
144 parameter that is used to refer to the disk image a.qcow2 ('node-A') --
147 ``node-name`` to each further disk image created (either via
148 ``blockdev-snapshot-sync``, or ``blockdev-add``) as part of the disk
150 ``node-name`` (where possible, because ``block-commit`` does not yet, as
151 of QEMU 2.9, accept ``node-name`` parameter) when performing various
155 ``qmp-shell`` utility (located at: ``qemu/scripts/qmp``, as part of the
156 QEMU source directory), which takes key-value pairs for QMP commands.
158 syntax for reference -- examples in the following sections)::
160 $ ./qmp-shell -v -p /tmp/qmp-sock
165 the first occurrence of it, show the ``qmp-shell`` invocation, *and*
167 invocations, present just the ``qmp-shell`` syntax, and omit the
172 ------------------------
177 [A] <-- [B] <-- [C] <-- [D]
180 overlay images; image [D] is the active layer -- i.e. live QEMU is
185 ``blockdev-snapshot-sync`` commands as following (which shows the
186 creation of overlay image [B]) using the ``qmp-shell`` (our invocation
189 …(QEMU) blockdev-snapshot-sync node-name=node-A snapshot-file=b.qcow2 snapshot-node-name=node-B for…
191 "execute": "blockdev-snapshot-sync",
193 "node-name": "node-A",
194 "snapshot-file": "b.qcow2",
196 "snapshot-node-name": "node-B"
200 Here, "node-A" is the name QEMU internally uses to refer to the base
201 image [A] -- it is the backing file, based on which the overlay image,
207 …(QEMU) blockdev-snapshot-sync node-name=node-B snapshot-file=c.qcow2 snapshot-node-name=node-C for…
208 …(QEMU) blockdev-snapshot-sync node-name=node-C snapshot-file=d.qcow2 snapshot-node-name=node-D for…
211 A note on points-in-time vs file names
212 --------------------------------------
216 [A] <-- [B] <-- [C] <-- [D]
220 - Point 1: Guest state when [B] was created is contained in file [A]
221 - Point 2: Guest state when [C] was created is contained in [A] + [B]
222 - Point 3: Guest state when [D] was created is contained in
224 - Active layer: Current guest state is contained in [A] + [B] + [C] +
229 - Naming a file after the time it is created is misleading -- the
230 guest data for that point in time is *not* contained in that file
232 - Rather, think of files as a *delta* from the backing file
235 Live block streaming --- ``block-stream``
236 -----------------------------------------
238 The ``block-stream`` command allows you to do live copy data from backing
243 [A] <-- [B] <-- [C] <-- [D]
248 .. _`Case-1`:
253 standalone image, [D] -- with contents from [A], [B] and [C] merged
258 .. _`Case-2`:
266 [A] <-- [D]
268 .. _`Case-3`:
279 [A] <-- [C] <-- [D]
282 QMP invocation for ``block-stream``
285 For `Case-1`_, to merge contents of all the backing files into the
286 active layer, where 'node-D' is the current active image (by default
287 ``block-stream`` will flatten the entire chain); ``qmp-shell`` (and its
290 (QEMU) block-stream device=node-D job-id=job0
292 "execute": "block-stream",
294 "device": "node-D",
295 "job-id": "job0"
299 For `Case-2`_, merge contents of the images [B] and [C] into [D], where
302 (QEMU) block-stream device=node-D base-node=node-A job-id=job0
304 And for `Case-3`_, of "intermediate" streaming", merge contents of
308 (QEMU) block-stream device=node-C base-node=node-A job-id=job0
310 Progress of a ``block-stream`` operation can be monitored via the QMP
313 (QEMU) query-block-jobs
315 "execute": "query-block-jobs",
320 Once the ``block-stream`` operation has completed, QEMU will emit an
323 overlays based on them. Finally, the ``block-stream`` jobs can be
327 Live block commit --- ``block-commit``
328 --------------------------------------
330 The ``block-commit`` command lets you merge live data from overlay
332 commit" (i.e. it is possible to merge the "active layer", the right-most
334 base image). This is analogous to ``block-stream``, but in the opposite
338 QEMU is writing to the right-most image in the chain, [D]::
340 [A] <-- [B] <-- [C] <-- [D]
344 .. _`block-commit_Case-1`:
350 [A] <-- [C] <-- [D]
356 [A] <-- [D]
358 .. _`block-commit_Case-3`:
369 [A] <-- [B] <-- [D]
374 [A] <-- [B]
377 QMP invocation for ``block-commit``
380 For :ref:`Case-1 <block-commit_Case-1>`, to merge contents only from
383 (QEMU) block-commit device=node-D base=a.qcow2 top=b.qcow2 job-id=job0
385 "execute": "block-commit",
387 "device": "node-D",
388 "job-id": "job0",
394 Once the above ``block-commit`` operation has completed, a
397 to point to image [A], and the original 4-image chain will end up being
400 [A] <-- [C] <-- [D]
407 represents that old point-in-time, and may be valid in that context.
409 represents any point-in-time, and is invalid in any context.
412 However, :ref:`Case-3 <block-commit_Case-3>` (also called: "active
413 ``block-commit``") is a *two-phase* operation: In the first phase, the
417 -- possible via issuing the command ``block-job-complete``. Optionally,
418 the ``block-commit`` operation can be cancelled by issuing the command
419 ``block-job-cancel``, but be careful when doing this.
421 Once the ``block-commit`` operation has completed, the event
424 command ``block-job-complete`` -- until such a command is issued, the
427 The following is the flow for :ref:`Case-3 <block-commit_Case-3>` to
430 [A] <-- [B] <-- [C] <-- [D]
437 the active layer, [D], is committed back to [A] -- which is where live
440 Start the "active ``block-commit``" operation::
442 (QEMU) block-commit device=node-D base=a.qcow2 top=d.qcow2 job-id=job0
444 "execute": "block-commit",
446 "device": "node-D",
447 "job-id": "job0",
461 (QEMU) query-block-jobs
463 "execute": "query-block-jobs",
474 "io-status": "ok",
484 (QEMU) block-job-complete device=job0
486 "execute": "block-job-complete",
503 Live disk synchronization --- ``drive-mirror`` and ``blockdev-mirror``
504 ----------------------------------------------------------------------
511 [A] <-- [B] <-- [C] <-- [D]
513 The ``drive-mirror`` (and its newer equivalent ``blockdev-mirror``)
514 allows you to copy data from the entire chain into a single target image
519 When you cancel an in-progress 'mirror' job *before* the source and
520 target are synchronized, ``block-job-cancel`` will emit the event
522 'mirror' job *after* it has indicated (via the event
524 synchronization, then the event emitted by ``block-job-cancel``
527 Besides the 'mirror' job, the "active ``block-commit``" is the only
529 The rest of the block device jobs ('stream', "non-active
530 ``block-commit``", and 'backup') end automatically.
532 So there are two possible actions to take, after a 'mirror' job has
536 (1) Issuing the command ``block-job-cancel`` (after it emits the event
537 ``BLOCK_JOB_COMPLETED``) will create a point-in-time (which is at
539 image chain (or only the top-most image, depending on the ``sync``
541 live VM migration with non-shared storage.
543 (2) Issuing the command ``block-job-complete`` (after it emits the event
552 (1) ``full`` -- Synchronize the content of entire disk image chain to
555 (2) ``top`` -- Synchronize only the contents of the top-most disk image
558 (3) ``none`` -- Synchronize only the new writes from this point on.
560 .. note:: In the case of ``blockdev-backup`` (or deprecated
561 ``drive-backup``), the behavior of ``none``
569 (4) ``incremental`` -- Synchronize content that is described by the
578 QMP invocation for ``drive-mirror``
582 way to [D], to a new target (``drive-mirror`` will create the destination
585 (QEMU) drive-mirror device=node-D target=e.qcow2 sync=full job-id=job0
587 "execute": "drive-mirror",
589 "device": "node-D",
590 "job-id": "job0",
600 'mirror' job is "ready" to be completed (and QEMU will also emit an
603 (QEMU) query-block-jobs
605 "execute": "query-block-jobs",
612 "type": "mirror",
616 "io-status": "ok",
627 (a) Create a point-in-time snapshot by ending the synchronization. The
628 point-in-time is at the time of *ending* the sync. (The result of
632 (QEMU) block-job-cancel device=job0
634 "execute": "block-job-cancel",
643 (QEMU) block-job-complete device=job0
646 ``query-block-jobs`` command, there should not be any active block
649 Comparing 'commit' and 'mirror': In both then cases, the overlay images
652 the case of 'mirror', a *new* target image is populated with the data
656 QMP invocation for live storage migration with ``drive-mirror`` + NBD
660 common use-cases that takes advantage of the ``drive-mirror`` primitive
661 and QEMU's built-in Network Block Device (NBD) server. Here's a quick
662 walk-through of this setup.
666 [A] <-- [B] <-- [C] <-- [D]
669 contents of the *top*-most disk image (i.e. the active layer), [D], to a
675 -- whether by ``cp``, ``rsync``, or by some storage array-specific
678 Sometimes, this is also referred to as "shallow copy" -- because only
687 chain -- from images [A] to [C] -- are already expected to exist in some
688 form (e.g. in a file called, ``Contents-of-A-B-C.qcow2``). Now, on the
690 ``Contents-of-A-B-C.qcow2`` as its backing file), to which the contents
693 $ qemu-img create -f qcow2 -b ./Contents-of-A-B-C.qcow2 \
694 -F qcow2 ./target-disk.qcow2
697 -- discussed in the section: `Interacting with a QEMU instance`_)
702 .. parsed-literal::
704 $ |qemu_system| -display none -no-user-config -nodefaults \\
705 -m 512 -blockdev \\
706 …node-name=node-TargetDisk,driver=qcow2,file.driver=file,file.node-name=file,file.filename=./target…
707 -device virtio-blk,drive=node-TargetDisk,id=virtio0 \\
708 -S -monitor stdio -qmp unix:./qmp-sock2,server=on,wait=off \\
709 -incoming tcp:localhost:6666
713 [A] <-- [B] <-- [C] <-- [D]
716 ``[A] <-- [B] <-- [C]`` are *already* present, and therefore copy *only*
720 built-in NBD server on a given host (local host, represented by
723 (QEMU) nbd-server-start addr={"type":"inet","data":{"host":"::","port":"49153"}}
725 "execute": "nbd-server-start",
728 "data": {
738 QEMU's built-in NBD server::
740 (QEMU) nbd-server-add device=node-TargetDisk writable=true
742 "execute": "nbd-server-add",
744 "device": "node-TargetDisk"
748 (3) [On *source* QEMU] Then, invoke ``drive-mirror`` (NB: since we're
749 running ``drive-mirror`` with ``mode=existing`` (meaning:
750 synchronize to a pre-created file, therefore 'existing', file on the
754 …(QEMU) drive-mirror device=node-D target=nbd:localhost:49153:exportname=node-TargetDisk sync=top m…
756 "execute": "drive-mirror",
758 "device": "node-D",
760 "job-id": "job0",
761 "target": "nbd:localhost:49153:exportname=node-TargetDisk",
766 (4) [On *source* QEMU] Once ``drive-mirror`` copies the entire data, and the
767 event ``BLOCK_JOB_READY`` is emitted, issue ``block-job-cancel`` to
770 (QEMU) block-job-cancel device=job0
772 "execute": "block-job-cancel",
780 (QEMU) nbd-server-stop
782 "execute": "nbd-server-stop",
796 Higher-level libraries (e.g. libvirt) automate the entire above
797 process (although note that libvirt does not allow same-host
801 Notes on ``blockdev-mirror``
804 The ``blockdev-mirror`` command is equivalent in core functionality to
805 ``drive-mirror``, except that it operates at node-level in a BDS graph.
807 Also: for ``blockdev-mirror``, the 'target' image needs to be explicitly
808 created (using ``qemu-img``) and attach it to live QEMU via
809 ``blockdev-add``, which assigns a name to the to-be created target node.
811 E.g. the sequence of actions to create a point-in-time backup of an
812 entire disk image chain, to a target, using ``blockdev-mirror`` would be:
817 (1) Create the target image (using ``qemu-img``), say, ``e.qcow2``
819 (2) Attach the above created file (``e.qcow2``), run-time, using
820 ``blockdev-add`` to QEMU
822 (3) Perform ``blockdev-mirror`` (use ``"sync": "full"`` to copy the
826 (4) Optionally, query for active block jobs, there should be a 'mirror'
829 (5) Gracefully complete the 'mirror' block device job, and notice the
836 the target copy with ``qemu-img compare``. You should notice:
840 QMP invocation for ``blockdev-mirror``
845 [A] <-- [B] <-- [C] <-- [D]
852 …(QEMU) blockdev-snapshot-sync node-name=node-A snapshot-file=b.qcow2 snapshot-node-name=node-B for…
853 …(QEMU) blockdev-snapshot-sync node-name=node-B snapshot-file=c.qcow2 snapshot-node-name=node-C for…
854 …(QEMU) blockdev-snapshot-sync node-name=node-C snapshot-file=d.qcow2 snapshot-node-name=node-D for…
858 $ qemu-img create -f qcow2 e.qcow2 39M
860 Add the above created target image to QEMU, via ``blockdev-add``::
862 (QEMU) blockdev-add driver=qcow2 node-name=node-E file={"driver":"file","filename":"e.qcow2"}
864 "execute": "blockdev-add",
866 "node-name": "node-E",
875 Perform ``blockdev-mirror``, and notice the event ``BLOCK_JOB_READY``::
877 (QEMU) blockdev-mirror device=node-B target=node-E sync=full job-id=job0
879 "execute": "blockdev-mirror",
881 "device": "node-D",
882 "job-id": "job0",
883 "target": "node-E",
888 Query for active block jobs, there should be a 'mirror' job ready::
890 (QEMU) query-block-jobs
892 "execute": "query-block-jobs",
899 "type": "mirror",
903 "io-status": "ok",
914 (QEMU) block-job-complete device=job0
916 "execute": "block-job-complete",
934 Live disk backup --- ``blockdev-backup`` and the deprecated ``drive-backup``
935 ----------------------------------------------------------------------------
937 The ``blockdev-backup`` (and the deprecated ``drive-backup``) allows
938 you to create a point-in-time snapshot.
940 In this case, the point-in-time is when you *start* the
941 ``blockdev-backup`` (or deprecated ``drive-backup``) command.
944 QMP invocation for ``drive-backup``
947 Note that ``drive-backup`` command is deprecated since QEMU 6.2 and
952 [A] <-- [B] <-- [C] <-- [D]
956 image does not exist, ``drive-backup`` will create it)::
958 (QEMU) drive-backup device=node-D sync=full target=e.qcow2 job-id=job0
960 "execute": "drive-backup",
962 "device": "node-D",
963 "job-id": "job0",
969 Once the above ``drive-backup`` has completed, a ``BLOCK_JOB_COMPLETED`` event
974 Moving from the deprecated ``drive-backup`` to newer ``blockdev-backup``
977 ``blockdev-backup`` differs from ``drive-backup`` in how you specify
978 the backup target. With ``blockdev-backup`` you can't specify filename
979 as a target. Instead you use ``node-name`` of existing block node,
980 which you may add by ``blockdev-add`` or ``blockdev-create`` commands.
981 Correspondingly, ``blockdev-backup`` doesn't have ``mode`` and
986 Notes on ``blockdev-backup``
989 The ``blockdev-backup`` command operates at node-level in a Block Driver
992 E.g. the sequence of actions to create a point-in-time backup
993 of an entire disk image chain, to a target, using ``blockdev-backup``
999 (1) Create the target image (using ``qemu-img``), say, ``e.qcow2``
1001 (2) Attach the above created file (``e.qcow2``), run-time, using
1002 ``blockdev-add`` to QEMU
1004 (3) Perform ``blockdev-backup`` (use ``"sync": "full"`` to copy the
1012 the target copy with ``qemu-img compare``. You should notice:
1016 ``blockdev-backup``.
1018 QMP invocation for ``blockdev-backup``
1024 [A] <-- [B]
1032 …(QEMU) blockdev-snapshot-sync node-name=node-A snapshot-file=b.qcow2 snapshot-node-name=node-B for…
1034 "execute": "blockdev-snapshot-sync",
1036 "node-name": "node-A",
1037 "snapshot-file": "b.qcow2",
1039 "snapshot-node-name": "node-B"
1046 $ qemu-img create -f qcow2 e.qcow2 39M
1048 Then add it to QEMU via ``blockdev-add``::
1050 (QEMU) blockdev-add driver=qcow2 node-name=node-E file={"driver":"file","filename":"e.qcow2"}
1052 "execute": "blockdev-add",
1054 "node-name": "node-E",
1063 Then invoke ``blockdev-backup`` to copy the contents from the entire
1067 (QEMU) blockdev-backup device=node-B target=node-E sync=full job-id=job0
1069 "execute": "blockdev-backup",
1071 "device": "node-B",
1072 "job-id": "job0",
1073 "target": "node-E",
1084 (QEMU) query-block-jobs
1086 "execute": "query-block-jobs",
1103 ``qemu-img compare`` to verify the integrity of the disk image
1108 point-in-time backup of the disk image chain -- i.e. contents from
1109 images [A] and [B] at the time the ``blockdev-backup`` command was
1115 QEMU was launched with ``-S`` option, which will not start the CPUs at
1118 $ qemu-img compare b.qcow2 e.qcow2