1.. 2 Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc. 3 All rights reserved. 4 5 This file is licensed via The FreeBSD Documentation License, the full 6 text of which is included at the end of this document. 7 8==================================== 9Dirty Bitmaps and Incremental Backup 10==================================== 11 12Dirty Bitmaps are in-memory objects that track writes to block devices. They 13can be used in conjunction with various block job operations to perform 14incremental or differential backup regimens. 15 16This document explains the conceptual mechanisms, as well as up-to-date, 17complete and comprehensive documentation on the API to manipulate them. 18(Hopefully, the "why", "what", and "how".) 19 20The intended audience for this document is developers who are adding QEMU 21backup features to management applications, or power users who run and 22administer QEMU directly via QMP. 23 24.. contents:: 25 26Overview 27-------- 28 29Bitmaps are bit vectors where each '1' bit in the vector indicates a modified 30("dirty") segment of the corresponding block device. The size of the segment 31that is tracked is the granularity of the bitmap. If the granularity of a 32bitmap is 64K, each '1' bit means that a 64K region as a whole may have 33changed in some way, possibly by as little as one byte. 34 35Smaller granularities mean more accurate tracking of modified disk data, but 36requires more computational overhead and larger bitmap sizes. Larger 37granularities mean smaller bitmap sizes, but less targeted backups. 38 39The size of a bitmap (in bytes) can be computed as such: 40 ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8) 41 42e.g. the size of a 64KiB granularity bitmap on a 2TiB image is: 43 ``size`` = ((2147483648K / 64K) / 8) 44 = 4194304B = 4MiB. 45 46QEMU uses these bitmaps when making incremental backups to know which sections 47of the file to copy out. They are not enabled by default and must be 48explicitly added in order to begin tracking writes. 49 50Bitmaps can be created at any time and can be attached to any arbitrary block 51node in the storage graph, but are most useful conceptually when attached to 52the root node attached to the guest's storage device model. 53 54That is to say: It's likely most useful to track the guest's writes to disk, 55but you could theoretically track things like qcow2 metadata changes by 56attaching the bitmap elsewhere in the storage graph. This is beyond the scope 57of this document. 58 59QEMU supports persisting these bitmaps to disk via the qcow2 image format. 60Bitmaps which are stored or loaded in this way are called "persistent", 61whereas bitmaps that are not are called "transient". 62 63QEMU also supports the migration of both transient bitmaps (tracking any 64arbitrary image format) or persistent bitmaps (qcow2) via live migration. 65 66Supported Image Formats 67----------------------- 68 69QEMU supports all documented features below on the qcow2 image format. 70 71However, qcow2 is only strictly necessary for the persistence feature, which 72writes bitmap data to disk upon close. If persistence is not required for a 73specific use case, all bitmap features excepting persistence are available for 74any arbitrary image format. 75 76For example, Dirty Bitmaps can be combined with the 'raw' image format, but 77any changes to the bitmap will be discarded upon exit. 78 79.. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent 80 bitmaps are available only on qcow2 images. 81 82Dirty Bitmap Names 83------------------ 84 85Bitmap objects need a method to reference them in the API. All API-created and 86managed bitmaps have a human-readable name chosen by the user at creation 87time. 88 89- A bitmap's name is unique to the node, but bitmaps attached to different 90 nodes can share the same name. Therefore, all bitmaps are addressed via 91 their (node, name) pair. 92 93- The name of a user-created bitmap cannot be empty (""). 94 95- Transient bitmaps can have JSON unicode names that are effectively not 96 length limited. (QMP protocol may restrict messages to less than 64MiB.) 97 98- Persistent storage formats may impose their own requirements on bitmap names 99 and namespaces. Presently, only qcow2 supports persistent bitmaps. See 100 docs/interop/qcow2.txt for more details on restrictions. Notably: 101 102 - qcow2 bitmap names are limited to between 1 and 1023 bytes long. 103 104 - No two bitmaps saved to the same qcow2 file may share the same name. 105 106- QEMU occasionally uses bitmaps for internal use which have no name. They are 107 hidden from API query calls, cannot be manipulated by the external API, are 108 never persistent, nor ever migrated. 109 110Bitmap Status 111------------- 112 113Dirty Bitmap objects can be queried with the QMP command `query-block 114<qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the 115`BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure. 116 117This struct shows the name, granularity, and dirty byte count for each bitmap. 118Additionally, it shows several boolean status indicators: 119 120- ``recording``: This bitmap is recording writes. 121- ``busy``: This bitmap is in-use by an operation. 122- ``persistent``: This bitmap is a persistent type. 123- ``inconsistent``: This bitmap is corrupted and cannot be used. 124 125The ``+busy`` status prohibits you from deleting, clearing, or otherwise 126modifying a bitmap, and happens when the bitmap is being used for a backup 127operation or is in the process of being loaded from a migration. Many of the 128commands documented below will refuse to work on such bitmaps. 129 130The ``+inconsistent`` status similarly prohibits almost all operations, 131notably allowing only the ``block-dirty-bitmap-remove`` operation. 132 133There is also a deprecated ``status`` field of type `DirtyBitmapStatus 134<qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had 135five visible states: 136 137 #. ``Frozen``: This bitmap is currently in-use by an operation and is 138 immutable. It can't be deleted, renamed, reset, etc. 139 140 (This is now ``+busy``.) 141 142 #. ``Disabled``: This bitmap is not recording new writes. 143 144 (This is now ``-recording -busy``.) 145 146 #. ``Active``: This bitmap is recording new writes. 147 148 (This is now ``+recording -busy``.) 149 150 #. ``Locked``: This bitmap is in-use by an operation, and is immutable. 151 The difference from "Frozen" was primarily implementation details. 152 153 (This is now ``+busy``.) 154 155 #. ``Inconsistent``: This persistent bitmap was not saved to disk 156 correctly, and can no longer be used. It remains in memory to serve as 157 an indicator of failure. 158 159 (This is now ``+inconsistent``.) 160 161These states are directly replaced by the status indicators and should not be 162used. The difference between ``Frozen`` and ``Locked`` is an implementation 163detail and should not be relevant to external users. 164 165Basic QMP Usage 166--------------- 167 168The primary interface to manipulating bitmap objects is via the QMP 169interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad 170overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all 171QMP commands. 172 173Supported Commands 174~~~~~~~~~~~~~~~~~~ 175 176There are six primary bitmap-management API commands: 177 178- ``block-dirty-bitmap-add`` 179- ``block-dirty-bitmap-remove`` 180- ``block-dirty-bitmap-clear`` 181- ``block-dirty-bitmap-disable`` 182- ``block-dirty-bitmap-enable`` 183- ``block-dirty-bitmap-merge`` 184 185And one related query command: 186 187- ``query-block`` 188 189Creation: block-dirty-bitmap-add 190~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 191 192`block-dirty-bitmap-add 193<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_: 194 195Creates a new bitmap that tracks writes to the specified node. granularity, 196persistence, and recording state can be adjusted at creation time. 197 198.. admonition:: Example 199 200 to create a new, actively recording persistent bitmap: 201 202 .. code:: json 203 204 -> { "execute": "block-dirty-bitmap-add", 205 "arguments": { 206 "node": "drive0", 207 "name": "bitmap0", 208 "persistent": true, 209 } 210 } 211 212 <- { "return": {} } 213 214- This bitmap will have a default granularity that matches the cluster size of 215 its associated drive, if available, clamped to between [4KiB, 64KiB]. The 216 current default for qcow2 is 64KiB. 217 218.. admonition:: Example 219 220 To create a new, disabled (``-recording``), transient bitmap that tracks 221 changes in 32KiB segments: 222 223 .. code:: json 224 225 -> { "execute": "block-dirty-bitmap-add", 226 "arguments": { 227 "node": "drive0", 228 "name": "bitmap1", 229 "granularity": 32768, 230 "disabled": true 231 } 232 } 233 234 <- { "return": {} } 235 236Deletion: block-dirty-bitmap-remove 237~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 238 239`block-dirty-bitmap-remove 240<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_: 241 242Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. 243 244- Deleting a bitmap does not impact any other bitmaps attached to the same 245 node, nor does it affect any backups already created from this bitmap or 246 node. 247 248- Because bitmaps are only unique to the node to which they are attached, you 249 must specify the node/drive name here, too. 250 251- Deleting a persistent bitmap will remove it from the qcow2 file. 252 253.. admonition:: Example 254 255 Remove a bitmap named ``bitmap0`` from node ``drive0``: 256 257 .. code:: json 258 259 -> { "execute": "block-dirty-bitmap-remove", 260 "arguments": { 261 "node": "drive0", 262 "name": "bitmap0" 263 } 264 } 265 266 <- { "return": {} } 267 268Resetting: block-dirty-bitmap-clear 269~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 270 271`block-dirty-bitmap-clear 272<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_: 273 274Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. 275 276- An incremental backup created from an empty bitmap will copy no data, as if 277 nothing has changed. 278 279.. admonition:: Example 280 281 Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: 282 283 .. code:: json 284 285 -> { "execute": "block-dirty-bitmap-clear", 286 "arguments": { 287 "node": "drive0", 288 "name": "bitmap0" 289 } 290 } 291 292 <- { "return": {} } 293 294Enabling: block-dirty-bitmap-enable 295~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 296 297`block-dirty-bitmap-enable 298<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_: 299 300"Enables" a bitmap, setting the ``recording`` bit to true, causing writes to 301begin being recorded. ``+busy`` bitmaps cannot be enabled. 302 303- Bitmaps default to being enabled when created, unless configured otherwise. 304 305- Persistent enabled bitmaps will remember their ``+recording`` status on 306 load. 307 308.. admonition:: Example 309 310 To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: 311 312 .. code:: json 313 314 -> { "execute": "block-dirty-bitmap-enable", 315 "arguments": { 316 "node": "drive0", 317 "name": "bitmap0" 318 } 319 } 320 321 <- { "return": {} } 322 323Enabling: block-dirty-bitmap-disable 324~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 325 326`block-dirty-bitmap-disable 327<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_: 328 329"Disables" a bitmap, setting the ``recording`` bit to false, causing further 330writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. 331 332.. warning:: 333 334 This is potentially dangerous: QEMU makes no effort to stop any writes if 335 there are disabled bitmaps on a node, and will not mark any disabled bitmaps 336 as ``+inconsistent`` if any such writes do happen. Backups made from such 337 bitmaps will not be able to be used to reconstruct a coherent image. 338 339- Disabling a bitmap may be useful for examining which sectors of a disk 340 changed during a specific time period, or for explicit management of 341 differential backup windows. 342 343- Persistent disabled bitmaps will remember their ``-recording`` status on 344 load. 345 346.. admonition:: Example 347 348 To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: 349 350 .. code:: json 351 352 -> { "execute": "block-dirty-bitmap-disable", 353 "arguments": { 354 "node": "drive0", 355 "name": "bitmap0" 356 } 357 } 358 359 <- { "return": {} } 360 361Merging, Copying: block-dirty-bitmap-merge 362~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 363 364`block-dirty-bitmap-merge 365<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_: 366 367Merges one or more bitmaps into a target bitmap. For any segment that is dirty 368in any one source bitmap, the target bitmap will mark that segment dirty. 369 370- Merge takes one or more bitmaps as a source and merges them together into a 371 single destination, such that any segment marked as dirty in any source 372 bitmap(s) will be marked dirty in the destination bitmap. 373 374- Merge does not create the destination bitmap if it does not exist. A blank 375 bitmap can be created beforehand to achieve the same effect. 376 377- The destination is not cleared prior to merge, so subsequent merge 378 operations will continue to cumulatively mark more segments as dirty. 379 380- If the merge operation should fail, the destination bitmap is guaranteed to 381 be unmodified. The operation may fail if the source or destination bitmaps 382 are busy, or have different granularities. 383 384- Bitmaps can only be merged on the same node. There is only one "node" 385 argument, so all bitmaps must be attached to that same node. 386 387- Copy can be achieved by merging from a single source to an empty 388 destination. 389 390.. admonition:: Example 391 392 Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node 393 ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves 394 a copy. 395 396 .. code:: json 397 398 -> { "execute": "block-dirty-bitmap-merge", 399 "arguments": { 400 "node": "drive0", 401 "target": "new_bitmap", 402 "bitmaps: [ "bitmap0" ] 403 } 404 } 405 406 <- { "return": {} } 407 408Querying: query-block 409~~~~~~~~~~~~~~~~~~~~~ 410 411`query-block 412<qemu-qmp-ref.html#index-query_002dblock>`_: 413 414Not strictly a bitmaps command, but will return information about any bitmaps 415attached to nodes serving as the root for guest devices. 416 417- The "inconsistent" bit will not appear when it is false, appearing only when 418 the value is true to indicate there is a problem. 419 420.. admonition:: Example 421 422 Query the block sub-system of QEMU. The following json has trimmed irrelevant 423 keys from the response to highlight only the bitmap-relevant portions of the 424 API. This result highlights a bitmap ``bitmap0`` attached to the root node of 425 device ``drive0``. 426 427 .. code:: json 428 429 -> { 430 "execute": "query-block", 431 "arguments": {} 432 } 433 434 <- { 435 "return": [ { 436 "dirty-bitmaps": [ { 437 "status": "active", 438 "count": 0, 439 "busy": false, 440 "name": "bitmap0", 441 "persistent": false, 442 "recording": true, 443 "granularity": 65536 444 } ], 445 "device": "drive0", 446 } ] 447 } 448 449Bitmap Persistence 450------------------ 451 452As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2 453files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing 454``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to 455disk. 456 457Persistent bitmaps will be automatically loaded into memory upon load, and 458will be written back to disk upon close. Their usage should be mostly 459transparent. 460 461However, if QEMU does not get a chance to close the file cleanly, the bitmap 462will be marked as ``+inconsistent`` at next load and considered unsafe to use 463for any operation. At this point, the only valid operation on such bitmaps is 464``block-dirty-bitmap-remove``. 465 466Losing a bitmap in this way does not invalidate any existing backups that have 467been made from this bitmap, but no further backups will be able to be issued 468for this chain. 469 470Transactions 471------------ 472 473Transactions are a QMP feature that allows you to submit multiple QMP commands 474at once, being guaranteed that they will all succeed or fail atomically, 475together. The interaction of bitmaps and transactions are demonstrated below. 476 477See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference 478for more details. 479 480Justification 481~~~~~~~~~~~~~ 482 483Bitmaps can generally be modified at any time, but certain operations often 484only make sense when paired directly with other commands. When a VM is paused, 485it's easy to ensure that no guest writes occur between individual QMP 486commands. When a VM is running, this is difficult to accomplish with 487individual QMP commands that may allow guest writes to occur inbetween each 488command. 489 490For example, using only individual QMP commands, we could: 491 492#. Boot the VM in a paused state. 493#. Create a full drive backup of drive0. 494#. Create a new bitmap attached to drive0, confident that nothing has been 495 written to drive0 in the meantime. 496#. Resume execution of the VM. 497#. At a later point, issue incremental backups from ``bitmap0``. 498 499At this point, the bitmap and drive backup would be correctly in sync, and 500incremental backups made from this point forward would be correctly aligned to 501the full drive backup. 502 503This is not particularly useful if we decide we want to start incremental 504backups after the VM has been running for a while, for which we would want to 505perform actions such as the following: 506 507#. Boot the VM and begin execution. 508#. Using a single transaction, perform the following operations: 509 510 - Create ``bitmap0``. 511 - Create a full drive backup of ``drive0``. 512 513#. At a later point, issue incremental backups from ``bitmap0``. 514 515.. note:: As a consideration, if ``bitmap0`` is created prior to the full 516 drive backup, incremental backups can still be authored from this 517 bitmap, but they will copy extra segments reflecting writes that 518 occurred prior to the backup operation. Transactions allow us to 519 narrow critical points in time to reduce waste, or, in the other 520 direction, to ensure that no segments are omitted. 521 522Supported Bitmap Transactions 523~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 524 525- ``block-dirty-bitmap-add`` 526- ``block-dirty-bitmap-clear`` 527- ``block-dirty-bitmap-enable`` 528- ``block-dirty-bitmap-disable`` 529- ``block-dirty-bitmap-merge`` 530 531The usages for these commands are identical to their respective QMP commands, 532but see the sections below for concrete examples. 533 534Incremental Backups - Push Model 535-------------------------------- 536 537Incremental backups are simply partial disk images that can be combined with 538other partial disk images on top of a base image to reconstruct a full backup 539from the point in time at which the incremental backup was issued. 540 541The "Push Model" here references the fact that QEMU is "pushing" the modified 542blocks out to a destination. We will be using the `drive-backup 543<qemu-qmp-ref.html#index-drive_002dbackup>`_ and `blockdev-backup 544<qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP commands to create both 545full and incremental backups. 546 547Both of these commands are jobs, which have their own QMP API for querying and 548management documented in `Background jobs 549<qemu-qmp-ref.html#Background-jobs>`_. 550 551Example: New Incremental Backup Anchor Point 552~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 553 554As outlined in the Transactions - `Justification`_ section, perhaps we want to 555create a new incremental backup chain attached to a drive. 556 557This example creates a new, full backup of "drive0" and accompanies it with a 558new, empty bitmap that records writes from this point in time forward. 559 560.. note:: Any new writes that happen after this command is issued, even while 561 the backup job runs, will be written locally and not to the backup 562 destination. These writes will be recorded in the bitmap 563 accordingly. 564 565.. code:: json 566 567 -> { 568 "execute": "transaction", 569 "arguments": { 570 "actions": [ 571 { 572 "type": "block-dirty-bitmap-add", 573 "data": { 574 "node": "drive0", 575 "name": "bitmap0" 576 } 577 }, 578 { 579 "type": "drive-backup", 580 "data": { 581 "device": "drive0", 582 "target": "/path/to/drive0.full.qcow2", 583 "sync": "full", 584 "format": "qcow2" 585 } 586 } 587 ] 588 } 589 } 590 591 <- { "return": {} } 592 593 <- { 594 "timestamp": { 595 "seconds": 1555436945, 596 "microseconds": 179620 597 }, 598 "data": { 599 "status": "created", 600 "id": "drive0" 601 }, 602 "event": "JOB_STATUS_CHANGE" 603 } 604 605 ... 606 607 <- { 608 "timestamp": {...}, 609 "data": { 610 "device": "drive0", 611 "type": "backup", 612 "speed": 0, 613 "len": 68719476736, 614 "offset": 68719476736 615 }, 616 "event": "BLOCK_JOB_COMPLETED" 617 } 618 619 <- { 620 "timestamp": {...}, 621 "data": { 622 "status": "concluded", 623 "id": "drive0" 624 }, 625 "event": "JOB_STATUS_CHANGE" 626 } 627 628 <- { 629 "timestamp": {...}, 630 "data": { 631 "status": "null", 632 "id": "drive0" 633 }, 634 "event": "JOB_STATUS_CHANGE" 635 } 636 637A full explanation of the job transition semantics and the JOB_STATUS_CHANGE 638event are beyond the scope of this document and will be omitted in all 639subsequent examples; above, several more events have been omitted for brevity. 640 641.. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED 642 except where necessary to illustrate workflow differences. 643 644 Omitted events and json objects will be represented by ellipses: 645 ``...`` 646 647Example: Resetting an Incremental Backup Anchor Point 648~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 649 650If we want to start a new backup chain with an existing bitmap, we can also 651use a transaction to reset the bitmap while making a new full backup: 652 653.. code:: json 654 655 -> { 656 "execute": "transaction", 657 "arguments": { 658 "actions": [ 659 { 660 "type": "block-dirty-bitmap-clear", 661 "data": { 662 "node": "drive0", 663 "name": "bitmap0" 664 } 665 }, 666 { 667 "type": "drive-backup", 668 "data": { 669 "device": "drive0", 670 "target": "/path/to/drive0.new_full.qcow2", 671 "sync": "full", 672 "format": "qcow2" 673 } 674 } 675 ] 676 } 677 } 678 679 <- { "return": {} } 680 681 ... 682 683 <- { 684 "timestamp": {...}, 685 "data": { 686 "device": "drive0", 687 "type": "backup", 688 "speed": 0, 689 "len": 68719476736, 690 "offset": 68719476736 691 }, 692 "event": "BLOCK_JOB_COMPLETED" 693 } 694 695 ... 696 697The result of this example is identical to the first, but we clear an existing 698bitmap instead of adding a new one. 699 700.. tip:: In both of these examples, "bitmap0" is tied conceptually to the 701 creation of new, full backups. This relationship is not saved or 702 remembered by QEMU; it is up to the operator or management layer to 703 remember which bitmaps are associated with which backups. 704 705Example: First Incremental Backup 706~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 707 708#. Create a full backup and sync it to a dirty bitmap using any method: 709 710 - Either of the two live backup method demonstrated above, 711 - Using QMP commands with the VM paused as in the `Justification`_ section, 712 or 713 - With the VM offline, manually copy the image and start the VM in a paused 714 state, careful to add a new bitmap before the VM begins execution. 715 716 Whichever method is chosen, let's assume that at the end of this step: 717 718 - The full backup is named ``drive0.full.qcow2``. 719 - The bitmap we created is named ``bitmap0``, attached to ``drive0``. 720 721#. Create a destination image for the incremental backup that utilizes the 722 full backup as a backing image. 723 724 - Let's assume the new incremental image is named ``drive0.inc0.qcow2``: 725 726 .. code:: bash 727 728 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 729 -b drive0.full.qcow2 -F qcow2 730 731#. Issue an incremental backup command: 732 733 .. code:: json 734 735 -> { 736 "execute": "drive-backup", 737 "arguments": { 738 "device": "drive0", 739 "bitmap": "bitmap0", 740 "target": "drive0.inc0.qcow2", 741 "format": "qcow2", 742 "sync": "incremental", 743 "mode": "existing" 744 } 745 } 746 747 <- { "return": {} } 748 749 ... 750 751 <- { 752 "timestamp": {...}, 753 "data": { 754 "device": "drive0", 755 "type": "backup", 756 "speed": 0, 757 "len": 68719476736, 758 "offset": 68719476736 759 }, 760 "event": "BLOCK_JOB_COMPLETED" 761 } 762 763 ... 764 765This copies any blocks modified since the full backup was created into the 766``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked 767``+busy``. If the operation is successful, ``bitmap0`` will be cleared to 768reflect the "incremental" backup regimen, which only copies out new changes 769from each incremental backup. 770 771.. note:: Any new writes that occur after the backup operation starts do not 772 get copied to the destination. The backup's "point in time" is when 773 the backup starts, not when it ends. These writes are recorded in a 774 special bitmap that gets re-added to bitmap0 when the backup ends so 775 that the next incremental backup can copy them out. 776 777Example: Second Incremental Backup 778~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 779 780#. Create a new destination image for the incremental backup that points to 781 the previous one, e.g.: ``drive0.inc1.qcow2`` 782 783 .. code:: bash 784 785 $ qemu-img create -f qcow2 drive0.inc1.qcow2 \ 786 -b drive0.inc0.qcow2 -F qcow2 787 788#. Issue a new incremental backup command. The only difference here is that we 789 have changed the target image below. 790 791 .. code:: json 792 793 -> { 794 "execute": "drive-backup", 795 "arguments": { 796 "device": "drive0", 797 "bitmap": "bitmap0", 798 "target": "drive0.inc1.qcow2", 799 "format": "qcow2", 800 "sync": "incremental", 801 "mode": "existing" 802 } 803 } 804 805 <- { "return": {} } 806 807 ... 808 809 <- { 810 "timestamp": {...}, 811 "data": { 812 "device": "drive0", 813 "type": "backup", 814 "speed": 0, 815 "len": 68719476736, 816 "offset": 68719476736 817 }, 818 "event": "BLOCK_JOB_COMPLETED" 819 } 820 821 ... 822 823Because the first incremental backup from the previous example completed 824successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here, 825we use ``bitmap0`` again to create a new incremental backup that targets the 826previous one, creating a chain of three images: 827 828.. admonition:: Diagram 829 830 .. code:: text 831 832 +-------------------+ +-------------------+ +-------------------+ 833 | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 | 834 +-------------------+ +-------------------+ +-------------------+ 835 836Each new incremental backup re-synchronizes the bitmap to the latest backup 837authored, allowing a user to continue to "consume" it to create new backups on 838top of an existing chain. 839 840In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are 841complete images by themselves, but rely on their backing chain to reconstruct 842a full image. The dependency terminates with each full backup. 843 844Each backup in this chain remains independent, and is unchanged by new entries 845made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly 846valid backup of the disk as it was when that backup was issued. 847 848Example: Incremental Push Backups without Backing Files 849~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 850 851Backup images are best kept off-site, so we often will not have the preceding 852backups in a chain available to link against. This is not a problem at backup 853time; we simply do not set the backing image when creating the destination 854image: 855 856#. Create a new destination image with no backing file set. We will need to 857 specify the size of the base image, because the backing file isn't 858 available for QEMU to use to determine it. 859 860 .. code:: bash 861 862 $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G 863 864 .. note:: Alternatively, you can omit ``mode: "existing"`` from the push 865 backup commands to have QEMU create an image without a backing 866 file for you, but you lose control over format options like 867 compatibility and preallocation presets. 868 869#. Issue a new incremental backup command. Apart from the new destination 870 image, there is no difference from the last two examples. 871 872 .. code:: json 873 874 -> { 875 "execute": "drive-backup", 876 "arguments": { 877 "device": "drive0", 878 "bitmap": "bitmap0", 879 "target": "drive0.inc2.qcow2", 880 "format": "qcow2", 881 "sync": "incremental", 882 "mode": "existing" 883 } 884 } 885 886 <- { "return": {} } 887 888 ... 889 890 <- { 891 "timestamp": {...}, 892 "data": { 893 "device": "drive0", 894 "type": "backup", 895 "speed": 0, 896 "len": 68719476736, 897 "offset": 68719476736 898 }, 899 "event": "BLOCK_JOB_COMPLETED" 900 } 901 902 ... 903 904The only difference from the perspective of the user is that you will need to 905set the backing image when attempting to restore the backup: 906 907.. code:: bash 908 909 $ qemu-img rebase drive0.inc2.qcow2 \ 910 -u -b drive0.inc1.qcow2 911 912This uses the "unsafe" rebase mode to simply set the backing file to a file 913that isn't present. 914 915It is also possible to use ``--image-opts`` to specify the entire backing 916chain by hand as an ephemeral property at runtime, but that is beyond the 917scope of this document. 918 919Example: Multi-drive Incremental Backup 920~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 921 922Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back 923both of them up such that the two backups represent the same crash-consistent 924point in time. 925 926#. For each drive, create an empty image: 927 928 .. code:: bash 929 930 $ qemu-img create -f qcow2 drive0.full.qcow2 64G 931 $ qemu-img create -f qcow2 drive1.full.qcow2 64G 932 933#. Create a full (anchor) backup for each drive, with accompanying bitmaps: 934 935 .. code:: json 936 937 -> { 938 "execute": "transaction", 939 "arguments": { 940 "actions": [ 941 { 942 "type": "block-dirty-bitmap-add", 943 "data": { 944 "node": "drive0", 945 "name": "bitmap0" 946 } 947 }, 948 { 949 "type": "block-dirty-bitmap-add", 950 "data": { 951 "node": "drive1", 952 "name": "bitmap0" 953 } 954 }, 955 { 956 "type": "drive-backup", 957 "data": { 958 "device": "drive0", 959 "target": "/path/to/drive0.full.qcow2", 960 "sync": "full", 961 "format": "qcow2" 962 } 963 }, 964 { 965 "type": "drive-backup", 966 "data": { 967 "device": "drive1", 968 "target": "/path/to/drive1.full.qcow2", 969 "sync": "full", 970 "format": "qcow2" 971 } 972 } 973 ] 974 } 975 } 976 977 <- { "return": {} } 978 979 ... 980 981 <- { 982 "timestamp": {...}, 983 "data": { 984 "device": "drive0", 985 "type": "backup", 986 "speed": 0, 987 "len": 68719476736, 988 "offset": 68719476736 989 }, 990 "event": "BLOCK_JOB_COMPLETED" 991 } 992 993 ... 994 995 <- { 996 "timestamp": {...}, 997 "data": { 998 "device": "drive1", 999 "type": "backup", 1000 "speed": 0, 1001 "len": 68719476736, 1002 "offset": 68719476736 1003 }, 1004 "event": "BLOCK_JOB_COMPLETED" 1005 } 1006 1007 ... 1008 1009#. Later, create new destination images for each of the incremental backups 1010 that point to their respective full backups: 1011 1012 .. code:: bash 1013 1014 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1015 -b drive0.full.qcow2 -F qcow2 1016 $ qemu-img create -f qcow2 drive1.inc0.qcow2 \ 1017 -b drive1.full.qcow2 -F qcow2 1018 1019#. Issue a multi-drive incremental push backup transaction: 1020 1021 .. code:: json 1022 1023 -> { 1024 "execute": "transaction", 1025 "arguments": { 1026 "actions": [ 1027 { 1028 "type": "drive-backup", 1029 "data": { 1030 "device": "drive0", 1031 "bitmap": "bitmap0", 1032 "format": "qcow2", 1033 "mode": "existing", 1034 "sync": "incremental", 1035 "target": "drive0.inc0.qcow2" 1036 } 1037 }, 1038 { 1039 "type": "drive-backup", 1040 "data": { 1041 "device": "drive1", 1042 "bitmap": "bitmap0", 1043 "format": "qcow2", 1044 "mode": "existing", 1045 "sync": "incremental", 1046 "target": "drive1.inc0.qcow2" 1047 } 1048 }, 1049 ] 1050 } 1051 } 1052 1053 <- { "return": {} } 1054 1055 ... 1056 1057 <- { 1058 "timestamp": {...}, 1059 "data": { 1060 "device": "drive0", 1061 "type": "backup", 1062 "speed": 0, 1063 "len": 68719476736, 1064 "offset": 68719476736 1065 }, 1066 "event": "BLOCK_JOB_COMPLETED" 1067 } 1068 1069 ... 1070 1071 <- { 1072 "timestamp": {...}, 1073 "data": { 1074 "device": "drive1", 1075 "type": "backup", 1076 "speed": 0, 1077 "len": 68719476736, 1078 "offset": 68719476736 1079 }, 1080 "event": "BLOCK_JOB_COMPLETED" 1081 } 1082 1083 ... 1084 1085Push Backup Errors & Recovery 1086----------------------------- 1087 1088In the event of an error that occurs after a push backup job is successfully 1089launched, either by an individual QMP command or a QMP transaction, the user 1090will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message, 1091accompanied by a ``BLOCK_JOB_ERROR`` event. 1092 1093In the case of a job being cancelled, the user will receive a 1094``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR 1095events. 1096 1097In either failure case, the bitmap used for the failed operation is not 1098cleared. It will contain all of the dirty bits it did at the start of the 1099operation, plus any new bits that got marked during the operation. 1100 1101Effectively, the "point in time" that a bitmap is recording differences 1102against is kept at the issuance of the last successful incremental backup, 1103instead of being moved forward to the start of this now-failed backup. 1104 1105Once the underlying problem is addressed (e.g. more storage space is allocated 1106on the destination), the incremental backup command can be retried with the 1107same bitmap. 1108 1109Example: Individual Failures 1110~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1111 1112Incremental Push Backup jobs that fail individually behave simply as 1113described above. This example demonstrates the single-job failure case: 1114 1115#. Create a target image: 1116 1117 .. code:: bash 1118 1119 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1120 -b drive0.full.qcow2 -F qcow2 1121 1122#. Attempt to create an incremental backup via QMP: 1123 1124 .. code:: json 1125 1126 -> { 1127 "execute": "drive-backup", 1128 "arguments": { 1129 "device": "drive0", 1130 "bitmap": "bitmap0", 1131 "target": "drive0.inc0.qcow2", 1132 "format": "qcow2", 1133 "sync": "incremental", 1134 "mode": "existing" 1135 } 1136 } 1137 1138 <- { "return": {} } 1139 1140#. Receive a pair of events indicating failure: 1141 1142 .. code:: json 1143 1144 <- { 1145 "timestamp": {...}, 1146 "data": { 1147 "device": "drive0", 1148 "action": "report", 1149 "operation": "write" 1150 }, 1151 "event": "BLOCK_JOB_ERROR" 1152 } 1153 1154 <- { 1155 "timestamp": {...}, 1156 "data": { 1157 "speed": 0, 1158 "offset": 0, 1159 "len": 67108864, 1160 "error": "No space left on device", 1161 "device": "drive0", 1162 "type": "backup" 1163 }, 1164 "event": "BLOCK_JOB_COMPLETED" 1165 } 1166 1167#. Delete the failed image, and re-create it. 1168 1169 .. code:: bash 1170 1171 $ rm drive0.inc0.qcow2 1172 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1173 -b drive0.full.qcow2 -F qcow2 1174 1175#. Retry the command after fixing the underlying problem, such as 1176 freeing up space on the backup volume: 1177 1178 .. code:: json 1179 1180 -> { 1181 "execute": "drive-backup", 1182 "arguments": { 1183 "device": "drive0", 1184 "bitmap": "bitmap0", 1185 "target": "drive0.inc0.qcow2", 1186 "format": "qcow2", 1187 "sync": "incremental", 1188 "mode": "existing" 1189 } 1190 } 1191 1192 <- { "return": {} } 1193 1194#. Receive confirmation that the job completed successfully: 1195 1196 .. code:: json 1197 1198 <- { 1199 "timestamp": {...}, 1200 "data": { 1201 "device": "drive0", 1202 "type": "backup", 1203 "speed": 0, 1204 "len": 67108864, 1205 "offset": 67108864 1206 }, 1207 "event": "BLOCK_JOB_COMPLETED" 1208 } 1209 1210Example: Partial Transactional Failures 1211~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1212 1213QMP commands like `drive-backup <qemu-qmp-ref.html#index-drive_002dbackup>`_ 1214conceptually only start a job, and so transactions containing these commands 1215may succeed even if the job it created later fails. This might have surprising 1216interactions with notions of how a "transaction" ought to behave. 1217 1218This distinction means that on occasion, a transaction containing such job 1219launching commands may appear to succeed and return success, but later 1220individual jobs associated with the transaction may fail. It is possible that 1221a management application may have to deal with a partial backup failure after 1222a "successful" transaction. 1223 1224If multiple backup jobs are specified in a single transaction, if one of those 1225jobs fails, it will not interact with the other backup jobs in any way by 1226default. The job(s) that succeeded will clear the dirty bitmap associated with 1227the operation, but the job(s) that failed will not. It is therefore not safe 1228to delete any incremental backups that were created successfully in this 1229scenario, even though others failed. 1230 1231This example illustrates a transaction with two backup jobs, where one fails 1232and one succeeds: 1233 1234#. Issue the transaction to start a backup of both drives. 1235 1236 .. code:: json 1237 1238 -> { 1239 "execute": "transaction", 1240 "arguments": { 1241 "actions": [ 1242 { 1243 "type": "drive-backup", 1244 "data": { 1245 "device": "drive0", 1246 "bitmap": "bitmap0", 1247 "format": "qcow2", 1248 "mode": "existing", 1249 "sync": "incremental", 1250 "target": "drive0.inc0.qcow2" 1251 } 1252 }, 1253 { 1254 "type": "drive-backup", 1255 "data": { 1256 "device": "drive1", 1257 "bitmap": "bitmap0", 1258 "format": "qcow2", 1259 "mode": "existing", 1260 "sync": "incremental", 1261 "target": "drive1.inc0.qcow2" 1262 } 1263 }] 1264 } 1265 } 1266 1267#. Receive notice that the Transaction was accepted, and jobs were 1268 launched: 1269 1270 .. code:: json 1271 1272 <- { "return": {} } 1273 1274#. Receive notice that the first job has completed: 1275 1276 .. code:: json 1277 1278 <- { 1279 "timestamp": {...}, 1280 "data": { 1281 "device": "drive0", 1282 "type": "backup", 1283 "speed": 0, 1284 "len": 67108864, 1285 "offset": 67108864 1286 }, 1287 "event": "BLOCK_JOB_COMPLETED" 1288 } 1289 1290#. Receive notice that the second job has failed: 1291 1292 .. code:: json 1293 1294 <- { 1295 "timestamp": {...}, 1296 "data": { 1297 "device": "drive1", 1298 "action": "report", 1299 "operation": "read" 1300 }, 1301 "event": "BLOCK_JOB_ERROR" 1302 } 1303 1304 ... 1305 1306 <- { 1307 "timestamp": {...}, 1308 "data": { 1309 "speed": 0, 1310 "offset": 0, 1311 "len": 67108864, 1312 "error": "Input/output error", 1313 "device": "drive1", 1314 "type": "backup" 1315 }, 1316 "event": "BLOCK_JOB_COMPLETED" 1317 } 1318 1319At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and 1320must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be 1321deleted. If a VM-wide incremental backup of all drives at a point-in-time is 1322to be made, new backups for both drives will need to be made, taking into 1323account that a new incremental backup for drive0 needs to be based on top of 1324``drive0.inc0.qcow2``. 1325 1326For this example, an incremental backup for ``drive0`` was created, but not 1327for ``drive1``. The last VM-wide crash-consistent backup that is available in 1328this case is the full backup: 1329 1330.. code:: text 1331 1332 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] 1333 [drive1.full.qcow2] 1334 1335To repair this, issue a new incremental backup across both drives. The result 1336will be backup chains that resemble the following: 1337 1338.. code:: text 1339 1340 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2] 1341 [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2] 1342 1343Example: Grouped Completion Mode 1344~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1345 1346While jobs launched by transactions normally complete or fail individually, 1347it's possible to instruct them to complete or fail together as a group. QMP 1348transactions take an optional properties structure that can affect the 1349behavior of the transaction. 1350 1351The ``completion-mode`` transaction property can be either ``individual`` 1352which is the default legacy behavior described above, or ``grouped``, detailed 1353below. 1354 1355In ``grouped`` completion mode, no jobs will report success until all jobs are 1356ready to report success. If any job fails, all other jobs will be cancelled. 1357 1358Regardless of if a participating incremental backup job failed or was 1359cancelled, their associated bitmaps will all be held at their existing 1360points-in-time, as in individual failure cases. 1361 1362Here's the same multi-drive backup scenario from `Example: Partial 1363Transactional Failures`_, but with the ``grouped`` completion-mode property 1364applied: 1365 1366#. Issue the multi-drive incremental backup transaction: 1367 1368 .. code:: json 1369 1370 -> { 1371 "execute": "transaction", 1372 "arguments": { 1373 "properties": { 1374 "completion-mode": "grouped" 1375 }, 1376 "actions": [ 1377 { 1378 "type": "drive-backup", 1379 "data": { 1380 "device": "drive0", 1381 "bitmap": "bitmap0", 1382 "format": "qcow2", 1383 "mode": "existing", 1384 "sync": "incremental", 1385 "target": "drive0.inc0.qcow2" 1386 } 1387 }, 1388 { 1389 "type": "drive-backup", 1390 "data": { 1391 "device": "drive1", 1392 "bitmap": "bitmap0", 1393 "format": "qcow2", 1394 "mode": "existing", 1395 "sync": "incremental", 1396 "target": "drive1.inc0.qcow2" 1397 } 1398 }] 1399 } 1400 } 1401 1402#. Receive notice that the Transaction was accepted, and jobs were launched: 1403 1404 .. code:: json 1405 1406 <- { "return": {} } 1407 1408#. Receive notification that the backup job for ``drive1`` has failed: 1409 1410 .. code:: json 1411 1412 <- { 1413 "timestamp": {...}, 1414 "data": { 1415 "device": "drive1", 1416 "action": "report", 1417 "operation": "read" 1418 }, 1419 "event": "BLOCK_JOB_ERROR" 1420 } 1421 1422 <- { 1423 "timestamp": {...}, 1424 "data": { 1425 "speed": 0, 1426 "offset": 0, 1427 "len": 67108864, 1428 "error": "Input/output error", 1429 "device": "drive1", 1430 "type": "backup" 1431 }, 1432 "event": "BLOCK_JOB_COMPLETED" 1433 } 1434 1435#. Receive notification that the job for ``drive0`` has been cancelled: 1436 1437 .. code:: json 1438 1439 <- { 1440 "timestamp": {...} 1441 "data": { 1442 "device": "drive0", 1443 "type": "backup", 1444 "speed": 0, 1445 "len": 67108864, 1446 "offset": 16777216 1447 }, 1448 "event": "BLOCK_JOB_CANCELLED" 1449 } 1450 1451At the conclusion of *this* example, both jobs have been aborted due to a 1452failure. Both destination images should be deleted and are no longer of use. 1453 1454The transaction as a whole can simply be re-issued at a later time. 1455 1456.. raw:: html 1457 1458 <!-- 1459 The FreeBSD Documentation License 1460 1461 Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML, 1462 PDF, PostScript, RTF and so forth) with or without modification, are 1463 permitted provided that the following conditions are met: 1464 1465 Redistributions of source code (ReST) must retain the above copyright notice, 1466 this list of conditions and the following disclaimer of this file unmodified. 1467 1468 Redistributions in compiled form (transformed to other DTDs, converted to 1469 PDF, PostScript, RTF and other formats) must reproduce the above copyright 1470 notice, this list of conditions and the following disclaimer in the 1471 documentation and/or other materials provided with the distribution. 1472 1473 THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 1474 IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 1475 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 1476 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 1477 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 1478 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 1479 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 1480 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 1481 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 1482 ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF 1483 THE POSSIBILITY OF SUCH DAMAGE. 1484 --> 1485