1.. 2 Copyright 2019 John Snow <jsnow@redhat.com> and Red Hat, Inc. 3 All rights reserved. 4 5 This file is licensed via The FreeBSD Documentation License, the full 6 text of which is included at the end of this document. 7 8==================================== 9Dirty Bitmaps and Incremental Backup 10==================================== 11 12Dirty Bitmaps are in-memory objects that track writes to block devices. They 13can be used in conjunction with various block job operations to perform 14incremental or differential backup regimens. 15 16This document explains the conceptual mechanisms, as well as up-to-date, 17complete and comprehensive documentation on the API to manipulate them. 18(Hopefully, the "why", "what", and "how".) 19 20The intended audience for this document is developers who are adding QEMU 21backup features to management applications, or power users who run and 22administer QEMU directly via QMP. 23 24.. contents:: 25 26Overview 27-------- 28 29Bitmaps are bit vectors where each '1' bit in the vector indicates a modified 30("dirty") segment of the corresponding block device. The size of the segment 31that is tracked is the granularity of the bitmap. If the granularity of a 32bitmap is 64K, each '1' bit means that a 64K region as a whole may have 33changed in some way, possibly by as little as one byte. 34 35Smaller granularities mean more accurate tracking of modified disk data, but 36requires more computational overhead and larger bitmap sizes. Larger 37granularities mean smaller bitmap sizes, but less targeted backups. 38 39The size of a bitmap (in bytes) can be computed as such: 40 ``size`` = ceil(ceil(``image_size`` / ``granularity``) / 8) 41 42e.g. the size of a 64KiB granularity bitmap on a 2TiB image is: 43 ``size`` = ((2147483648K / 64K) / 8) 44 = 4194304B = 4MiB. 45 46QEMU uses these bitmaps when making incremental backups to know which sections 47of the file to copy out. They are not enabled by default and must be 48explicitly added in order to begin tracking writes. 49 50Bitmaps can be created at any time and can be attached to any arbitrary block 51node in the storage graph, but are most useful conceptually when attached to 52the root node attached to the guest's storage device model. 53 54That is to say: It's likely most useful to track the guest's writes to disk, 55but you could theoretically track things like qcow2 metadata changes by 56attaching the bitmap elsewhere in the storage graph. This is beyond the scope 57of this document. 58 59QEMU supports persisting these bitmaps to disk via the qcow2 image format. 60Bitmaps which are stored or loaded in this way are called "persistent", 61whereas bitmaps that are not are called "transient". 62 63QEMU also supports the migration of both transient bitmaps (tracking any 64arbitrary image format) or persistent bitmaps (qcow2) via live migration. 65 66Supported Image Formats 67----------------------- 68 69QEMU supports all documented features below on the qcow2 image format. 70 71However, qcow2 is only strictly necessary for the persistence feature, which 72writes bitmap data to disk upon close. If persistence is not required for a 73specific use case, all bitmap features excepting persistence are available for 74any arbitrary image format. 75 76For example, Dirty Bitmaps can be combined with the 'raw' image format, but 77any changes to the bitmap will be discarded upon exit. 78 79.. warning:: Transient bitmaps will not be saved on QEMU exit! Persistent 80 bitmaps are available only on qcow2 images. 81 82Dirty Bitmap Names 83------------------ 84 85Bitmap objects need a method to reference them in the API. All API-created and 86managed bitmaps have a human-readable name chosen by the user at creation 87time. 88 89- A bitmap's name is unique to the node, but bitmaps attached to different 90 nodes can share the same name. Therefore, all bitmaps are addressed via 91 their (node, name) pair. 92 93- The name of a user-created bitmap cannot be empty (""). 94 95- Transient bitmaps can have JSON unicode names that are effectively not 96 length limited. (QMP protocol may restrict messages to less than 64MiB.) 97 98- Persistent storage formats may impose their own requirements on bitmap names 99 and namespaces. Presently, only qcow2 supports persistent bitmaps. See 100 docs/interop/qcow2.txt for more details on restrictions. Notably: 101 102 - qcow2 bitmap names are limited to between 1 and 1023 bytes long. 103 104 - No two bitmaps saved to the same qcow2 file may share the same name. 105 106- QEMU occasionally uses bitmaps for internal use which have no name. They are 107 hidden from API query calls, cannot be manipulated by the external API, are 108 never persistent, nor ever migrated. 109 110Bitmap Status 111------------- 112 113Dirty Bitmap objects can be queried with the QMP command `query-block 114<qemu-qmp-ref.html#index-query_002dblock>`_, and are visible via the 115`BlockDirtyInfo <qemu-qmp-ref.html#index-BlockDirtyInfo>`_ QAPI structure. 116 117This struct shows the name, granularity, and dirty byte count for each bitmap. 118Additionally, it shows several boolean status indicators: 119 120- ``recording``: This bitmap is recording writes. 121- ``busy``: This bitmap is in-use by an operation. 122- ``persistent``: This bitmap is a persistent type. 123- ``inconsistent``: This bitmap is corrupted and cannot be used. 124 125The ``+busy`` status prohibits you from deleting, clearing, or otherwise 126modifying a bitmap, and happens when the bitmap is being used for a backup 127operation or is in the process of being loaded from a migration. Many of the 128commands documented below will refuse to work on such bitmaps. 129 130The ``+inconsistent`` status similarly prohibits almost all operations, 131notably allowing only the ``block-dirty-bitmap-remove`` operation. 132 133There is also a deprecated ``status`` field of type `DirtyBitmapStatus 134<qemu-qmp-ref.html#index-DirtyBitmapStatus>`_. A bitmap historically had 135five visible states: 136 137 #. ``Frozen``: This bitmap is currently in-use by an operation and is 138 immutable. It can't be deleted, renamed, reset, etc. 139 140 (This is now ``+busy``.) 141 142 #. ``Disabled``: This bitmap is not recording new writes. 143 144 (This is now ``-recording -busy``.) 145 146 #. ``Active``: This bitmap is recording new writes. 147 148 (This is now ``+recording -busy``.) 149 150 #. ``Locked``: This bitmap is in-use by an operation, and is immutable. 151 The difference from "Frozen" was primarily implementation details. 152 153 (This is now ``+busy``.) 154 155 #. ``Inconsistent``: This persistent bitmap was not saved to disk 156 correctly, and can no longer be used. It remains in memory to serve as 157 an indicator of failure. 158 159 (This is now ``+inconsistent``.) 160 161These states are directly replaced by the status indicators and should not be 162used. The difference between ``Frozen`` and ``Locked`` is an implementation 163detail and should not be relevant to external users. 164 165Basic QMP Usage 166--------------- 167 168The primary interface to manipulating bitmap objects is via the QMP 169interface. If you are not familiar, see docs/interop/qmp-intro.txt for a broad 170overview, and `qemu-qmp-ref <qemu-qmp-ref.html>`_ for a full reference of all 171QMP commands. 172 173Supported Commands 174~~~~~~~~~~~~~~~~~~ 175 176There are six primary bitmap-management API commands: 177 178- ``block-dirty-bitmap-add`` 179- ``block-dirty-bitmap-remove`` 180- ``block-dirty-bitmap-clear`` 181- ``block-dirty-bitmap-disable`` 182- ``block-dirty-bitmap-enable`` 183- ``block-dirty-bitmap-merge`` 184 185And one related query command: 186 187- ``query-block`` 188 189Creation: block-dirty-bitmap-add 190~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 191 192`block-dirty-bitmap-add 193<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dadd>`_: 194 195Creates a new bitmap that tracks writes to the specified node. granularity, 196persistence, and recording state can be adjusted at creation time. 197 198.. admonition:: Example 199 200 to create a new, actively recording persistent bitmap: 201 202 .. code-block:: QMP 203 204 -> { "execute": "block-dirty-bitmap-add", 205 "arguments": { 206 "node": "drive0", 207 "name": "bitmap0", 208 "persistent": true, 209 } 210 } 211 212 <- { "return": {} } 213 214- This bitmap will have a default granularity that matches the cluster size of 215 its associated drive, if available, clamped to between [4KiB, 64KiB]. The 216 current default for qcow2 is 64KiB. 217 218.. admonition:: Example 219 220 To create a new, disabled (``-recording``), transient bitmap that tracks 221 changes in 32KiB segments: 222 223 .. code-block:: QMP 224 225 -> { "execute": "block-dirty-bitmap-add", 226 "arguments": { 227 "node": "drive0", 228 "name": "bitmap1", 229 "granularity": 32768, 230 "disabled": true 231 } 232 } 233 234 <- { "return": {} } 235 236Deletion: block-dirty-bitmap-remove 237~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 238 239`block-dirty-bitmap-remove 240<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dremove>`_: 241 242Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed. 243 244- Deleting a bitmap does not impact any other bitmaps attached to the same 245 node, nor does it affect any backups already created from this bitmap or 246 node. 247 248- Because bitmaps are only unique to the node to which they are attached, you 249 must specify the node/drive name here, too. 250 251- Deleting a persistent bitmap will remove it from the qcow2 file. 252 253.. admonition:: Example 254 255 Remove a bitmap named ``bitmap0`` from node ``drive0``: 256 257 .. code-block:: QMP 258 259 -> { "execute": "block-dirty-bitmap-remove", 260 "arguments": { 261 "node": "drive0", 262 "name": "bitmap0" 263 } 264 } 265 266 <- { "return": {} } 267 268Resetting: block-dirty-bitmap-clear 269~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 270 271`block-dirty-bitmap-clear 272<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dclear>`_: 273 274Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared. 275 276- An incremental backup created from an empty bitmap will copy no data, as if 277 nothing has changed. 278 279.. admonition:: Example 280 281 Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``: 282 283 .. code-block:: QMP 284 285 -> { "execute": "block-dirty-bitmap-clear", 286 "arguments": { 287 "node": "drive0", 288 "name": "bitmap0" 289 } 290 } 291 292 <- { "return": {} } 293 294Enabling: block-dirty-bitmap-enable 295~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 296 297`block-dirty-bitmap-enable 298<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002denable>`_: 299 300"Enables" a bitmap, setting the ``recording`` bit to true, causing writes to 301begin being recorded. ``+busy`` bitmaps cannot be enabled. 302 303- Bitmaps default to being enabled when created, unless configured otherwise. 304 305- Persistent enabled bitmaps will remember their ``+recording`` status on 306 load. 307 308.. admonition:: Example 309 310 To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``: 311 312 .. code-block:: QMP 313 314 -> { "execute": "block-dirty-bitmap-enable", 315 "arguments": { 316 "node": "drive0", 317 "name": "bitmap0" 318 } 319 } 320 321 <- { "return": {} } 322 323Enabling: block-dirty-bitmap-disable 324~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 325 326`block-dirty-bitmap-disable 327<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002ddisable>`_: 328 329"Disables" a bitmap, setting the ``recording`` bit to false, causing further 330writes to begin being ignored. ``+busy`` bitmaps cannot be disabled. 331 332.. warning:: 333 334 This is potentially dangerous: QEMU makes no effort to stop any writes if 335 there are disabled bitmaps on a node, and will not mark any disabled bitmaps 336 as ``+inconsistent`` if any such writes do happen. Backups made from such 337 bitmaps will not be able to be used to reconstruct a coherent image. 338 339- Disabling a bitmap may be useful for examining which sectors of a disk 340 changed during a specific time period, or for explicit management of 341 differential backup windows. 342 343- Persistent disabled bitmaps will remember their ``-recording`` status on 344 load. 345 346.. admonition:: Example 347 348 To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``: 349 350 .. code-block:: QMP 351 352 -> { "execute": "block-dirty-bitmap-disable", 353 "arguments": { 354 "node": "drive0", 355 "name": "bitmap0" 356 } 357 } 358 359 <- { "return": {} } 360 361Merging, Copying: block-dirty-bitmap-merge 362~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 363 364`block-dirty-bitmap-merge 365<qemu-qmp-ref.html#index-block_002ddirty_002dbitmap_002dmerge>`_: 366 367Merges one or more bitmaps into a target bitmap. For any segment that is dirty 368in any one source bitmap, the target bitmap will mark that segment dirty. 369 370- Merge takes one or more bitmaps as a source and merges them together into a 371 single destination, such that any segment marked as dirty in any source 372 bitmap(s) will be marked dirty in the destination bitmap. 373 374- Merge does not create the destination bitmap if it does not exist. A blank 375 bitmap can be created beforehand to achieve the same effect. 376 377- The destination is not cleared prior to merge, so subsequent merge 378 operations will continue to cumulatively mark more segments as dirty. 379 380- If the merge operation should fail, the destination bitmap is guaranteed to 381 be unmodified. The operation may fail if the source or destination bitmaps 382 are busy, or have different granularities. 383 384- Bitmaps can only be merged on the same node. There is only one "node" 385 argument, so all bitmaps must be attached to that same node. 386 387- Copy can be achieved by merging from a single source to an empty 388 destination. 389 390.. admonition:: Example 391 392 Merge the data from ``bitmap0`` into the bitmap ``new_bitmap`` on node 393 ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves 394 a copy. 395 396 .. code-block:: QMP 397 398 -> { "execute": "block-dirty-bitmap-merge", 399 "arguments": { 400 "node": "drive0", 401 "target": "new_bitmap", 402 "bitmaps": [ "bitmap0" ] 403 } 404 } 405 406 <- { "return": {} } 407 408Querying: query-block 409~~~~~~~~~~~~~~~~~~~~~ 410 411`query-block 412<qemu-qmp-ref.html#index-query_002dblock>`_: 413 414Not strictly a bitmaps command, but will return information about any bitmaps 415attached to nodes serving as the root for guest devices. 416 417- The "inconsistent" bit will not appear when it is false, appearing only when 418 the value is true to indicate there is a problem. 419 420.. admonition:: Example 421 422 Query the block sub-system of QEMU. The following json has trimmed irrelevant 423 keys from the response to highlight only the bitmap-relevant portions of the 424 API. This result highlights a bitmap ``bitmap0`` attached to the root node of 425 device ``drive0``. 426 427 .. code-block:: QMP 428 429 -> { 430 "execute": "query-block", 431 "arguments": {} 432 } 433 434 <- { 435 "return": [ { 436 "dirty-bitmaps": [ { 437 "status": "active", 438 "count": 0, 439 "busy": false, 440 "name": "bitmap0", 441 "persistent": false, 442 "recording": true, 443 "granularity": 65536 444 } ], 445 "device": "drive0", 446 } ] 447 } 448 449Bitmap Persistence 450------------------ 451 452As outlined in `Supported Image Formats`_, QEMU can persist bitmaps to qcow2 453files. Demonstrated in `Creation: block-dirty-bitmap-add`_, passing 454``persistent: true`` to ``block-dirty-bitmap-add`` will persist that bitmap to 455disk. 456 457Persistent bitmaps will be automatically loaded into memory upon load, and 458will be written back to disk upon close. Their usage should be mostly 459transparent. 460 461However, if QEMU does not get a chance to close the file cleanly, the bitmap 462will be marked as ``+inconsistent`` at next load and considered unsafe to use 463for any operation. At this point, the only valid operation on such bitmaps is 464``block-dirty-bitmap-remove``. 465 466Losing a bitmap in this way does not invalidate any existing backups that have 467been made from this bitmap, but no further backups will be able to be issued 468for this chain. 469 470Transactions 471------------ 472 473Transactions are a QMP feature that allows you to submit multiple QMP commands 474at once, being guaranteed that they will all succeed or fail atomically, 475together. The interaction of bitmaps and transactions are demonstrated below. 476 477See `transaction <qemu-qmp.ref.html#index-transaction>`_ in the QMP reference 478for more details. 479 480Justification 481~~~~~~~~~~~~~ 482 483Bitmaps can generally be modified at any time, but certain operations often 484only make sense when paired directly with other commands. When a VM is paused, 485it's easy to ensure that no guest writes occur between individual QMP 486commands. When a VM is running, this is difficult to accomplish with 487individual QMP commands that may allow guest writes to occur between each 488command. 489 490For example, using only individual QMP commands, we could: 491 492#. Boot the VM in a paused state. 493#. Create a full drive backup of drive0. 494#. Create a new bitmap attached to drive0, confident that nothing has been 495 written to drive0 in the meantime. 496#. Resume execution of the VM. 497#. At a later point, issue incremental backups from ``bitmap0``. 498 499At this point, the bitmap and drive backup would be correctly in sync, and 500incremental backups made from this point forward would be correctly aligned to 501the full drive backup. 502 503This is not particularly useful if we decide we want to start incremental 504backups after the VM has been running for a while, for which we would want to 505perform actions such as the following: 506 507#. Boot the VM and begin execution. 508#. Using a single transaction, perform the following operations: 509 510 - Create ``bitmap0``. 511 - Create a full drive backup of ``drive0``. 512 513#. At a later point, issue incremental backups from ``bitmap0``. 514 515.. note:: As a consideration, if ``bitmap0`` is created prior to the full 516 drive backup, incremental backups can still be authored from this 517 bitmap, but they will copy extra segments reflecting writes that 518 occurred prior to the backup operation. Transactions allow us to 519 narrow critical points in time to reduce waste, or, in the other 520 direction, to ensure that no segments are omitted. 521 522Supported Bitmap Transactions 523~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 524 525- ``block-dirty-bitmap-add`` 526- ``block-dirty-bitmap-clear`` 527- ``block-dirty-bitmap-enable`` 528- ``block-dirty-bitmap-disable`` 529- ``block-dirty-bitmap-merge`` 530 531The usages for these commands are identical to their respective QMP commands, 532but see the sections below for concrete examples. 533 534Incremental Backups - Push Model 535-------------------------------- 536 537Incremental backups are simply partial disk images that can be combined with 538other partial disk images on top of a base image to reconstruct a full backup 539from the point in time at which the incremental backup was issued. 540 541The "Push Model" here references the fact that QEMU is "pushing" the modified 542blocks out to a destination. We will be using the `blockdev-backup 543<qemu-qmp-ref.html#index-blockdev_002dbackup>`_ QMP command to create both 544full and incremental backups. 545 546The command is a background job, which has its own QMP API for querying and 547management documented in `Background jobs 548<qemu-qmp-ref.html#Background-jobs>`_. 549 550Example: New Incremental Backup Anchor Point 551~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 552 553As outlined in the Transactions - `Justification`_ section, perhaps we want to 554create a new incremental backup chain attached to a drive. 555 556This example creates a new, full backup of "drive0" and accompanies it with a 557new, empty bitmap that records writes from this point in time forward. 558 559The target can be created with the help of `blockdev-add 560<qemu-qmp-ref.html#index-blockdev_002dadd>`_ or `blockdev-create 561<qemu-qmp-ref.html#index-blockdev_002dcreate>`_ command. 562 563.. note:: Any new writes that happen after this command is issued, even while 564 the backup job runs, will be written locally and not to the backup 565 destination. These writes will be recorded in the bitmap 566 accordingly. 567 568.. code-block:: QMP 569 570 -> { 571 "execute": "transaction", 572 "arguments": { 573 "actions": [ 574 { 575 "type": "block-dirty-bitmap-add", 576 "data": { 577 "node": "drive0", 578 "name": "bitmap0" 579 } 580 }, 581 { 582 "type": "blockdev-backup", 583 "data": { 584 "device": "drive0", 585 "target": "target0", 586 "sync": "full" 587 } 588 } 589 ] 590 } 591 } 592 593 <- { "return": {} } 594 595 <- { 596 "timestamp": { 597 "seconds": 1555436945, 598 "microseconds": 179620 599 }, 600 "data": { 601 "status": "created", 602 "id": "drive0" 603 }, 604 "event": "JOB_STATUS_CHANGE" 605 } 606 607 ... 608 609 <- { 610 "timestamp": {...}, 611 "data": { 612 "device": "drive0", 613 "type": "backup", 614 "speed": 0, 615 "len": 68719476736, 616 "offset": 68719476736 617 }, 618 "event": "BLOCK_JOB_COMPLETED" 619 } 620 621 <- { 622 "timestamp": {...}, 623 "data": { 624 "status": "concluded", 625 "id": "drive0" 626 }, 627 "event": "JOB_STATUS_CHANGE" 628 } 629 630 <- { 631 "timestamp": {...}, 632 "data": { 633 "status": "null", 634 "id": "drive0" 635 }, 636 "event": "JOB_STATUS_CHANGE" 637 } 638 639A full explanation of the job transition semantics and the JOB_STATUS_CHANGE 640event are beyond the scope of this document and will be omitted in all 641subsequent examples; above, several more events have been omitted for brevity. 642 643.. note:: Subsequent examples will omit all events except BLOCK_JOB_COMPLETED 644 except where necessary to illustrate workflow differences. 645 646 Omitted events and json objects will be represented by ellipses: 647 ``...`` 648 649Example: Resetting an Incremental Backup Anchor Point 650~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 651 652If we want to start a new backup chain with an existing bitmap, we can also 653use a transaction to reset the bitmap while making a new full backup: 654 655.. code-block:: QMP 656 657 -> { 658 "execute": "transaction", 659 "arguments": { 660 "actions": [ 661 { 662 "type": "block-dirty-bitmap-clear", 663 "data": { 664 "node": "drive0", 665 "name": "bitmap0" 666 } 667 }, 668 { 669 "type": "blockdev-backup", 670 "data": { 671 "device": "drive0", 672 "target": "target0", 673 "sync": "full" 674 } 675 } 676 ] 677 } 678 } 679 680 <- { "return": {} } 681 682 ... 683 684 <- { 685 "timestamp": {...}, 686 "data": { 687 "device": "drive0", 688 "type": "backup", 689 "speed": 0, 690 "len": 68719476736, 691 "offset": 68719476736 692 }, 693 "event": "BLOCK_JOB_COMPLETED" 694 } 695 696 ... 697 698The result of this example is identical to the first, but we clear an existing 699bitmap instead of adding a new one. 700 701.. tip:: In both of these examples, "bitmap0" is tied conceptually to the 702 creation of new, full backups. This relationship is not saved or 703 remembered by QEMU; it is up to the operator or management layer to 704 remember which bitmaps are associated with which backups. 705 706Example: First Incremental Backup 707~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 708 709#. Create a full backup and sync it to a dirty bitmap using any method: 710 711 - Either of the two live backup method demonstrated above, 712 - Using QMP commands with the VM paused as in the `Justification`_ section, 713 or 714 - With the VM offline, manually copy the image and start the VM in a paused 715 state, careful to add a new bitmap before the VM begins execution. 716 717 Whichever method is chosen, let's assume that at the end of this step: 718 719 - The full backup is named ``drive0.full.qcow2``. 720 - The bitmap we created is named ``bitmap0``, attached to ``drive0``. 721 722#. Create a destination image for the incremental backup that utilizes the 723 full backup as a backing image. 724 725 - Let's assume the new incremental image is named ``drive0.inc0.qcow2``: 726 727 .. code:: bash 728 729 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 730 -b drive0.full.qcow2 -F qcow2 731 732#. Add target block node: 733 734 .. code-block:: QMP 735 736 -> { 737 "execute": "blockdev-add", 738 "arguments": { 739 "node-name": "target0", 740 "driver": "qcow2", 741 "file": { 742 "driver": "file", 743 "filename": "drive0.inc0.qcow2" 744 } 745 } 746 } 747 748 <- { "return": {} } 749 750#. Issue an incremental backup command: 751 752 .. code-block:: QMP 753 754 -> { 755 "execute": "blockdev-backup", 756 "arguments": { 757 "device": "drive0", 758 "bitmap": "bitmap0", 759 "target": "target0", 760 "sync": "incremental" 761 } 762 } 763 764 <- { "return": {} } 765 766 ... 767 768 <- { 769 "timestamp": {...}, 770 "data": { 771 "device": "drive0", 772 "type": "backup", 773 "speed": 0, 774 "len": 68719476736, 775 "offset": 68719476736 776 }, 777 "event": "BLOCK_JOB_COMPLETED" 778 } 779 780 ... 781 782This copies any blocks modified since the full backup was created into the 783``drive0.inc0.qcow2`` file. During the operation, ``bitmap0`` is marked 784``+busy``. If the operation is successful, ``bitmap0`` will be cleared to 785reflect the "incremental" backup regimen, which only copies out new changes 786from each incremental backup. 787 788.. note:: Any new writes that occur after the backup operation starts do not 789 get copied to the destination. The backup's "point in time" is when 790 the backup starts, not when it ends. These writes are recorded in a 791 special bitmap that gets re-added to bitmap0 when the backup ends so 792 that the next incremental backup can copy them out. 793 794Example: Second Incremental Backup 795~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 796 797#. Create a new destination image for the incremental backup that points to 798 the previous one, e.g.: ``drive0.inc1.qcow2`` 799 800 .. code:: bash 801 802 $ qemu-img create -f qcow2 drive0.inc1.qcow2 \ 803 -b drive0.inc0.qcow2 -F qcow2 804 805#. Add target block node: 806 807 .. code-block:: QMP 808 809 -> { 810 "execute": "blockdev-add", 811 "arguments": { 812 "node-name": "target0", 813 "driver": "qcow2", 814 "file": { 815 "driver": "file", 816 "filename": "drive0.inc1.qcow2" 817 } 818 } 819 } 820 821 <- { "return": {} } 822 823#. Issue a new incremental backup command. The only difference here is that we 824 have changed the target image below. 825 826 .. code-block:: QMP 827 828 -> { 829 "execute": "blockdev-backup", 830 "arguments": { 831 "device": "drive0", 832 "bitmap": "bitmap0", 833 "target": "target0", 834 "sync": "incremental" 835 } 836 } 837 838 <- { "return": {} } 839 840 ... 841 842 <- { 843 "timestamp": {...}, 844 "data": { 845 "device": "drive0", 846 "type": "backup", 847 "speed": 0, 848 "len": 68719476736, 849 "offset": 68719476736 850 }, 851 "event": "BLOCK_JOB_COMPLETED" 852 } 853 854 ... 855 856Because the first incremental backup from the previous example completed 857successfully, ``bitmap0`` was synchronized with ``drive0.inc0.qcow2``. Here, 858we use ``bitmap0`` again to create a new incremental backup that targets the 859previous one, creating a chain of three images: 860 861.. admonition:: Diagram 862 863 .. code:: text 864 865 +-------------------+ +-------------------+ +-------------------+ 866 | drive0.full.qcow2 |<--| drive0.inc0.qcow2 |<--| drive0.inc1.qcow2 | 867 +-------------------+ +-------------------+ +-------------------+ 868 869Each new incremental backup re-synchronizes the bitmap to the latest backup 870authored, allowing a user to continue to "consume" it to create new backups on 871top of an existing chain. 872 873In the above diagram, neither drive0.inc1.qcow2 nor drive0.inc0.qcow2 are 874complete images by themselves, but rely on their backing chain to reconstruct 875a full image. The dependency terminates with each full backup. 876 877Each backup in this chain remains independent, and is unchanged by new entries 878made later in the chain. For instance, drive0.inc0.qcow2 remains a perfectly 879valid backup of the disk as it was when that backup was issued. 880 881Example: Incremental Push Backups without Backing Files 882~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 883 884Backup images are best kept off-site, so we often will not have the preceding 885backups in a chain available to link against. This is not a problem at backup 886time; we simply do not set the backing image when creating the destination 887image: 888 889#. Create a new destination image with no backing file set. We will need to 890 specify the size of the base image, because the backing file isn't 891 available for QEMU to use to determine it. 892 893 .. code:: bash 894 895 $ qemu-img create -f qcow2 drive0.inc2.qcow2 64G 896 897 .. note:: Alternatively, you can omit ``mode: "existing"`` from the push 898 backup commands to have QEMU create an image without a backing 899 file for you, but you lose control over format options like 900 compatibility and preallocation presets. 901 902#. Add target block node: 903 904 .. code-block:: QMP 905 906 -> { 907 "execute": "blockdev-add", 908 "arguments": { 909 "node-name": "target0", 910 "driver": "qcow2", 911 "file": { 912 "driver": "file", 913 "filename": "drive0.inc2.qcow2" 914 } 915 } 916 } 917 918 <- { "return": {} } 919 920#. Issue a new incremental backup command. Apart from the new destination 921 image, there is no difference from the last two examples. 922 923 .. code-block:: QMP 924 925 -> { 926 "execute": "blockdev-backup", 927 "arguments": { 928 "device": "drive0", 929 "bitmap": "bitmap0", 930 "target": "target0", 931 "sync": "incremental" 932 } 933 } 934 935 <- { "return": {} } 936 937 ... 938 939 <- { 940 "timestamp": {...}, 941 "data": { 942 "device": "drive0", 943 "type": "backup", 944 "speed": 0, 945 "len": 68719476736, 946 "offset": 68719476736 947 }, 948 "event": "BLOCK_JOB_COMPLETED" 949 } 950 951 ... 952 953The only difference from the perspective of the user is that you will need to 954set the backing image when attempting to restore the backup: 955 956.. code:: bash 957 958 $ qemu-img rebase drive0.inc2.qcow2 \ 959 -u -b drive0.inc1.qcow2 960 961This uses the "unsafe" rebase mode to simply set the backing file to a file 962that isn't present. 963 964It is also possible to use ``--image-opts`` to specify the entire backing 965chain by hand as an ephemeral property at runtime, but that is beyond the 966scope of this document. 967 968Example: Multi-drive Incremental Backup 969~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 970 971Assume we have a VM with two drives, "drive0" and "drive1" and we wish to back 972both of them up such that the two backups represent the same crash-consistent 973point in time. 974 975#. For each drive, create an empty image: 976 977 .. code:: bash 978 979 $ qemu-img create -f qcow2 drive0.full.qcow2 64G 980 $ qemu-img create -f qcow2 drive1.full.qcow2 64G 981 982#. Add target block nodes: 983 984 .. code-block:: QMP 985 986 -> { 987 "execute": "blockdev-add", 988 "arguments": { 989 "node-name": "target0", 990 "driver": "qcow2", 991 "file": { 992 "driver": "file", 993 "filename": "drive0.full.qcow2" 994 } 995 } 996 } 997 998 <- { "return": {} } 999 1000 -> { 1001 "execute": "blockdev-add", 1002 "arguments": { 1003 "node-name": "target1", 1004 "driver": "qcow2", 1005 "file": { 1006 "driver": "file", 1007 "filename": "drive1.full.qcow2" 1008 } 1009 } 1010 } 1011 1012 <- { "return": {} } 1013 1014#. Create a full (anchor) backup for each drive, with accompanying bitmaps: 1015 1016 .. code-block:: QMP 1017 1018 -> { 1019 "execute": "transaction", 1020 "arguments": { 1021 "actions": [ 1022 { 1023 "type": "block-dirty-bitmap-add", 1024 "data": { 1025 "node": "drive0", 1026 "name": "bitmap0" 1027 } 1028 }, 1029 { 1030 "type": "block-dirty-bitmap-add", 1031 "data": { 1032 "node": "drive1", 1033 "name": "bitmap0" 1034 } 1035 }, 1036 { 1037 "type": "blockdev-backup", 1038 "data": { 1039 "device": "drive0", 1040 "target": "target0", 1041 "sync": "full" 1042 } 1043 }, 1044 { 1045 "type": "blockdev-backup", 1046 "data": { 1047 "device": "drive1", 1048 "target": "target1", 1049 "sync": "full" 1050 } 1051 } 1052 ] 1053 } 1054 } 1055 1056 <- { "return": {} } 1057 1058 ... 1059 1060 <- { 1061 "timestamp": {...}, 1062 "data": { 1063 "device": "drive0", 1064 "type": "backup", 1065 "speed": 0, 1066 "len": 68719476736, 1067 "offset": 68719476736 1068 }, 1069 "event": "BLOCK_JOB_COMPLETED" 1070 } 1071 1072 ... 1073 1074 <- { 1075 "timestamp": {...}, 1076 "data": { 1077 "device": "drive1", 1078 "type": "backup", 1079 "speed": 0, 1080 "len": 68719476736, 1081 "offset": 68719476736 1082 }, 1083 "event": "BLOCK_JOB_COMPLETED" 1084 } 1085 1086 ... 1087 1088#. Later, create new destination images for each of the incremental backups 1089 that point to their respective full backups: 1090 1091 .. code:: bash 1092 1093 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1094 -b drive0.full.qcow2 -F qcow2 1095 $ qemu-img create -f qcow2 drive1.inc0.qcow2 \ 1096 -b drive1.full.qcow2 -F qcow2 1097 1098#. Add target block nodes: 1099 1100 .. code-block:: QMP 1101 1102 -> { 1103 "execute": "blockdev-add", 1104 "arguments": { 1105 "node-name": "target0", 1106 "driver": "qcow2", 1107 "file": { 1108 "driver": "file", 1109 "filename": "drive0.inc0.qcow2" 1110 } 1111 } 1112 } 1113 1114 <- { "return": {} } 1115 1116 -> { 1117 "execute": "blockdev-add", 1118 "arguments": { 1119 "node-name": "target1", 1120 "driver": "qcow2", 1121 "file": { 1122 "driver": "file", 1123 "filename": "drive1.inc0.qcow2" 1124 } 1125 } 1126 } 1127 1128 <- { "return": {} } 1129 1130#. Issue a multi-drive incremental push backup transaction: 1131 1132 .. code-block:: QMP 1133 1134 -> { 1135 "execute": "transaction", 1136 "arguments": { 1137 "actions": [ 1138 { 1139 "type": "blockev-backup", 1140 "data": { 1141 "device": "drive0", 1142 "bitmap": "bitmap0", 1143 "sync": "incremental", 1144 "target": "target0" 1145 } 1146 }, 1147 { 1148 "type": "blockdev-backup", 1149 "data": { 1150 "device": "drive1", 1151 "bitmap": "bitmap0", 1152 "sync": "incremental", 1153 "target": "target1" 1154 } 1155 }, 1156 ] 1157 } 1158 } 1159 1160 <- { "return": {} } 1161 1162 ... 1163 1164 <- { 1165 "timestamp": {...}, 1166 "data": { 1167 "device": "drive0", 1168 "type": "backup", 1169 "speed": 0, 1170 "len": 68719476736, 1171 "offset": 68719476736 1172 }, 1173 "event": "BLOCK_JOB_COMPLETED" 1174 } 1175 1176 ... 1177 1178 <- { 1179 "timestamp": {...}, 1180 "data": { 1181 "device": "drive1", 1182 "type": "backup", 1183 "speed": 0, 1184 "len": 68719476736, 1185 "offset": 68719476736 1186 }, 1187 "event": "BLOCK_JOB_COMPLETED" 1188 } 1189 1190 ... 1191 1192Push Backup Errors & Recovery 1193----------------------------- 1194 1195In the event of an error that occurs after a push backup job is successfully 1196launched, either by an individual QMP command or a QMP transaction, the user 1197will receive a ``BLOCK_JOB_COMPLETE`` event with a failure message, 1198accompanied by a ``BLOCK_JOB_ERROR`` event. 1199 1200In the case of a job being cancelled, the user will receive a 1201``BLOCK_JOB_CANCELLED`` event instead of a pair of COMPLETE and ERROR 1202events. 1203 1204In either failure case, the bitmap used for the failed operation is not 1205cleared. It will contain all of the dirty bits it did at the start of the 1206operation, plus any new bits that got marked during the operation. 1207 1208Effectively, the "point in time" that a bitmap is recording differences 1209against is kept at the issuance of the last successful incremental backup, 1210instead of being moved forward to the start of this now-failed backup. 1211 1212Once the underlying problem is addressed (e.g. more storage space is allocated 1213on the destination), the incremental backup command can be retried with the 1214same bitmap. 1215 1216Example: Individual Failures 1217~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1218 1219Incremental Push Backup jobs that fail individually behave simply as 1220described above. This example demonstrates the single-job failure case: 1221 1222#. Create a target image: 1223 1224 .. code:: bash 1225 1226 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1227 -b drive0.full.qcow2 -F qcow2 1228 1229#. Add target block node: 1230 1231 .. code-block:: QMP 1232 1233 -> { 1234 "execute": "blockdev-add", 1235 "arguments": { 1236 "node-name": "target0", 1237 "driver": "qcow2", 1238 "file": { 1239 "driver": "file", 1240 "filename": "drive0.inc0.qcow2" 1241 } 1242 } 1243 } 1244 1245 <- { "return": {} } 1246 1247#. Attempt to create an incremental backup via QMP: 1248 1249 .. code-block:: QMP 1250 1251 -> { 1252 "execute": "blockdev-backup", 1253 "arguments": { 1254 "device": "drive0", 1255 "bitmap": "bitmap0", 1256 "target": "target0", 1257 "sync": "incremental" 1258 } 1259 } 1260 1261 <- { "return": {} } 1262 1263#. Receive a pair of events indicating failure: 1264 1265 .. code-block:: QMP 1266 1267 <- { 1268 "timestamp": {...}, 1269 "data": { 1270 "device": "drive0", 1271 "action": "report", 1272 "operation": "write" 1273 }, 1274 "event": "BLOCK_JOB_ERROR" 1275 } 1276 1277 <- { 1278 "timestamp": {...}, 1279 "data": { 1280 "speed": 0, 1281 "offset": 0, 1282 "len": 67108864, 1283 "error": "No space left on device", 1284 "device": "drive0", 1285 "type": "backup" 1286 }, 1287 "event": "BLOCK_JOB_COMPLETED" 1288 } 1289 1290#. Remove target node: 1291 1292 .. code-block:: QMP 1293 1294 -> { 1295 "execute": "blockdev-del", 1296 "arguments": { 1297 "node-name": "target0", 1298 } 1299 } 1300 1301 <- { "return": {} } 1302 1303#. Delete the failed image, and re-create it. 1304 1305 .. code:: bash 1306 1307 $ rm drive0.inc0.qcow2 1308 $ qemu-img create -f qcow2 drive0.inc0.qcow2 \ 1309 -b drive0.full.qcow2 -F qcow2 1310 1311#. Add target block node: 1312 1313 .. code-block:: QMP 1314 1315 -> { 1316 "execute": "blockdev-add", 1317 "arguments": { 1318 "node-name": "target0", 1319 "driver": "qcow2", 1320 "file": { 1321 "driver": "file", 1322 "filename": "drive0.inc0.qcow2" 1323 } 1324 } 1325 } 1326 1327 <- { "return": {} } 1328 1329#. Retry the command after fixing the underlying problem, such as 1330 freeing up space on the backup volume: 1331 1332 .. code-block:: QMP 1333 1334 -> { 1335 "execute": "blockdev-backup", 1336 "arguments": { 1337 "device": "drive0", 1338 "bitmap": "bitmap0", 1339 "target": "target0", 1340 "sync": "incremental" 1341 } 1342 } 1343 1344 <- { "return": {} } 1345 1346#. Receive confirmation that the job completed successfully: 1347 1348 .. code-block:: QMP 1349 1350 <- { 1351 "timestamp": {...}, 1352 "data": { 1353 "device": "drive0", 1354 "type": "backup", 1355 "speed": 0, 1356 "len": 67108864, 1357 "offset": 67108864 1358 }, 1359 "event": "BLOCK_JOB_COMPLETED" 1360 } 1361 1362Example: Partial Transactional Failures 1363~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1364 1365QMP commands like `blockdev-backup 1366<qemu-qmp-ref.html#index-blockdev_002dbackup>`_ 1367conceptually only start a job, and so transactions containing these commands 1368may succeed even if the job it created later fails. This might have surprising 1369interactions with notions of how a "transaction" ought to behave. 1370 1371This distinction means that on occasion, a transaction containing such job 1372launching commands may appear to succeed and return success, but later 1373individual jobs associated with the transaction may fail. It is possible that 1374a management application may have to deal with a partial backup failure after 1375a "successful" transaction. 1376 1377If multiple backup jobs are specified in a single transaction, if one of those 1378jobs fails, it will not interact with the other backup jobs in any way by 1379default. The job(s) that succeeded will clear the dirty bitmap associated with 1380the operation, but the job(s) that failed will not. It is therefore not safe 1381to delete any incremental backups that were created successfully in this 1382scenario, even though others failed. 1383 1384This example illustrates a transaction with two backup jobs, where one fails 1385and one succeeds: 1386 1387#. Issue the transaction to start a backup of both drives. 1388 1389 .. code-block:: QMP 1390 1391 -> { 1392 "execute": "transaction", 1393 "arguments": { 1394 "actions": [ 1395 { 1396 "type": "blockdev-backup", 1397 "data": { 1398 "device": "drive0", 1399 "bitmap": "bitmap0", 1400 "sync": "incremental", 1401 "target": "target0" 1402 } 1403 }, 1404 { 1405 "type": "blockdev-backup", 1406 "data": { 1407 "device": "drive1", 1408 "bitmap": "bitmap0", 1409 "sync": "incremental", 1410 "target": "target1" 1411 } 1412 }] 1413 } 1414 } 1415 1416#. Receive notice that the Transaction was accepted, and jobs were 1417 launched: 1418 1419 .. code-block:: QMP 1420 1421 <- { "return": {} } 1422 1423#. Receive notice that the first job has completed: 1424 1425 .. code-block:: QMP 1426 1427 <- { 1428 "timestamp": {...}, 1429 "data": { 1430 "device": "drive0", 1431 "type": "backup", 1432 "speed": 0, 1433 "len": 67108864, 1434 "offset": 67108864 1435 }, 1436 "event": "BLOCK_JOB_COMPLETED" 1437 } 1438 1439#. Receive notice that the second job has failed: 1440 1441 .. code-block:: QMP 1442 1443 <- { 1444 "timestamp": {...}, 1445 "data": { 1446 "device": "drive1", 1447 "action": "report", 1448 "operation": "read" 1449 }, 1450 "event": "BLOCK_JOB_ERROR" 1451 } 1452 1453 ... 1454 1455 <- { 1456 "timestamp": {...}, 1457 "data": { 1458 "speed": 0, 1459 "offset": 0, 1460 "len": 67108864, 1461 "error": "Input/output error", 1462 "device": "drive1", 1463 "type": "backup" 1464 }, 1465 "event": "BLOCK_JOB_COMPLETED" 1466 } 1467 1468At the conclusion of the above example, ``drive0.inc0.qcow2`` is valid and 1469must be kept, but ``drive1.inc0.qcow2`` is incomplete and should be 1470deleted. If a VM-wide incremental backup of all drives at a point-in-time is 1471to be made, new backups for both drives will need to be made, taking into 1472account that a new incremental backup for drive0 needs to be based on top of 1473``drive0.inc0.qcow2``. 1474 1475For this example, an incremental backup for ``drive0`` was created, but not 1476for ``drive1``. The last VM-wide crash-consistent backup that is available in 1477this case is the full backup: 1478 1479.. code:: text 1480 1481 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] 1482 [drive1.full.qcow2] 1483 1484To repair this, issue a new incremental backup across both drives. The result 1485will be backup chains that resemble the following: 1486 1487.. code:: text 1488 1489 [drive0.full.qcow2] <-- [drive0.inc0.qcow2] <-- [drive0.inc1.qcow2] 1490 [drive1.full.qcow2] <-------------------------- [drive1.inc1.qcow2] 1491 1492Example: Grouped Completion Mode 1493~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1494 1495While jobs launched by transactions normally complete or fail individually, 1496it's possible to instruct them to complete or fail together as a group. QMP 1497transactions take an optional properties structure that can affect the 1498behavior of the transaction. 1499 1500The ``completion-mode`` transaction property can be either ``individual`` 1501which is the default legacy behavior described above, or ``grouped``, detailed 1502below. 1503 1504In ``grouped`` completion mode, no jobs will report success until all jobs are 1505ready to report success. If any job fails, all other jobs will be cancelled. 1506 1507Regardless of if a participating incremental backup job failed or was 1508cancelled, their associated bitmaps will all be held at their existing 1509points-in-time, as in individual failure cases. 1510 1511Here's the same multi-drive backup scenario from `Example: Partial 1512Transactional Failures`_, but with the ``grouped`` completion-mode property 1513applied: 1514 1515#. Issue the multi-drive incremental backup transaction: 1516 1517 .. code-block:: QMP 1518 1519 -> { 1520 "execute": "transaction", 1521 "arguments": { 1522 "properties": { 1523 "completion-mode": "grouped" 1524 }, 1525 "actions": [ 1526 { 1527 "type": "blockdev-backup", 1528 "data": { 1529 "device": "drive0", 1530 "bitmap": "bitmap0", 1531 "sync": "incremental", 1532 "target": "target0" 1533 } 1534 }, 1535 { 1536 "type": "blockdev-backup", 1537 "data": { 1538 "device": "drive1", 1539 "bitmap": "bitmap0", 1540 "sync": "incremental", 1541 "target": "target1" 1542 } 1543 }] 1544 } 1545 } 1546 1547#. Receive notice that the Transaction was accepted, and jobs were launched: 1548 1549 .. code-block:: QMP 1550 1551 <- { "return": {} } 1552 1553#. Receive notification that the backup job for ``drive1`` has failed: 1554 1555 .. code-block:: QMP 1556 1557 <- { 1558 "timestamp": {...}, 1559 "data": { 1560 "device": "drive1", 1561 "action": "report", 1562 "operation": "read" 1563 }, 1564 "event": "BLOCK_JOB_ERROR" 1565 } 1566 1567 <- { 1568 "timestamp": {...}, 1569 "data": { 1570 "speed": 0, 1571 "offset": 0, 1572 "len": 67108864, 1573 "error": "Input/output error", 1574 "device": "drive1", 1575 "type": "backup" 1576 }, 1577 "event": "BLOCK_JOB_COMPLETED" 1578 } 1579 1580#. Receive notification that the job for ``drive0`` has been cancelled: 1581 1582 .. code-block:: QMP 1583 1584 <- { 1585 "timestamp": {...}, 1586 "data": { 1587 "device": "drive0", 1588 "type": "backup", 1589 "speed": 0, 1590 "len": 67108864, 1591 "offset": 16777216 1592 }, 1593 "event": "BLOCK_JOB_CANCELLED" 1594 } 1595 1596At the conclusion of *this* example, both jobs have been aborted due to a 1597failure. Both destination images should be deleted and are no longer of use. 1598 1599The transaction as a whole can simply be re-issued at a later time. 1600 1601.. raw:: html 1602 1603 <!-- 1604 The FreeBSD Documentation License 1605 1606 Redistribution and use in source (ReST) and 'compiled' forms (SGML, HTML, 1607 PDF, PostScript, RTF and so forth) with or without modification, are 1608 permitted provided that the following conditions are met: 1609 1610 Redistributions of source code (ReST) must retain the above copyright notice, 1611 this list of conditions and the following disclaimer of this file unmodified. 1612 1613 Redistributions in compiled form (transformed to other DTDs, converted to 1614 PDF, PostScript, RTF and other formats) must reproduce the above copyright 1615 notice, this list of conditions and the following disclaimer in the 1616 documentation and/or other materials provided with the distribution. 1617 1618 THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 1619 IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 1620 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 1621 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 1622 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 1623 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 1624 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 1625 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 1626 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 1627 ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF 1628 THE POSSIBILITY OF SUCH DAMAGE. 1629 --> 1630