Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration bl

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration block node permissions
- iotests fixes (051 and mirror-ready-cancel-error)
- Remove an outdated comment

# gpg: Signature made Tue 01 Feb 2022 13:34:54 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* remotes/hreitz-gitlab/tags/pull-block-2022-02-01: (24 commits)
block.h: remove outdated comment
iotests/migration-permissions: New test
block-backend: Retain permissions after migration
iotests: declare lack of support for compresion_type in IMGOPTS
iotest 214: explicit compression type
iotests 60: more accurate set dirty bit in qcow2 header
iotests: bash tests: filter compression type
iotest 39: use _qcow2_dump_header
iotests: massive use _qcow2_dump_header
iotests/common.rc: introduce _qcow2_dump_header helper
qcow2: simple case support for downgrading of qcow2 images with zstd
iotest 302: use img_info_log() helper
iotests.py: filter compression type out
iotests.py: filter out successful output of qemu-img create
iotest 065: explicit compression type
iotest 303: explicit compression type
iotests.py: rewrite default luks support in qemu_img
iotests: drop qemu_img_verbose() helper
iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
iotests: specify some unsupported_imgopts for python iotests
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: drop qemu_img_verbose() helper

qemu_img_verbose() has a drawback of not going through generic
qemu_img_pipe_and_status(). qemu_img_verbose() is not very popular, so
update the only two user

iotests: drop qemu_img_verbose() helper

qemu_img_verbose() has a drawback of not going through generic
qemu_img_pipe_and_status(). qemu_img_verbose() is not very popular, so
update the only two users to qemu_img_log() and drop qemu_img_verbose()
at all.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20211223160144.1097696-6-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2017-07-11' into staging

Block layer patches

# gpg: Signature made Tue 11 Jul 2017 17:05:56 BST
# gpg: using RSA key 0x

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2017-07-11' into staging

Block layer patches

# gpg: Signature made Tue 11 Jul 2017 17:05:56 BST
# gpg: using RSA key 0xF407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2017-07-11: (85 commits)
iotests: Add preallocated growth test for qcow2
iotests: Add preallocated resize test for raw
block/qcow2: falloc/full preallocating growth
block/qcow2: Rename "fail_block" to just "fail"
block/qcow2: Add qcow2_refcount_area()
block/qcow2: Metadata preallocation for truncate
block/qcow2: Lock s->lock in preallocate()
block/qcow2: Generalize preallocate()
block/file-posix: Preallocation for truncate
block/file-posix: Generalize raw_regular_truncate
block/file-posix: Extract raw_regular_truncate()
block/file-posix: Small fixes in raw_create()
qemu-img: Expose PreallocMode for resizing
block: Add PreallocMode to blk_truncate()
block: Add PreallocMode to bdrv_truncate()
block: Add PreallocMode to BD.bdrv_truncate()
iotests: add test 178 for qemu-img measure
qemu-iotests: support per-format golden output files
qemu-img: add measure subcommand
qcow2: add bdrv_measure() support
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block/qcow2: Add qcow2_refcount_area()

This function creates a collection of self-describing refcount
structures (including a new refcount table) at the end of a qcow2 image
file. Optionally, these

block/qcow2: Add qcow2_refcount_area()

This function creates a collection of self-describing refcount
structures (including a new refcount table) at the end of a qcow2 image
file. Optionally, these structures can also describe a number of
additional clusters beyond themselves; this will be important for
preallocated truncation, which will place the data clusters and L2
tables there.

For now, we can use this function to replace the part of
alloc_refcount_block() that grows the refcount table (from which it is
actually derived).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20170613202107.10125-13-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Tue 01 Apr 2014 18:11:16 BST using RSA key ID 81AB73C8
# gpg: Good sign

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Tue 01 Apr 2014 18:11:16 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/block-pull-request: (51 commits)
qcow2: link all L2 meta updates in preallocate()
parallels: Sanity check for s->tracks (CVE-2014-0142)
parallels: Fix catalog size integer overflow (CVE-2014-0143)
qcow2: Limit snapshot table size
qcow2: Check maximum L1 size in qcow2_snapshot_load_tmp() (CVE-2014-0143)
qcow2: Fix L1 allocation size in qcow2_snapshot_load_tmp() (CVE-2014-0145)
qcow2: Fix NULL dereference in qcow2_open() error path (CVE-2014-0146)
qcow2: Fix copy_sectors() with VM state
block: Limit request size (CVE-2014-0143)
block: vdi bounds check qemu-io tests
dmg: prevent chunk buffer overflow (CVE-2014-0145)
dmg: use uint64_t consistently for sectors and lengths
dmg: sanitize chunk length and sectorcount (CVE-2014-0145)
dmg: use appropriate types when reading chunks
dmg: drop broken bdrv_pread() loop
dmg: prevent out-of-bounds array access on terminator
dmg: coding style and indentation cleanup
qcow2: Fix new L1 table size check (CVE-2014-0143)
qcow2: Protect against some integer overflows in bdrv_check
qcow2: Fix types in qcow2_alloc_clusters and alloc_clusters_noref
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: Don't rely on free_cluster_index in alloc_refcount_block() (CVE-2014-0147)

free_cluster_index is only correct if update_refcount() was called from
an allocation function, and even there it's

qcow2: Don't rely on free_cluster_index in alloc_refcount_block() (CVE-2014-0147)

free_cluster_index is only correct if update_refcount() was called from
an allocation function, and even there it's brittle because it's used to
protect unfinished allocations which still have a refcount of 0 - if it
moves in the wrong place, the unfinished allocation can be corrupted.

So not using it any more seems to be a good idea. Instead, use the
first requested cluster to do the calculations. Return -EAGAIN if
unfinished allocations could become invalid and let the caller restart
its search for some free clusters.

The context of creating a snapsnot is one situation where
update_refcount() is called outside of a cluster allocation. For this
case, the change fixes a buffer overflow if a cluster is referenced in
an L2 table that cannot be represented by an existing refcount block.
(new_table[refcount_table_index] was out of bounds)

[Bump the qemu-iotests 026 refblock_alloc.write leak count from 10 to
11.
--Stefan]

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

Merge remote-tracking branch 'stefanha/block' into staging

# By Kevin Wolf (22) and Peter Lieven (1)
# Via Stefan Hajnoczi
* stefanha/block: (23 commits)
block: Fix direct use of protocols as driv

Merge remote-tracking branch 'stefanha/block' into staging

# By Kevin Wolf (22) and Peter Lieven (1)
# Via Stefan Hajnoczi
* stefanha/block: (23 commits)
block: Fix direct use of protocols as driver for bdrv_open()
qcow2: Gather clusters in a looping loop
qcow2: Move cluster gathering to a non-looping loop
qcow2: Allow requests with multiple l2metas
qcow2: Use byte granularity in qcow2_alloc_cluster_offset()
qcow2: Prepare handle_alloc/copied() for byte granularity
qcow2: handle_copied(): Implement non-zero host_offset
qcow2: handle_copied(): Get rid of keep_clusters parameter
qcow2: handle_copied(): Get rid of nb_clusters parameter
qcow2: Factor out handle_copied()
qcow2: Clean up handle_alloc()
qcow2: Finalise interface of handle_alloc()
qcow2: handle_alloc(): Get rid of keep_clusters parameter
qcow2: handle_alloc(): Get rid of nb_clusters parameter
qcow2: Factor out handle_alloc()
qcow2: Decouple cluster allocation from cluster reuse code
qcow2: Change handle_dependency to byte granularity
qcow2: Improve check for overlapping allocations
qcow2: Handle dependencies earlier
qcow2: Remove bogus unlock of s->lock
...

show more ...

qcow2: Gather clusters in a looping loop

Instead of just checking once in exactly this order if there are
dependendies, non-COW clusters and new allocation, this starts looping
around these. This wa

qcow2: Gather clusters in a looping loop

Instead of just checking once in exactly this order if there are
dependendies, non-COW clusters and new allocation, this starts looping
around these. This way we can, for example, gather non-COW clusters after
new allocations as long as the host cluster offsets stay contiguous.

Once handle_dependencies() is extended so that COW areas of in-flight
allocations can be overwritten, this allows to continue with gathering
other clusters (we wouldn't be able to do that without this change
because we would have missed a possible second dependency in one of the
next clusters).

This means that in the typical sequential write case, we can combine the
COW overwrite of one cluster with the allocation of the next cluster as
soon as something like Delayed COW gets actually implemented. It is only
by avoiding splitting requests this way that Delayed COW actually starts
improving performance noticably.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

qcow2: Fix "total clusters" number in bdrv_check

This should be based on the virtual disk size, not on the size of the
image.

Interesting observation: With some VM state stored in the image file,
p

qcow2: Fix "total clusters" number in bdrv_check

This should be based on the virtual disk size, not on the size of the
image.

Interesting observation: With some VM state stored in the image file,
percentages higher than 100% are possible, even though snapshots
themselves are ignored. This is a qcow2 bug to be fixed another day: The
VM state should be discarded in the active L2 tables after completing
the snapshot creation.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

Merge remote-tracking branch 'kwolf/for-anthony' into staging

# By Paolo Bonzini (7) and others
# Via Kevin Wolf
* kwolf/for-anthony: (22 commits)
pc: add compatibility machine types for 1.4
blo

Merge remote-tracking branch 'kwolf/for-anthony' into staging

# By Paolo Bonzini (7) and others
# Via Kevin Wolf
* kwolf/for-anthony: (22 commits)
pc: add compatibility machine types for 1.4
blockdev: enable discard by default
qemu-nbd: add --discard option
blockdev: add discard suboption to -drive
block: implement BDRV_O_UNMAP
block: complete all IOs before .bdrv_truncate
coroutine: trim down nesting level in perf_nesting test
coroutine: move pooling to common code
qemu-iotests: Test qcow2 image creation options
qemu-iotests: Add qemu-img compare test
qemu-img: Add compare subcommand
qemu-img: Add "Quiet mode" option
block: Add synchronous wrapper for bdrv_co_is_allocated_above
block: refuse negative iops and bps values
block: use Error in do_check_io_limits()
qcow2: support compressed clusters in BlockFragInfo
qemu-img: add compressed clusters to BlockFragInfo
qemu-img: fix missing space in qemu-img check output
qcow2: record fragmentation statistics during check
qcow2: introduce check_refcounts_l1/l2() flags
...

show more ...

qemu-img: add compressed clusters to BlockFragInfo

Show how many clusters are compressed. This can be used to monitor how
many compressed clusters remain and whether to recompress the image.

Sugge

qemu-img: add compressed clusters to BlockFragInfo

Show how many clusters are compressed. This can be used to monitor how
many compressed clusters remain and whether to recompress the image.

Suggested-by: Cole Robinson <crobinso@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

qemu-img: find the image end offset during check

This patch adds the support for reporting the image end offset (in
bytes). This is particularly useful after a conversion (or a rebase)
where the des

qemu-img: find the image end offset during check

This patch adds the support for reporting the image end offset (in
bytes). This is particularly useful after a conversion (or a rebase)
where the destination is a block device in order to find the first
unused byte at the end of the image.

Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge branch 'master' of git://git.qemu.org/qemu into prep-up

Conflicts:
hw/Makefile.objs
hw/ppc_prep.c

Signed-off-by: Andreas Färber <andreas.faerber@web.de>

Merge remote-tracking branch 'kwolf/for-anthony' into staging

* kwolf/for-anthony: (26 commits)
qemu-io: Use bdrv_drain_all instead of qemu_aio_flush
megasas: Use bdrv_drain_all instead of qemu_

Merge remote-tracking branch 'kwolf/for-anthony' into staging

* kwolf/for-anthony: (26 commits)
qemu-io: Use bdrv_drain_all instead of qemu_aio_flush
megasas: Use bdrv_drain_all instead of qemu_aio_flush
vmdk: Fix data corruption bug in WRITE and READ handling
fdc: remove last usage of FD_STATE_SEEK
fdc: fix typo in zero constant
fdc: remove double affectation of FD_MSR_CMDBUSY flag
fdc-tests: add tests for VERIFY command
fdc: implement VERIFY command
fdc-test: Check READ ID
fdc: fix false FD_SR0_SEEK
fdc: fix FD_SR0_SEEK for initial seek on DMA transfers
fdc: fix FD_SR0_SEEK for non-DMA transfers and multi sectors transfers
fdc: use status0 field instead of a local variable
fdc-test: add tests for non-DMA READ command
fdc-test: insert media before fuzzing registers
fdc-test: split test_media_change() test, so insert part can be reused
fdc: Remove status0 parameter from fdctrl_set_fifo()
aio: rename AIOPool to AIOCBInfo
aio: use g_slice_alloc() for AIOCB pooling
aio: switch aiocb_size type int -> size_t
...

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

show more ...

qemu-iotests: qcow2: Test growing large refcount table

Actually writing all the content with 512 byte sector size would take
forever, therefore build the image file with a Python script and use
qemu

qemu-iotests: qcow2: Test growing large refcount table

Actually writing all the content with 512 byte sector size would take
forever, therefore build the image file with a Python script and use
qemu-io for the last write that actually triggers the refcount table
growth.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...