Merge tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu into staging

Block patches for 7.0-rc1:
- iotest fixes:
- Fix some iotests for riscv targets
- Use GNU sed in more places wher

Merge tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu into staging

Block patches for 7.0-rc1:
- iotest fixes:
- Fix some iotests for riscv targets
- Use GNU sed in more places where required
- Meson-related fixes (i.e. to print errors when they occur)
- Have qemu-img calls (from Python tests) generally raise nicely
formattable exceptions on errors
- Fix iotest 207
- Allow RBD images to be growable by writing zeroes past the end of
file, fixing qcow2 on rbd

# gpg: Signature made Tue 22 Mar 2022 11:51:10 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* tag 'pull-block-2022-03-22' of https://gitlab.com/hreitz/qemu: (25 commits)
iotests/207: Filter host fingerprint
iotests.py: Filters for VM.run_job()
iotests: make qemu_img_log and img_info_log raise on error
iotests: remove qemu_img_pipe_and_status()
iotests: replace qemu_img_log('create', ...) calls
iotests: use qemu_img() in has_working_luks()
iotests: remove remaining calls to qemu_img_pipe()
iotests/149: Remove qemu_img_pipe() call
iotests: replace unchecked calls to qemu_img_pipe()
iotests: change supports_quorum to use qemu_img
iotests: add qemu_img_map() function
iotests/remove-bitmap-from-backing: use qemu_img_info()
iotests: add qemu_img_info()
iotests: use qemu_img_json() when applicable
iotests: add qemu_img_json()
iotests: fortify compare_images() against crashes
iotests: make qemu_img raise on non-zero rc by default
iotests: Remove explicit checks for qemu_img() == 0
python/utils: add VerboseProcessError
python/utils: add add_visual_margin() text decoration utility
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: replace qemu_img_log('create', ...) calls

qemu_img_log() calls into qemu_img_pipe(), which always removes output
for 'create' commands on success anyway. Replace all of these calls to
the s

iotests: replace qemu_img_log('create', ...) calls

qemu_img_log() calls into qemu_img_pipe(), which always removes output
for 'create' commands on success anyway. Replace all of these calls to
the simpler qemu_img_create(...) which doesn't log, but raises a
detailed exception object on failure instead.

Blank lines are removed from output files where appropriate.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220321201618.903471-17-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration bl

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration block node permissions
- iotests fixes (051 and mirror-ready-cancel-error)
- Remove an outdated comment

# gpg: Signature made Tue 01 Feb 2022 13:34:54 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* remotes/hreitz-gitlab/tags/pull-block-2022-02-01: (24 commits)
block.h: remove outdated comment
iotests/migration-permissions: New test
block-backend: Retain permissions after migration
iotests: declare lack of support for compresion_type in IMGOPTS
iotest 214: explicit compression type
iotests 60: more accurate set dirty bit in qcow2 header
iotests: bash tests: filter compression type
iotest 39: use _qcow2_dump_header
iotests: massive use _qcow2_dump_header
iotests/common.rc: introduce _qcow2_dump_header helper
qcow2: simple case support for downgrading of qcow2 images with zstd
iotest 302: use img_info_log() helper
iotests.py: filter compression type out
iotests.py: filter out successful output of qemu-img create
iotest 065: explicit compression type
iotest 303: explicit compression type
iotests.py: rewrite default luks support in qemu_img
iotests: drop qemu_img_verbose() helper
iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
iotests: specify some unsupported_imgopts for python iotests
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests.py: filter out successful output of qemu-img create

The only "feature" of this "Formatting ..." line is that we have to
update it every time we add new option. Let's drop it.

Signed-off-by:

iotests.py: filter out successful output of qemu-img create

The only "feature" of this "Formatting ..." line is that we have to
update it every time we add new option. Let's drop it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20211223160144.1097696-10-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
#

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-08-26: (34 commits)
iotests: Add tests for qcow2 images with extended L2 entries
qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters
qcow2: Allow preallocation and backing files if extended_l2 is set
qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
qcow2: Add prealloc field to QCowL2Meta
qcow2: Add subcluster support to qcow2_measure()
qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
qcow2: Add subcluster support to handle_alloc_space()
qcow2: Clear the L2 bitmap when allocating a compressed cluster
qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
qcow2: Add subcluster support to check_refcounts_l2()
qcow2: Add subcluster support to discard_in_l2_slice()
qcow2: Add subcluster support to zero_in_l2_slice()
qcow2: Add subcluster support to qcow2_get_host_offset()
qcow2: Add subcluster support to calculate_l2_meta()
qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
qcow2: Add cluster type parameter to qcow2_get_host_offset()
qcow2: Add qcow2_cluster_is_allocated()
qcow2: Add qcow2_get_subcluster_range_type()
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with th

qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with this feature,
which we call "extended L2 entries".

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <6476caaa73216bd05b7bb2d504a20415e1665176.1594396418.git.berto@igalia.com>
[mreitz: %s/5\.1/5.2/; fixed 302's and 303's reference output]
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-06' into staging

Block patches for 5.1:
- LUKS keyslot amendment
(+ patches to make the iotests pass on non-Linux systems, an

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-06' into staging

Block patches for 5.1:
- LUKS keyslot amendment
(+ patches to make the iotests pass on non-Linux systems, and to keep
the tests passing for qcow v1, and to skip LUKS tests (including
qcow2 LUKS) when the built qemu does not support it)
- Refactoring in the block layer: Drop the basically unnecessary
unallocated_blocks_are_zero field from BlockDriverInfo
- Fix qcow2 preallocation when the image size is not a multiple of the
cluster size
- Fix in block-copy code

# gpg: Signature made Mon 06 Jul 2020 11:02:53 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-07-06: (31 commits)
qed: Simplify backing reads
block: drop unallocated_blocks_are_zero
block/vhdx: drop unallocated_blocks_are_zero
block/file-posix: drop unallocated_blocks_are_zero
block/iscsi: drop unallocated_blocks_are_zero
block/crypto: drop unallocated_blocks_are_zero
block/vpc: return ZERO block-status when appropriate
block/vdi: return ZERO block-status when appropriate
block: inline bdrv_unallocated_blocks_are_zero()
qemu-img: convert: don't use unallocated_blocks_are_zero
iotests: add tests for blockdev-amend
block/qcow2: implement blockdev-amend
block/crypto: implement blockdev-amend
block/core: add generic infrastructure for x-blockdev-amend qmp command
iotests: qemu-img tests for luks key management
block/qcow2: extend qemu-img amend interface with crypto options
block/crypto: implement the encryption key management
block/crypto: rename two functions
block/amend: refactor qcow2 amend options
block/amend: separate amend and create options for qemu-img
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block/amend: refactor qcow2 amend options

Some qcow2 create options can't be used for amend.
Remove them from the qcow2 create options and add generic logic to detect
such options in qemu-img

Signe

block/amend: refactor qcow2 amend options

Some qcow2 create options can't be used for amend.
Remove them from the qcow2 create options and add generic logic to detect
such options in qemu-img

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[mreitz: Dropped some iotests reference output hunks that became
unnecessary thanks to
"iotests: Make _filter_img_create more active"]
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200625125548.870061-12-mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-13' into staging

Block patches:
- zstd compression for qcow2
- Fix use-after-free

# gpg: Signature made Wed 13 May 2020 15:14:

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-13' into staging

Block patches:
- zstd compression for qcow2
- Fix use-after-free

# gpg: Signature made Wed 13 May 2020 15:14:06 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-05-13:
block/block-copy: fix use-after-free of task pointer
iotests: 287: add qcow2 compression type test
qcow2: add zstd cluster compression
qcow2: rework the cluster compression routine
qcow2: introduce compression type feature

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (

qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.

It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.

The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.

Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.

The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- block: AioContext management, part 2
- Avoid recursive block_status call (i.e. lseek() calls) if p

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- block: AioContext management, part 2
- Avoid recursive block_status call (i.e. lseek() calls) if possible
- linux-aio: Drop unused BlockAIOCB submission method
- nvme: add Get/Set Feature Timestamp support
- Fix crash on commit job start with active I/O on base node
- Fix crash in bdrv_drained_end
- Fix integer overflow in qcow2 discard

# gpg: Signature made Tue 04 Jun 2019 16:20:02 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (29 commits)
iotests: Fix duplicated diff output on failure
iotests: test big qcow2 shrink
block/io: bdrv_pdiscard: support int64_t bytes parameter
block/qcow2-refcount: add trace-point to qcow2_process_discards
block: Remove bdrv_set_aio_context()
test-bdrv-drain: Use bdrv_try_set_aio_context()
iotests: Attach new devices to node in non-default iothread
virtio-scsi-test: Test attaching new overlay with iothreads
block: Remove wrong bdrv_set_aio_context() calls
blockdev: Use bdrv_try_set_aio_context() for monitor commands
block: Move node without parents to main AioContext
test-block-iothread: BlockBackend AioContext across root node change
test-block-iothread: Test adding parent to iothread node
block: Adjust AioContexts when attaching nodes
scsi-disk: Use qdev_prop_drive_iothread
block: Add qdev_prop_drive_iothread property type
block: Add BlockBackend.ctx
block: Add Error to blk_set_aio_context()
nbd-server: Call blk_set_allow_aio_context_change()
test-block-iothread: Check filter node in test_propagate_mirror
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Test cancelling a job and closing the VM

This patch adds a test where we cancel a throttled mirror job and
immediately close the VM before it can be cancelled. Doing so will
invoke bdrv_dr

iotests: Test cancelling a job and closing the VM

This patch adds a test where we cancel a throttled mirror job and
immediately close the VM before it can be cancelled. Doing so will
invoke bdrv_drain_all() while the mirror job tries to drain the
throttled node. When bdrv_drain_all_end() tries to lift its drain on
the throttle node, the job will exit and replace the current root node
of the BB drive0 (which is the job's filter node) by the throttle node.
Before the previous patch, this replacement did not increase drive0's
quiesce_counter by a sufficient amount, so when
bdrv_parent_drained_end() (invoked by bdrv_do_drained_end(), invoked by
bdrv_drain_all_end()) tried to end the drain on all of the throttle
node's parents, it decreased drive0's quiesce_counter below 0 -- which
fails an assertion.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

iotests: Test commit job start with concurrent I/O

This tests that concurrent requests are correctly drained before making
graph modifications instead of running into assertions in
bdrv_replace_node

iotests: Test commit job start with concurrent I/O

This tests that concurrent requests are correctly drained before making
graph modifications instead of running into assertions in
bdrv_replace_node().

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>

show more ...