Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- mirror: Fix deadlock
- nbd/server: Fix race in draining the export
- qemu-img snapshot: Fix formatting w

Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- mirror: Fix deadlock
- nbd/server: Fix race in draining the export
- qemu-img snapshot: Fix formatting with large values
- Fix blockdev-snapshot-sync error reporting for no medium
- iotests fixes

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAmX4OG8RHGt3b2xmQHJl
# ZGhhdC5jb20ACgkQfwmycsiPL9YdiQ//faXfGmbK6rBW4AkpwfrRM8SDHvm6hz7L
# 043ujAi3ziSXXoiec2/RK5wZ27nMJkfIrRHXpH41hgQvC6/3a4eIW6KSTaFV1PdG
# JtHCeopmVmgu7TZQ+kt/J6eLUTTLovoO94HgEfmxpr4CGZfx9RJftf2kCKILcYkh
# 9r04zSZLByVd4FJ5ZrqsFulWif5mXoGKdT/YisY3tKiCwFRWQDOoTymvJA012VtO
# MVmID593zwem3O3qtlGiGlK9qodBR4yof66xa/0gaYP98BZgv+LWnwLKha+OzSpX
# bQlxT26LY4JnSQkTdjF0QYnQiH4Q1kveUcNRZrGpA4iZxVDq1aks5DisThDwqoGG
# rhaPOWyJwJsonM1Enzim5Jd60JqvGdpTLjSA5oSyTjw62lAulnYihInERYSAFyyz
# UhQaO7qSog1//RpPEXEsiVkJBq8BE9l5I+L7+l5SCBhNr/UwZAOer/4m4X6d0SKN
# GEPRx0kH1voikzx7gIQs+Oldqvb0sg+zAvOynBxzpd+Ac6s8bFtWe+eSyWYL/ZGr
# Jg9+PL1xir/Uh7KmOnzt/iVBAmfSRpAo1O72xQXvHFYYtIP7hTkPO/vzqF206WMc
# WQFHHjfp5gVcMZ5AYg6txw+Bbtzu8g0AfB054lgnhihuShpf0E923TTDQFdV755s
# NUlrzuGu2fs=
# =+JIK
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 18 Mar 2024 12:49:51 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* tag 'for-upstream' of https://repo.or.cz/qemu/kevin:
iotests: adapt to output change for recently introduced 'detached header' field
tests/qemu-iotests: Restrict tests using "--blockdev file" to the file protocol
tests/qemu-iotests: Fix some tests that use --image-opts for other protocols
tests/qemu-iotests: Restrict tests that use --image-opts to the 'file' protocol
tests/qemu-iotests: Restrict test 156 to the 'file' protocol
tests/qemu-iotests: Restrict test 134 and 158 to the 'file' protocol
tests/qemu-iotests: Restrict test 130 to the 'file' protocol
tests/qemu-iotests: Restrict test 114 to the 'file' protocol
tests/qemu-iotests: Restrict test 066 to the 'file' protocol
tests/qemu-iotests: Fix test 033 for running with non-file protocols
qemu-img: Fix Column Width and Improve Formatting in snapshot list
blockdev: Fix blockdev-snapshot-sync error reporting for no medium
iotests: Add test for reset/AioContext switches with NBD exports
nbd/server: Fix race in draining the export
mirror: Don't call job_pause_point() under graph lock

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

tests/qemu-iotests: Fix some tests that use --image-opts for other protocols

Tests 263, 284 and detect-zeroes-registered-buf use qemu-io
with --image-opts so we have to enforce IMGOPTSSYNTAX=true he

tests/qemu-iotests: Fix some tests that use --image-opts for other protocols

Tests 263, 284 and detect-zeroes-registered-buf use qemu-io
with --image-opts so we have to enforce IMGOPTSSYNTAX=true here
to get $TEST_IMG in shape for other protocols than "file".

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240315111108.153201-9-thuth@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server use

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server uses iothreads
- iotest refactorings in preparation for rewriting ./check to be more
flexible, and preparing for more nbd server reconnect features

# gpg: Signature made Thu 21 Jan 2021 02:28:19 GMT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-01-20:
iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
iotests.py: fix qemu_tool_pipe_and_status()
iotests/264: fix style
iotests: define group in each iotest
iotests/294: add shebang line
iotests: make tests executable
iotests: fix some whitespaces in test output files
iotests/303: use dot slash for qcow2.py running
iotests/277: use dot slash for nbd-fault-injector.py running
nbd/server: Quiesce coroutines on context switch
block: Honor blk_set_aio_context() context requirements
qemu-nbd: Fix a memleak in nbd_client_thread()
qemu-nbd: Fix a memleak in qemu_nbd_client_list()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' grou

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-06' into staging

Block patches for 5.1:
- LUKS keyslot amendment
(+ patches to make the iotests pass on non-Linux systems, an

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-07-06' into staging

Block patches for 5.1:
- LUKS keyslot amendment
(+ patches to make the iotests pass on non-Linux systems, and to keep
the tests passing for qcow v1, and to skip LUKS tests (including
qcow2 LUKS) when the built qemu does not support it)
- Refactoring in the block layer: Drop the basically unnecessary
unallocated_blocks_are_zero field from BlockDriverInfo
- Fix qcow2 preallocation when the image size is not a multiple of the
cluster size
- Fix in block-copy code

# gpg: Signature made Mon 06 Jul 2020 11:02:53 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-07-06: (31 commits)
qed: Simplify backing reads
block: drop unallocated_blocks_are_zero
block/vhdx: drop unallocated_blocks_are_zero
block/file-posix: drop unallocated_blocks_are_zero
block/iscsi: drop unallocated_blocks_are_zero
block/crypto: drop unallocated_blocks_are_zero
block/vpc: return ZERO block-status when appropriate
block/vdi: return ZERO block-status when appropriate
block: inline bdrv_unallocated_blocks_are_zero()
qemu-img: convert: don't use unallocated_blocks_are_zero
iotests: add tests for blockdev-amend
block/qcow2: implement blockdev-amend
block/crypto: implement blockdev-amend
block/core: add generic infrastructure for x-blockdev-amend qmp command
iotests: qemu-img tests for luks key management
block/qcow2: extend qemu-img amend interface with crypto options
block/crypto: implement the encryption key management
block/crypto: rename two functions
block/amend: refactor qcow2 amend options
block/amend: separate amend and create options for qemu-img
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Check whether luks works

Whenever running an iotest for the luks format, we should check whether
luks actually really works.

Tests that try to create luks-encrypted qcow2 images should do

iotests: Check whether luks works

Whenever running an iotest for the luks format, we should check whether
luks actually really works.

Tests that try to create luks-encrypted qcow2 images should do the same.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200625125548.870061-7-mreitz@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-02-20' into staging

Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression typ

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-02-20' into staging

Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression type flag
- optionally flat output for query-named-block-nodes
- some fixes
- pseudo-creation of images on block devices is now done by a generic
block layer function

# gpg: Signature made Thu 20 Feb 2020 16:05:34 GMT
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-02-20:
iotests: Test snapshot -l field separation
block: Fix VM size field width in snapshot dump
iotests: Test convert -n -B to backing-less target
qemu-img: Fix convert -n -B for backing-less targets
iotests: Add test for image creation fallback
iscsi: Drop iscsi_co_create_opts()
file-posix: Drop hdev_co_create_opts()
block: Generic file creation fallback
block/nbd: Fix hang in .bdrv_close()
iotests/279: Fix for non-qcow2 formats
block/backup-top: fix flags handling
block: always fill entire LUKS header space with zeros
qemu-img: Add --target-is-zero to convert
qapi: Allow getting flat output from 'query-named-block-nodes'
iotests/147: Fix drive parameters
iotests: Remove the superfluous 2nd check for the availability of quorum
docs: qcow2: introduce compression type feature
docs: improve qcow2 spec about extending image header

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: always fill entire LUKS header space with zeros

When initializing the LUKS header the size with default encryption
parameters will currently be 2068480 bytes. This is rounded up to
a multiple

block: always fill entire LUKS header space with zeros

When initializing the LUKS header the size with default encryption
parameters will currently be 2068480 bytes. This is rounded up to
a multiple of the cluster size, 2081792, with 64k sectors. If the
end of the header is not the same as the end of the cluster we fill
the extra space with zeros. This was forgetting that not even the
space allocated for the header will be fully initialized, as we
only write key material for the first key slot. The space left
for the other 7 slots is never written to.

An optimization to the ref count checking code:

commit a5fff8d4b4d928311a5005efa12d0991fe3b66f9 (refs/bisect/bad)
Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Date: Wed Feb 27 16:14:30 2019 +0300

qcow2-refcount: avoid eating RAM

made the assumption that every cluster which was allocated would
have at least some data written to it. This was violated by way
the LUKS header is only partially written, with much space simply
reserved for future use.

Depending on the cluster size this problem was masked by the
logic which wrote zeros between the end of the LUKS header and
the end of the cluster.

$ qemu-img create --object secret,id=cluster_encrypt0,data=123456 \
-f qcow2 -o cluster_size=2k,encrypt.iter-time=1,\
encrypt.format=luks,encrypt.key-secret=cluster_encrypt0 \
cluster_size_check.qcow2 100M
Formatting 'cluster_size_check.qcow2', fmt=qcow2 size=104857600
encrypt.format=luks encrypt.key-secret=cluster_encrypt0
encrypt.iter-time=1 cluster_size=2048 lazy_refcounts=off refcount_bits=16

$ qemu-img check --object secret,id=cluster_encrypt0,data=redhat \
'json:{"driver": "qcow2", "encrypt.format": "luks", \
"encrypt.key-secret": "cluster_encrypt0", \
"file.driver": "file", "file.filename": "cluster_size_check.qcow2"}'
ERROR: counting reference for region exceeding the end of the file by one cluster or more: offset 0x2000 size 0x1f9000
Leaked cluster 4 refcount=1 reference=0
...snip...
Leaked cluster 130 refcount=1 reference=0

1 errors were found on the image.
Data may be corrupted, or further writes to the image may corrupt it.

127 leaked clusters were found on the image.
This means waste of disk space, but no harm to data.
Image end offset: 268288

The problem only exists when the disk image is entirely empty. Writing
data to the disk image payload will solve the problem by causing the
end of the file to be extended further.

The change fixes it by ensuring that the entire allocated LUKS header
region is fully initialized with zeros. The qemu-img check will still
fail for any pre-existing disk images created prior to this change,
unless at least 1 byte of the payload is written to.

Fully writing zeros to the entire LUKS header is a good idea regardless
as it ensures that space has been allocated on the host filesystem (or
whatever block storage backend is used).

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200207135520.2669430-1-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...