monitor: only run coroutine commands in qemu_aio_context

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not
polled during nested event loops. The coroutine currently reschedule

monitor: only run coroutine commands in qemu_aio_context

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not
polled during nested event loops. The coroutine currently reschedules
itself in the main loop's qemu_aio_context AioContext, which is polled
during nested event loops. One known problem is that QMP device-add
calls drain_call_rcu(), which temporarily drops the BQL, leading to all
sorts of havoc like other vCPU threads re-entering device emulation code
while another vCPU thread is waiting in device emulation code with
aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the
iohandler AioContext. This avoids trouble with nested event loops. His
original idea was to move coroutine rescheduling to
monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch()
because we don't know if the QMP handler needs to run in coroutine
context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have
been nicer since it's associated with the monitor implementation and not
as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of
QMP events vs QMP responses has changed.

Solves Issue #1933.

Cc: qemu-stable@nongnu.org
Fixes: 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2215192
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2214985
Buglink: https://issues.redhat.com/browse/RHEL-17369
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20240118144823.1497953-4-stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
(cherry picked from commit effd60c878176bcaf97fa7ce2b12d04bb8ead6f7)
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>

show more ...

Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- virtio-blk: Multiqueue fixes and cleanups
- blklogwrites: Fixes for write_zeroes and superblock update r

Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- virtio-blk: Multiqueue fixes and cleanups
- blklogwrites: Fixes for write_zeroes and superblock update races
- commit/stream: Allow users to request only format driver names in
backing file format
- monitor: only run coroutine commands in qemu_aio_context
- Some iotest fixes

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAmWzpOwRHGt3b2xmQHJl
# ZGhhdC5jb20ACgkQfwmycsiPL9ZNzg//W1+C7HxLft4Jc4O1BcOoOLlGCg4Esupt
# z0/XLZ9+xVQUtjQ82pFzf9XaWQs8CuNT3FBUKi+ngdwZ0JBThIv0aGiMZBcAeQjD
# qshPFgDM1lGL4ICIaT73/qfUzQgO3oruZj9F+ShBBzoasNWVoRzqqVDR3pinLwTp
# D4TU+3A6LkdhlYGT60SYfRq/UKNmCA1s2wysdjqXxS6KOEURNF2VBnz0Nu76qrVb
# 3P/a55GPiJIn+VVsdQ0J4vyyzn23m7I7WZOJ7Sjm1EfSJ6SvcDbhWsZTUonaV2rU
# qZ3WI/jggqxXRV8F2AaA4suS/Cc8RkX2KfcN8fB6wDC2eI5USSatjh6xfw5xH9Ll
# NRKUO4vFFR3Lf8wN9apg0Bwxqi0GOm9kvBJT5QqjQ16R1dvqBLqbZqcx6ZXqWFXe
# /Iy243Tz19mWTFVUj0EgCKQpNz9F4SyXxV83HtSR1lJ5mhthnLxkvUOe7jsFPE4d
# 1Z3uBNWnx2mKFkhlwocMTKayYqxPuKQ+YjqrRoplLW1GZoBeoalKRGf8/RHa6kQx
# gh4cguihlb71AH1AO1QuYpiZt9G4RJR2RZlIoCPJY5TaKJedcxMVn8H+8/F0PnQd
# gPysZf7hTU1xCUV6TClDd+f2fuvqZYwXdwHJ9iiohNkbFq4HFQUp4nk4/eEPGSe/
# uv8oE813E30=
# =KQJl
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 26 Jan 2024 12:26:20 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* tag 'for-upstream' of https://repo.or.cz/qemu/kevin:
iotests/277: Use iotests.sock_dir for socket creation
iotests/iothreads-stream: Use the right TimeoutError
tests/unit: Bump test-replication timeout to 60 seconds
iotests/264: Use iotests.sock_dir for socket creation
block/blklogwrites: Protect mutable driver state with a mutex.
virtio-blk: always set ioeventfd during startup
virtio-blk: tolerate failure to set BlockBackend AioContext
virtio-blk: restart s->rq reqs in vq AioContexts
virtio-blk: rename dataplane to ioeventfd
virtio-blk: rename dataplane create/destroy functions
virtio-blk: move dataplane code into virtio-blk.c
monitor: only run coroutine commands in qemu_aio_context
iotests: port 141 to Python for reliable QMP testing
iotests: add filter_qmp_generated_node_ids()
stream: Allow users to request only format driver names in backing file format
commit: Allow users to request only format driver names in backing file format
string-output-visitor: Fix (pseudo) struct handling
block/blklogwrites: Fix a bug when logging "write zeroes" operations.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

monitor: only run coroutine commands in qemu_aio_context

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not
polled during nested event loops. The coroutine currently reschedule

monitor: only run coroutine commands in qemu_aio_context

monitor_qmp_dispatcher_co() runs in the iohandler AioContext that is not
polled during nested event loops. The coroutine currently reschedules
itself in the main loop's qemu_aio_context AioContext, which is polled
during nested event loops. One known problem is that QMP device-add
calls drain_call_rcu(), which temporarily drops the BQL, leading to all
sorts of havoc like other vCPU threads re-entering device emulation code
while another vCPU thread is waiting in device emulation code with
aio_poll().

Paolo Bonzini suggested running non-coroutine QMP handlers in the
iohandler AioContext. This avoids trouble with nested event loops. His
original idea was to move coroutine rescheduling to
monitor_qmp_dispatch(), but I resorted to moving it to qmp_dispatch()
because we don't know if the QMP handler needs to run in coroutine
context in monitor_qmp_dispatch(). monitor_qmp_dispatch() would have
been nicer since it's associated with the monitor implementation and not
as general as qmp_dispatch(), which is also used by qemu-ga.

A number of qemu-iotests need updated .out files because the order of
QMP events vs QMP responses has changed.

Solves Issue #1933.

Cc: qemu-stable@nongnu.org
Fixes: 7bed89958bfbf40df9ca681cefbdca63abdde39d ("device_core: use drain_call_rcu in in qmp_device_add")
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2215192
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2214985
Buglink: https://issues.redhat.com/browse/RHEL-17369
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20240118144823.1497953-4-stefanha@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration bl

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration block node permissions
- iotests fixes (051 and mirror-ready-cancel-error)
- Remove an outdated comment

# gpg: Signature made Tue 01 Feb 2022 13:34:54 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* remotes/hreitz-gitlab/tags/pull-block-2022-02-01: (24 commits)
block.h: remove outdated comment
iotests/migration-permissions: New test
block-backend: Retain permissions after migration
iotests: declare lack of support for compresion_type in IMGOPTS
iotest 214: explicit compression type
iotests 60: more accurate set dirty bit in qcow2 header
iotests: bash tests: filter compression type
iotest 39: use _qcow2_dump_header
iotests: massive use _qcow2_dump_header
iotests/common.rc: introduce _qcow2_dump_header helper
qcow2: simple case support for downgrading of qcow2 images with zstd
iotest 302: use img_info_log() helper
iotests.py: filter compression type out
iotests.py: filter out successful output of qemu-img create
iotest 065: explicit compression type
iotest 303: explicit compression type
iotests.py: rewrite default luks support in qemu_img
iotests: drop qemu_img_verbose() helper
iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
iotests: specify some unsupported_imgopts for python iotests
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: bash tests: filter compression type

We want iotests pass with both the default zlib compression and with
IMGOPTS='compression_type=zstd'.

Actually the only test that is interested in real

iotests: bash tests: filter compression type

We want iotests pass with both the default zlib compression and with
IMGOPTS='compression_type=zstd'.

Actually the only test that is interested in real compression type in
test output is 287 (test for qcow2 compression type), so implement
specific option for it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20211223160144.1097696-17-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
#

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-08-26: (34 commits)
iotests: Add tests for qcow2 images with extended L2 entries
qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters
qcow2: Allow preallocation and backing files if extended_l2 is set
qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
qcow2: Add prealloc field to QCowL2Meta
qcow2: Add subcluster support to qcow2_measure()
qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
qcow2: Add subcluster support to handle_alloc_space()
qcow2: Clear the L2 bitmap when allocating a compressed cluster
qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
qcow2: Add subcluster support to check_refcounts_l2()
qcow2: Add subcluster support to discard_in_l2_slice()
qcow2: Add subcluster support to zero_in_l2_slice()
qcow2: Add subcluster support to qcow2_get_host_offset()
qcow2: Add subcluster support to calculate_l2_meta()
qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
qcow2: Add cluster type parameter to qcow2_get_host_offset()
qcow2: Add qcow2_cluster_is_allocated()
qcow2: Add qcow2_get_subcluster_range_type()
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with th

qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

Now that the implementation of subclusters is complete we can finally
add the necessary options to create and read images with this feature,
which we call "extended L2 entries".

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <6476caaa73216bd05b7bb2d504a20415e1665176.1594396418.git.berto@igalia.com>
[mreitz: %s/5\.1/5.2/; fixed 302's and 303's reference output]
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

qcow2: Add subcluster support to check_refcounts_l2()

The offset field of an uncompressed cluster's L2 entry must be aligned
to the cluster size, otherwise it is invalid. If the cluster has no
data

qcow2: Add subcluster support to check_refcounts_l2()

The offset field of an uncompressed cluster's L2 entry must be aligned
to the cluster size, otherwise it is invalid. If the cluster has no
data then it means that the offset points to a preallocation, so we
can clear the offset field without affecting the guest-visible data.
This is what 'qemu-img check' does when run in repair mode.

On traditional qcow2 images this can only happen when QCOW_OFLAG_ZERO
is set, and repairing such entries turns the clusters from ZERO_ALLOC
into ZERO_PLAIN.

Extended L2 entries have no ZERO_ALLOC clusters and no QCOW_OFLAG_ZERO
but the idea is the same: if none of the subclusters are allocated
then we can clear the offset field and leave the bitmap untouched.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <9f4ed1d0a34b0a545b032c31ecd8c14734065342.1594396418.git.berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads

# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
block: Avoid stale pointer dereference in blk_get_aio_context()
qemu-img: Deprecate use of -b without -F
block: Add support to warn on backing file change without format
iotests: Specify explicit backing format where sensible
qcow2: Deprecate use of qemu-img amend to change backing file
block: Error if backing file fails during creation without -u
qcow: Tolerate backing_fmt=
vmdk: Add trivial backing_fmt support
sheepdog: Add trivial backing_fmt support
block: Finish deprecation of 'qemu-img convert -n -o'
qemu-img: Flush stdout before before potential stderr messages
file-posix: Mitigate file fragmentation with extent size hints
iotests/059: Filter out disk size with more standard filter
qemu-img map: Don't limit block status request size
iotests: Simplify _filter_img_create() a bit

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Specify explicit backing format where sensible

There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become

iotests: Specify explicit backing format where sensible

There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.

The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.

iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-13' into staging

Block patches:
- zstd compression for qcow2
- Fix use-after-free

# gpg: Signature made Wed 13 May 2020 15:14:

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-13' into staging

Block patches:
- zstd compression for qcow2
- Fix use-after-free

# gpg: Signature made Wed 13 May 2020 15:14:06 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-05-13:
block/block-copy: fix use-after-free of task pointer
iotests: 287: add qcow2 compression type test
qcow2: add zstd cluster compression
qcow2: rework the cluster compression routine
qcow2: introduce compression type feature

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (

qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.

It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.

The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.

Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.

The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-04-07' into staging

Block patches for 5.0-rc2:
- Fix double QLIST_REMOVE() and potential request object leak in
xen-block
- Prev

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-04-07' into staging

Block patches for 5.0-rc2:
- Fix double QLIST_REMOVE() and potential request object leak in
xen-block
- Prevent a potential assertion failure in qcow2's code for compressed
clusters by rejecting invalid (unaligned) requests with -EIO
- Prevent discards on qcow2 v2 images from making backing data reappear
- Make qemu-img convert report I/O error locations by byte offsets
consistently
- Fix for potential I/O test errors (accidental globbing due to missing
quotes)

# gpg: Signature made Tue 07 Apr 2020 13:30:01 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-04-07:
xen-block: Fix double qlist remove and request leak
iotests/common.pattern: Quote echos
qcow2: Check request size in qcow2_co_pwritev_compressed_part()
qemu-img: Report convert errors by bytes, not sectors
qcow2: Forbid discard in qcow2 v2 images with backing files

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: Forbid discard in qcow2 v2 images with backing files

A discard request deallocates the selected clusters so they read back
as zeroes. This is done by clearing the cluster offset field and
set

qcow2: Forbid discard in qcow2 v2 images with backing files

A discard request deallocates the selected clusters so they read back
as zeroes. This is done by clearing the cluster offset field and
setting QCOW_OFLAG_ZERO in the L2 entry.

This flag is however only supported when qcow_version >= 3. In older
images the cluster is simply deallocated, exposing any possible stale
data from the backing file.

Since discard is an advisory operation it's safer to simply forbid it
in this scenario.

Note that we are adding this check to qcow2_co_pdiscard() and not to
qcow2_cluster_discard() or discard_in_l2_slice() because the last
two are also used by qcow2_snapshot_create() to discard the clusters
used by the VM state. In this case there's no risk of exposing stale
data to the guest and we really want that the clusters are always
discarded.

Signed-off-by: Alberto Garcia <berto@igalia.com>
Message-Id: <20200331114345.29993-1-berto@igalia.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-01-06' into staging

Block patches:
- Minor fixes and tests from the freeze period (too minor to be included
in 4.2)
- Allow many

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-01-06' into staging

Block patches:
- Minor fixes and tests from the freeze period (too minor to be included
in 4.2)
- Allow many bash iotests to test qcow2's external data file feature
- Add compress filter driver
- Fix Python iotests after 6f6e1698a6
- Fix for the backup job

# gpg: Signature made Mon 06 Jan 2020 14:33:06 GMT
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-01-06: (34 commits)
backup-top: Begin drain earlier
tests/qemu-iotests: Update tests to recent desugarized -accel option
tests/qemu-iotests: add case to write compressed data of multiple clusters
qcow2: Allow writing compressed data of multiple clusters
block: introduce compress filter driver
iotests: Allow check -o data_file
iotests: Disable data_file where it cannot be used
iotests: Make 198 work with data_file
iotests: Make 137 work with data_file
iotests: Make 110 work with data_file
iotests: Make 091 work with data_file
iotests: Avoid cp/mv of test images
iotests: Use _rm_test_img for deleting test images
iotests: Avoid qemu-img create
iotests: Drop IMGOPTS use in 267
iotests: Replace IMGOPTS='' by --no-opts
iotests: Replace IMGOPTS= by -o
iotests: Inject space into -ocompat=0.10 in 051
iotests: Add -o and --no-opts to _make_test_img
iotests: Let _make_test_img parse its parameters
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests/qcow2.py: Split feature fields into bits

Print the feature fields as a set of bits so that filtering is easier.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Maxim Levitsky <mle

iotests/qcow2.py: Split feature fields into bits

Print the feature fields as a set of bits so that filtering is easier.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-id: 20191107163708.833192-4-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-05-28' into staging

Block patches:
- qcow2: Use threads for encrypted I/O
- qemu-img rebase: Optimizations
- backup job: Allow any

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-05-28' into staging

Block patches:
- qcow2: Use threads for encrypted I/O
- qemu-img rebase: Optimizations
- backup job: Allow any source node, and some refactoring
- Some general simplifications in the block layer

# gpg: Signature made Tue 28 May 2019 20:26:56 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-05-28: (21 commits)
blockdev: loosen restrictions on drive-backup source node
qcow2-bitmap: initialize bitmap directory alignment
qcow2: skip writing zero buffers to empty COW areas
qemu-img: rebase: Reuse in-chain BlockDriverState
qemu-img: rebase: Reduce reads on in-chain rebase
qemu-img: rebase: Reuse parent BlockDriverState
block: Make bdrv_root_attach_child() unref child_bs on failure
block: Use bdrv_unref_child() for all children in bdrv_close()
block/backup: refactor: split out backup_calculate_cluster_size
block/backup: unify different modes code path
block/backup: refactor and tolerate unallocated cluster skipping
block/backup: move to copy_bitmap with granularity
block/backup: simplify backup_incremental_init_copy_bitmap
qcow2: do encryption in threads
qcow2: bdrv_co_pwritev: move encryption code out of the lock
qcow2: qcow2_co_preadv: improve locking
qcow2-threads: split out generic path
qcow2-threads: qcow2_co_do_compress: protect queuing by mutex
qcow2-threads: use thread_pool_submit_co
qcow2: add separate file for threaded data processing functions
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2: skip writing zero buffers to empty COW areas

If COW areas of the newly allocated clusters are zeroes on the backing
image, efficient bdrv_write_zeroes(flags=BDRV_REQ_NO_FALLBACK) can be
used

qcow2: skip writing zero buffers to empty COW areas

If COW areas of the newly allocated clusters are zeroes on the backing
image, efficient bdrv_write_zeroes(flags=BDRV_REQ_NO_FALLBACK) can be
used on the whole cluster instead of writing explicit zero buffers later
in perform_cow().

iotest 060:
write to the discarded cluster does not trigger COW anymore.
Use a backing image instead.

Signed-off-by: Anton Nefedov <anton.nefedov@virtuozzo.com>
Message-id: 20190516142749.81019-2-anton.nefedov@virtuozzo.com
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- iotests: Fix output of qemu-io related tests
- Don't ignore bdrv_set_aio_context() for nodes with bs->drv = NUL
- vmdk: Set vmdk parent backing_format to vmdk
- qcow2: Preallocation fixes (especially for external data files)
- Add linear-buffer-based APIs (as wrappers around qiov-based ones)
- Various code cleanups and small corner case fixes

# gpg: Signature made Tue 30 Apr 2019 16:35:09 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (27 commits)
block/qed: add missed coroutine_fn markers
iotests: Check that images are in read-only mode after block-commit
commit: Make base read-only if there is an early failure
qemu-img: use buffer-based io
block/stream: use buffer-based io
block/commit: use buffer-based io
block/backup: use buffer-based io
block/parallels: use buffer-based io
block/qed: use buffer-based io
block/qcow: use buffer-based io
block/qcow2: use buffer-based io
block: introduce byte-based io helpers
qcow2: Fix error handling in the compression code
qcow2: Fix qcow2_make_empty() with external data file
qemu-img: Make create hint at protocol options
iotests: Perform the correct test in 082
qcow2: Fix full preallocation with external data file
qcow2: Add errp to preallocate_co()
qcow2: Avoid COW during metadata preallocation
qemu-img: Saner printing of large file sizes
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nb

qemu-img: Saner printing of large file sizes

Disk sizes close to INT64_MAX cause overflow, for some pretty
ridiculous output:

$ ./nbdkit -U - memory size=$((2**63 - 512)) --run 'qemu-img info $nbd'
image: nbd+unix://?socket=/tmp/nbdkitHSAzNz/socket
file format: raw
virtual size: -8388607T (9223372036854775296 bytes)
disk size: unavailable

But there's no reason to have two separate implementations of integer
to human-readable abbreviation, where one has overflow and stops at
'T', while the other avoids overflow and goes all the way to 'E'. With
this patch, the output now claims 8EiB instead of -8388607T, which
really is the correct rounding of largest file size supported by qemu
(we could go 511 bytes larger if we used byte-accurate sizing instead
of rounding up to the next sector boundary, but that wouldn't change
the human-readable result).

Quite a few iotests need updates to expected output to match.

Reported-by: Richard W.M. Jones <rjones@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Tested-by: Richard W.M. Jones <rjones@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Tested-by: Max Reitz <mreitz@redhat.com>

show more ...

tests/qemu-iotests: Fix output of qemu-io related tests

One of the recent commits changed the way qemu-io prints out its
errors and warnings - they are now prefixed with the program name.
We've got

tests/qemu-iotests: Fix output of qemu-io related tests

One of the recent commits changed the way qemu-io prints out its
errors and warnings - they are now prefixed with the program name.
We've got to adapt the iotests accordingly to prevent that they
are failing.

Fixes: 99e98d7c9fc1a1639fad ("qemu-io: Use error_[gs]et_progname()")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2018-12-18' into staging

QAPI patches for 2018-12-18

# gpg: Signature made Tue 18 Dec 2018 07:20:11 GMT
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2018-12-18' into staging

QAPI patches for 2018-12-18

# gpg: Signature made Tue 18 Dec 2018 07:20:11 GMT
# gpg: using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2018-12-18:
qapi: fix flat union on uncovered branches conditionals
qmp hmp: Make system_wakeup check wake-up support and run state
qga: update guest-suspend-ram and guest-suspend-hybrid descriptions
qmp: query-current-machine with wakeup-suspend-support
qmp: Split ShutdownCause host-qmp into quit and system-reset
qmp: Add reason to SHUTDOWN and RESET events
qapi: Turn ShutdownCause into QAPI enum

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qmp: Split ShutdownCause host-qmp into quit and system-reset

It is interesting to know whether the shutdown cause was 'quit' or
'reset', especially when using "--no-reboot". In that case, a manageme

qmp: Split ShutdownCause host-qmp into quit and system-reset

It is interesting to know whether the shutdown cause was 'quit' or
'reset', especially when using "--no-reboot". In that case, a management
layer can now determine if the guest wanted a reboot or shutdown, and
can act accordingly.

Changes the output of the reason in the iotests from 'host-qmp' to
'host-qmp-quit'. This does not break compatibility because
the field was introduced in the same version.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Message-Id: <20181205110131.23049-4-d.csapak@proxmox.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qmp: Add reason to SHUTDOWN and RESET events

This makes it possible to determine what the exact reason was for
a RESET or a SHUTDOWN. A management layer might need the specific reason
of those event

qmp: Add reason to SHUTDOWN and RESET events

This makes it possible to determine what the exact reason was for
a RESET or a SHUTDOWN. A management layer might need the specific reason
of those events to determine which cleanups or other actions it needs to do.

This patch also updates the iotests to the new expected output that includes
the reason.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Message-Id: <20181205110131.23049-3-d.csapak@proxmox.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2018-06-30' into staging

Monitor patches for 2018-06-30

# gpg: Signature made Sat 30 Jun 2018 17:22:12 BST
# gpg: using

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2018-06-30' into staging

Monitor patches for 2018-06-30

# gpg: Signature made Sat 30 Jun 2018 17:22:12 BST
# gpg: using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-monitor-2018-06-30:
docs: mention shared state protect for OOB
tests: iotests: drop some stderr line
monitor: flush qmp responses when CLOSED
monitor: rename *_pop_one to *_pop_any
chardev: comment details for CLOSED event

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...