Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- Graph locking part 4 (node management)
- qemu-img map: report compressed data blocks
- block-backend: pr

Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- Graph locking part 4 (node management)
- qemu-img map: report compressed data blocks
- block-backend: process I/O in the current AioContext

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAmULHnURHGt3b2xmQHJl
# ZGhhdC5jb20ACgkQfwmycsiPL9aB5hAAqH8To7WIUtg1rj1PY809ck78ghm18PKg
# TNdN7IbrXQghX5foh2VgPwVVl+JaW2CSrJYWQcAO6AbvFduNIi9iKzI6RT0xKXpb
# b8oQXS7zntFzwBv8ohOU5NSVJOgVmNP4h5qJIMmXgB9ZcLFG40zggVH2qQT7guUf
# 9MAc81kI/d5vvSHY0ZjdHjNOgwG4q1j8yytL7OFqWUfB8sXloUCA9lT7w4jIYD8L
# v2StUOLWB01Zts2o8SCNaFxuajs6wUee8b/DM1cyPyLy4KtOdXvLKhq2NlXpLo2i
# aZFr4PtizTVwrQZIJttA9jqM+QCsDOsiSat3BLNNsKUaCWHZB0rOGLCzMCtisyOo
# 4PzuL4UI21ik2zieO1qVM+Thqvw16kHtp6dD9pGk4X4ogGreGYEIxzBl79luR+AV
# NCRizoeFWTHKymS1tSoKrWT9ZNHcLmwemO6Tt1rMYk9jV3T4uY5e1NwxaUavEfsX
# f8dLfQjhNiySOoDknT1OSerBOVdTXURS2ri5H3GZxrxvJ4jOeFkn52C8r3YlZ3Wp
# Cr9LCUJZeXgwY+Q1JQ3D4VLY8aZ83txpw6XKEy0eTEv5wxkBj5LWhXx7hNb5F3lg
# bqaRYijVJn+P82wVxlftIzMfNeVBFHzFE90taPV5grJjr8lgrGBFmD7Puc97kfDX
# oTDBwRxJeew=
# =qTNA
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 20 Sep 2023 12:31:49 EDT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* tag 'for-upstream' of https://repo.or.cz/qemu/kevin: (28 commits)
block: mark aio_poll as non-coroutine
block-backend: process zoned requests in the current AioContext
block-backend: process I/O in the current AioContext
test-bdrv-drain: avoid race with BH in IOThread drain test
block: remove AIOCBInfo->get_aio_context()
qemu-img: map: report compressed data blocks
block: add BDRV_BLOCK_COMPRESSED flag for bdrv_block_status()
block: Mark bdrv_add/del_child() and caller GRAPH_WRLOCK
block: Mark bdrv_unref_child() GRAPH_WRLOCK
block: Mark bdrv_root_unref_child() GRAPH_WRLOCK
block: Take graph rdlock in bdrv_change_aio_context()
block: Take graph rdlock in bdrv_drop_intermediate()
block: Mark bdrv_parent_cb_change_media() GRAPH_RDLOCK
block: Mark bdrv_child_perm() GRAPH_RDLOCK
block: Mark bdrv_get_cumulative_perm() and callers GRAPH_RDLOCK
block: Mark bdrv_parent_perms_conflict() and callers GRAPH_RDLOCK
block: Mark bdrv_attach_child() GRAPH_WRLOCK
block: Call transaction callbacks with lock held
block: Mark bdrv_attach_child_common() GRAPH_WRLOCK
block: Mark bdrv_replace_child_tran() GRAPH_WRLOCK
...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

qemu-img: map: report compressed data blocks

Right now "qemu-img map" reports compressed blocks as containing data
but having no host offset. This is not very informative. Instead,
let's add anoth

qemu-img: map: report compressed data blocks

Right now "qemu-img map" reports compressed blocks as containing data
but having no host offset. This is not very informative. Instead,
let's add another boolean field named "compressed" in case JSON output
mode is specified. This is achieved by utilizing new allocation status
flag BDRV_BLOCK_COMPRESSED for bdrv_block_status().

Also update the expected qemu-iotests outputs to contain the new field.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-ID: <20230907210226.953821-3-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

- qemu-storage-daemon: Add vhost-user-blk help
- block-backend: Fix use-after-free for BDS pointers a

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

- qemu-storage-daemon: Add vhost-user-blk help
- block-backend: Fix use-after-free for BDS pointers after aio_poll()
- qemu-img: Fix sparseness of output image with unaligned ranges
- vvfat: Fix crashes in read-write mode
- Fix device deletion events with -device JSON syntax
- Code cleanups

# gpg: Signature made Fri 14 Jan 2022 13:50:16 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
iotests/testrunner.py: refactor test_field_width
block: drop BLK_PERM_GRAPH_MOD
qemu-img: make is_allocated_sectors() more efficient
iotests: Test qemu-img convert of zeroed data cluster
vvfat: Fix vvfat_write() for writes before the root directory
vvfat: Fix size of temporary qcow file
iotests/308: Fix for CAP_DAC_OVERRIDE
iotests/stream-error-on-reset: New test
block-backend: prevent dangling BDS pointers across aio_poll()
qapi/block: Restrict vhost-user-blk to CONFIG_VHOST_USER_BLK_SERVER
qemu-storage-daemon: Add vhost-user-blk help
docs: Correct 'vhost-user-blk' spelling
softmmu: fix device deletion events with -device JSON syntax
include/sysemu/blockdev.h: remove drive_get_max_devs
include/sysemu/blockdev.h: remove drive_mark_claimed_by_board and inline drive_def
block_int: make bdrv_backing_overridden static

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: make is_allocated_sectors() more efficient

Consider the case when the whole buffer is zero and end is unaligned.

If i <= tail, we return 1 and do one unaligned WRITE, RMW happens.

If i >

qemu-img: make is_allocated_sectors() more efficient

Consider the case when the whole buffer is zero and end is unaligned.

If i <= tail, we return 1 and do one unaligned WRITE, RMW happens.

If i > tail, we do on aligned WRITE_ZERO (or skip if target is zeroed)
and again one unaligned WRITE, RMW happens.

Let's do better: don't fragment the whole-zero buffer and report it as
ZERO: in case of zeroed target we just do nothing and avoid RMW. If
target is not zeroes, one unaligned WRITE_ZERO should not be much worse
than one unaligned WRITE.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20211217164654.1184218-3-vsementsov@virtuozzo.com>
Tested-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

iotests: Test qemu-img convert of zeroed data cluster

This demonstrates what happens when the block status changes in
sub-min_sparse granularity, but all of the parts are zeroed out. The
alignment l

iotests: Test qemu-img convert of zeroed data cluster

This demonstrates what happens when the block status changes in
sub-min_sparse granularity, but all of the parts are zeroed out. The
alignment logic in is_allocated_sectors() prevents that the target image
remains fully sparse as expected, but turns it into a data cluster of
explicit zeros.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20211217164654.1184218-2-vsementsov@virtuozzo.com>
Tested-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-07-09-v2' into staging

nbd patches for 2021-07-09

- enhance 'qemu-img map --output=json' to make it easier to duplicate
backing chain

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-07-09-v2' into staging

nbd patches for 2021-07-09

- enhance 'qemu-img map --output=json' to make it easier to duplicate
backing chain allocation patterns
- fix a race in the 'yank' QMP command in relation to NBD requests

# gpg: Signature made Mon 12 Jul 2021 18:43:37 BST
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-07-09-v2:
nbd: register yank function earlier
qemu-img: Reword 'qemu-img map --output=json' docs
qemu-img: Make unallocated part of backing chain obvious in map
iotests: Improve and rename test 309 to nbd-qemu-allocation

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: Make unallocated part of backing chain obvious in map

The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
spars

qemu-img: Make unallocated part of backing chain obvious in map

The recently-added NBD context qemu:allocation-depth is able to
distinguish between locally-present data (even when that data is
sparse) [shown as depth 1 over NBD], and data that could not be found
anywhere in the backing chain [shown as depth 0]; and the libnbd
project was recently patched to give the human-readable name "absent"
to an allocation-depth of 0. But qemu-img map --output=json predates
that addition, and has the unfortunate behavior that all portions of
the backing chain that resolve without finding a hit in any backing
layer report the same depth as the final backing layer. This makes it
harder to reconstruct a qcow2 backing chain using just 'qemu-img map'
output, especially when using "backing":null to artificially limit a
backing chain, because it is impossible to distinguish between a
QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file)
and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any
backing file), since both types of clusters otherwise show as
"data":false,"zero":true" (but note that we can distinguish a
QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset":
listing).

The task of reconstructing a qcow2 chain was made harder in commit
0da9856851 (nbd: server: Report holes for raw images), because prior
to that point, it was possible to abuse NBD's block status command to
see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED
(showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain
(showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports
more accurate sparseness information over NBD.

An obvious solution is to make 'qemu-img map --output=json' add an
additional "present":false designation to any cluster lacking an
allocation anywhere in the chain, without any change to the "depth"
parameter to avoid breaking existing clients. The iotests have
several examples where this distinction demonstrates the additional
accuracy.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210701190655.2131223-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: fix more iotest fallout]
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Handle `EINVAL` fallocate return value
- qemu-img convert -n: Keep qcow2 v2 target spa

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Handle `EINVAL` fallocate return value
- qemu-img convert -n: Keep qcow2 v2 target sparse

# gpg: Signature made Tue 21 Jul 2020 16:45:28 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
iotests: Test sparseness for qemu-img convert -n
qcow2: Implement v2 zero writes with discard if possible
file-posix: Handle `EINVAL` fallocate return value

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Test sparseness for qemu-img convert -n

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200721135520.72355-3-kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-

iotests: Test sparseness for qemu-img convert -n

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200721135520.72355-3-kwolf@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads

# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
block: Avoid stale pointer dereference in blk_get_aio_context()
qemu-img: Deprecate use of -b without -F
block: Add support to warn on backing file change without format
iotests: Specify explicit backing format where sensible
qcow2: Deprecate use of qemu-img amend to change backing file
block: Error if backing file fails during creation without -u
qcow: Tolerate backing_fmt=
vmdk: Add trivial backing_fmt support
sheepdog: Add trivial backing_fmt support
block: Finish deprecation of 'qemu-img convert -n -o'
qemu-img: Flush stdout before before potential stderr messages
file-posix: Mitigate file fragmentation with extent size hints
iotests/059: Filter out disk size with more standard filter
qemu-img map: Don't limit block status request size
iotests: Simplify _filter_img_create() a bit

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Specify explicit backing format where sensible

There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become

iotests: Specify explicit backing format where sensible

There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.

The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.

iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: Finish deprecation of 'qemu-img convert -n -o'

It's been two releases since we started warning; time to make the
combination an error as promised. There was no iotest coverage, so
add some.

block: Finish deprecation of 'qemu-img convert -n -o'

It's been two releases since we started warning; time to make the
combination an error as promised. There was no iotest coverage, so
add some.

While touching the documentation, tweak another section heading for
consistent style.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-3-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-02-20' into staging

Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression typ

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-02-20' into staging

Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression type flag
- optionally flat output for query-named-block-nodes
- some fixes
- pseudo-creation of images on block devices is now done by a generic
block layer function

# gpg: Signature made Thu 20 Feb 2020 16:05:34 GMT
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-02-20:
iotests: Test snapshot -l field separation
block: Fix VM size field width in snapshot dump
iotests: Test convert -n -B to backing-less target
qemu-img: Fix convert -n -B for backing-less targets
iotests: Add test for image creation fallback
iscsi: Drop iscsi_co_create_opts()
file-posix: Drop hdev_co_create_opts()
block: Generic file creation fallback
block/nbd: Fix hang in .bdrv_close()
iotests/279: Fix for non-qcow2 formats
block/backup-top: fix flags handling
block: always fill entire LUKS header space with zeros
qemu-img: Add --target-is-zero to convert
qapi: Allow getting flat output from 'query-named-block-nodes'
iotests/147: Fix drive parameters
iotests: Remove the superfluous 2nd check for the availability of quorum
docs: qcow2: introduce compression type feature
docs: improve qcow2 spec about extending image header

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Test convert -n -B to backing-less target

This must not crash.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200121155915.98232-3-mreitz@redhat.com>
Reviewed-by: John Snow <j

iotests: Test convert -n -B to backing-less target

This must not crash.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200121155915.98232-3-mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-08-19' into staging

Block patches:
- preallocation=falloc/full support for LUKS
- Various minor fixes

# gpg: Signature made Mon 1

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-08-19' into staging

Block patches:
- preallocation=falloc/full support for LUKS
- Various minor fixes

# gpg: Signature made Mon 19 Aug 2019 16:36:45 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-08-19:
doc: Preallocation does not require writing zeroes
iotests: Fix 141 when run with qed
vpc: Do not return RAW from block_status
vmdk: Make block_status recurse for flat extents
vdi: Make block_status recurse for fixed images
iotests: Full mirror to existing non-zero image
iotests: Test convert -n to pre-filled image
iotests: Convert to preallocated encrypted qcow2
vhdx: Fix .bdrv_has_zero_init()
vdi: Fix .bdrv_has_zero_init()
qcow2: Fix .bdrv_has_zero_init()
block: Use bdrv_has_zero_init_truncate()
block: Implement .bdrv_has_zero_init_truncate()
block: Add bdrv_has_zero_init_truncate()
mirror: Fix bdrv_has_zero_init() use
qemu-img: Fix bdrv_has_zero_init() use in convert
LUKS: support preallocation

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Test convert -n to pre-filled image

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-11-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signe

iotests: Test convert -n to pre-filled image

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190724171239.8764-11-mreitz@redhat.com
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Check correct file type (regular file for 'file',
character or block device for 'hos

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Check correct file type (regular file for 'file',
character or block device for 'host_device'/'host_cdrom')
- scsi-disk: Block Device Characteristics emulation fix
- qemu-img: Consider required alignment for sparse area detection
- Documentation and test improvements

# gpg: Signature made Thu 12 Jul 2018 17:29:17 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
qemu-img: align result of is_allocated_sectors
scsi-disk: Block Device Characteristics emulation fix
iotests: add test 226 for file driver types
file-posix: specify expected filetypes
qemu-img: Document copy offloading implications with -S and -c
iotests: nbd: Stop qemu-nbd before remaking image
iotests: 153: Fix dead code

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-img: align result of is_allocated_sectors

We currently don't enforce that the sparse segments we detect during convert are
aligned. This leads to unnecessary and costly read-modify-write cycles

qemu-img: align result of is_allocated_sectors

We currently don't enforce that the sparse segments we detect during convert are
aligned. This leads to unnecessary and costly read-modify-write cycles either
internally in Qemu or in the background on the storage device as nearly all
modern filesystems or hardware have a 4k alignment internally.

This patch modifies is_allocated_sectors so that its *pnum result will always
end at an alignment boundary. This way all requests will end at an alignment
boundary. The start of all requests will also be aligned as long as the results
of get_block_status do not lead to an unaligned offset.

The number of RMW cycles when converting an example image [1] to a raw device that
has 4k sector size is about 4600 4k read requests to perform a total of about 15000
write requests. With this path the additional 4600 read requests are eliminated while
the number of total write requests stays constant.

[1] https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.vmdk

Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2018-06-11' into staging

Block patches:
- Various bug fixes
- Removal of qemu-img convert's deprecated -s option
- qemu-io now exits wi

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2018-06-11' into staging

Block patches:
- Various bug fixes
- Removal of qemu-img convert's deprecated -s option
- qemu-io now exits with an error when a command failed

# gpg: Signature made Mon 11 Jun 2018 15:23:42 BST
# gpg: using RSA key F407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2018-06-11: (29 commits)
iotests: Add case for a corrupted inactive image
qcow2: Do not mark inactive images corrupt
block: Make bdrv_is_writable() public
throttle: Fix crash on reopen
block/qcow2-bitmap: fix free_bitmap_clusters
qemu-img: Remove deprecated -s snapshot_id_or_name option
iotests: Fix 219's timing
iotests: improve pause_job
iotests: Test post-backing convert target behavior
qemu-img: Special post-backing convert handling
iotests: Add test for rebasing with relative paths
qemu-img: Resolve relative backing paths in rebase
iotests: Let 216 make use of qemu-io's exit code
iotests.py: Add qemu_io_silent
qemu-io: Exit with error when a command failed
qemu-io: Let command functions return error code
qemu-io: Drop command functions' return values
iotests: Repairing error during snapshot deletion
qcow2: Repair OFLAG_COPIED when fixing leaks
iotests: Rework 113
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Test post-backing convert target behavior

This adds a test case to 122 for what happens when you convert to a
target with a backing file that is shorter than the target, and the
image forma

iotests: Test post-backing convert target behavior

This adds a test case to 122 for what happens when you convert to a
target with a backing file that is shorter than the target, and the
image format does not support efficient zero writes (as is the case with
qcow2 v2).

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20180501165750.19242-3-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Switch AIO/callback based block drivers to a byte-based interface
- Block jobs: Expose error strin

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Switch AIO/callback based block drivers to a byte-based interface
- Block jobs: Expose error string via query-block-jobs
- Block job cleanups and fixes
- hmp: Allow using a qdev id in block_set_io_throttle

# gpg: Signature made Tue 15 May 2018 16:33:10 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (37 commits)
iotests: Add test for -U/force-share conflicts
qemu-img: Use only string options in img_open_opts
qemu-io: Use purely string blockdev options
block: Document BDRV_REQ_WRITE_UNCHANGED support
qemu-img: Check post-truncation size
iotests: Add test for COR across nodes
iotests: Copy 197 for COR filter driver
iotests: Clean up wrap image in 197
block: Support BDRV_REQ_WRITE_UNCHANGED in filters
block/quorum: Support BDRV_REQ_WRITE_UNCHANGED
block: Set BDRV_REQ_WRITE_UNCHANGED for COR writes
block: Add BDRV_REQ_WRITE_UNCHANGED flag
block: BLK_PERM_WRITE includes ..._UNCHANGED
block: Add COR filter driver
iotests: Skip 181 and 201 without userfaultfd
iotests: Add failure matching to common.qemu
docs: Document the new default sizes of the qcow2 caches
qcow2: Give the refcount cache the minimum possible size by default
specs/qcow2: Clarify that compressed clusters have the COPIED bit reset
Fix error message about compressed clusters with OFLAG_COPIED
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'mreitz/tags/pull-block-2018-05-15' into queue-block

- Copy-on-read block driver
- The qcow2 default refcount cache size has been decreased
- Various bug fixes

# gpg: S

Merge remote-tracking branch 'mreitz/tags/pull-block-2018-05-15' into queue-block

- Copy-on-read block driver
- The qcow2 default refcount cache size has been decreased
- Various bug fixes

# gpg: Signature made Tue May 15 16:18:25 2018 CEST
# gpg: using RSA key F407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* mreitz/tags/pull-block-2018-05-15: (21 commits)
iotests: Add test for -U/force-share conflicts
qemu-img: Use only string options in img_open_opts
qemu-io: Use purely string blockdev options
block: Document BDRV_REQ_WRITE_UNCHANGED support
qemu-img: Check post-truncation size
iotests: Add test for COR across nodes
iotests: Copy 197 for COR filter driver
iotests: Clean up wrap image in 197
block: Support BDRV_REQ_WRITE_UNCHANGED in filters
block/quorum: Support BDRV_REQ_WRITE_UNCHANGED
block: Set BDRV_REQ_WRITE_UNCHANGED for COR writes
block: Add BDRV_REQ_WRITE_UNCHANGED flag
block: BLK_PERM_WRITE includes ..._UNCHANGED
block: Add COR filter driver
iotests: Skip 181 and 201 without userfaultfd
iotests: Add failure matching to common.qemu
docs: Document the new default sizes of the qcow2 caches
qcow2: Give the refcount cache the minimum possible size by default
specs/qcow2: Clarify that compressed clusters have the COPIED bit reset
Fix error message about compressed clusters with OFLAG_COPIED
...

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

iotests: Split 214 off of 122

Commit abd3622cc03cf41ed542126a540385f30a4c0175 added a case to 122
regarding how the qcow2 driver handles an incorrect compressed data
length value. This does not rea

iotests: Split 214 off of 122

Commit abd3622cc03cf41ed542126a540385f30a4c0175 added a case to 122
regarding how the qcow2 driver handles an incorrect compressed data
length value. This does not really fit into 122, as that file is
supposed to contain qemu-img convert test cases, which this case is not.
So this patch splits it off into its own file; maybe we will even get
more qcow2-only compression tests in the future.

Also, that test case does not work with refcount_bits=1, so mark that
option as unsupported.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20180406164108.26118-1-mreitz@redhat.com
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

# gpg: Signature made Tue 03 Apr 2018 16:48:53 BST
# gpg: using RSA key 7F09B272C88F2F

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches

# gpg: Signature made Tue 03 Apr 2018 16:48:53 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
iotests: Test abnormally large size in compressed cluster descriptor
qemu-iotests: Use ppc64 qemu_arch on ppc64le host
iotests: Test preallocated truncate of 2G image
block/file-posix: Fix fully preallocated truncate
iotests: fix 208 for luks format
iotests: Update 186 after commit ac64273c66ab136c44043259162
iotests: Update 051 and 186 after commit 1454509726719e0933c
block: handle invalid lseek returns gracefully
gluster: Fix blockdev-add with server.N.type=unix

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'mreitz/tags/pull-block-2018-04-03' into queue-block

A fix for preallocated truncation, a new iotest, and a fix to make the iotests work more comfortably on ppc64

# gpg

Merge remote-tracking branch 'mreitz/tags/pull-block-2018-04-03' into queue-block

A fix for preallocated truncation, a new iotest, and a fix to make the iotests work more comfortably on ppc64

# gpg: Signature made Tue Apr 3 17:40:57 2018 CEST
# gpg: using RSA key F407DB0061D5CF40
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* mreitz/tags/pull-block-2018-04-03:
iotests: Test abnormally large size in compressed cluster descriptor
qemu-iotests: Use ppc64 qemu_arch on ppc64le host
iotests: Test preallocated truncate of 2G image
block/file-posix: Fix fully preallocated truncate

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...