iotests: add tests for "qemu-img rebase" with compression

The test cases considered so far:

314 (new test suite):

1. Check that compression mode isn't compatible with "-f raw" (raw
format doesn

iotests: add tests for "qemu-img rebase" with compression

The test cases considered so far:

314 (new test suite):

1. Check that compression mode isn't compatible with "-f raw" (raw
format doesn't support compression).
2. Check that rebasing an image onto no backing file preserves the data
and writes the copied clusters actually compressed.
3. Same as 2, but with a raw backing file (i.e. the clusters copied from the
backing are originally uncompressed -- we check they end up compressed
after being merged).
4. Remove a single delta from a backing chain, perform the same checks
as in 2.
5. Check that even when backing and overlay are initially uncompressed,
copied clusters end up compressed when rebase with compression is
performed.

271:

1. Check that when target image has subclusters, rebase with compression
will make an entire cluster containing the written subcluster
compressed.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-9-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

iotests/{024, 271}: add testcases for qemu-img rebase

As the previous commit changes the logic of "qemu-img rebase" (it's using
write alignment now), let's add a couple more test cases which would
e

iotests/{024, 271}: add testcases for qemu-img rebase

As the previous commit changes the logic of "qemu-img rebase" (it's using
write alignment now), let's add a couple more test cases which would
ensure it works correctly. In particular, the following scenarios:

024: add test case for rebase within one backing chain when the overlay
cluster size > backings cluster size;
271: add test case for rebase images that contain subclusters. Check
that no extra allocations are being made.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-7-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

tests/qemu-iotests: Rework the checks and spots using GNU sed

Instead of failing the iotests if GNU sed is not available (or skipping
them completely in the check-block.sh script), it would be bette

tests/qemu-iotests: Rework the checks and spots using GNU sed

Instead of failing the iotests if GNU sed is not available (or skipping
them completely in the check-block.sh script), it would be better to
simply skip the bash-based tests that rely on GNU sed, so that the other
tests could still be run. Thus we now explicitely use "gsed" (either as
direct program or as a wrapper around "sed" if it's the GNU version)
in the spots that rely on the GNU sed behavior. Statements that use the
"-r" parameter of sed have been switched to use "-E" instead, since this
switch is supported by all sed versions on our supported build hosts
(most also support "-r", but macOS' sed only supports "-E"). With all
these changes in place, we then can also remove the sed checks from the
check-block.sh script, so that "make check-block" can now be run on
systems without GNU sed, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220216125454.465041-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qcow2: handle_dependencies(): relax conflict detection

There is no conflict and no dependency if we have parallel writes to
different subclusters of one cluster when the cluster itself is already
al

qcow2: handle_dependencies(): relax conflict detection

There is no conflict and no dependency if we have parallel writes to
different subclusters of one cluster when the cluster itself is already
allocated. So, relax extra dependency.

Measure performance:
First, prepare build/qemu-img-old and build/qemu-img-new images.

cd scripts/simplebench
./img_bench_templater.py

Paste the following to stdin of running script:

qemu_img=../../build/qemu-img-{old|new}
$qemu_img create -f qcow2 -o extended_l2=on /ssd/x.qcow2 1G
$qemu_img bench -c 100000 -d 8 [-s 2K|-s 2K -o 512|-s $((1024*2+512))] \
-w -t none -n /ssd/x.qcow2

The result:

All results are in seconds

------------------ --------- ---------
old new
-s 2K 6.7 ± 15% 6.2 ± 12%
-7%
-s 2K -o 512 13 ± 3% 11 ± 5%
-16%
-s $((1024*2+512)) 9.5 ± 4% 8.4
-12%
------------------ --------- ---------

So small writes are more independent now and that helps to keep deeper
io queue which improves performance.

271 iotest output becomes racy for three allocation in one cluster.
Second and third writes may finish in different order. Second and
third requests don't depend on each other any more. Still they both
depend on first request anyway. Filter out second and third write
offsets to cover both possible outputs.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210824101517.59802-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
[hreitz: s/ an / and /]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/vivier2/tags/linux-user-for-6.0-pull-request' into staging

linux-user pull request 20210119-v3

Remove obsolete F_SHLCK and F_EXLCK translation

Merge remote-tracking branch 'remotes/vivier2/tags/linux-user-for-6.0-pull-request' into staging

linux-user pull request 20210119-v3

Remove obsolete F_SHLCK and F_EXLCK translation
Update sockopt
Add F_ADD_SEALS and F_GET_SEALS

# gpg: Signature made Thu 21 Jan 2021 12:33:04 GMT
# gpg: using RSA key CD2F75DDC8E3A4DC2E4F5173F30C38BD3F2FBE3C
# gpg: issuer "laurent@vivier.eu"
# gpg: Good signature from "Laurent Vivier <lvivier@redhat.com>" [full]
# gpg: aka "Laurent Vivier <laurent@vivier.eu>" [full]
# gpg: aka "Laurent Vivier (Red Hat) <lvivier@redhat.com>" [full]
# Primary key fingerprint: CD2F 75DD C8E3 A4DC 2E4F 5173 F30C 38BD 3F2F BE3C

* remotes/vivier2/tags/linux-user-for-6.0-pull-request:
linux-user: Remove obsolete F_SHLCK and F_EXLCK translation
linux-user: Add IPv6 options to do_print_sockopt()
linux-user: add missing IPv6 get/setsockopt option
linux-user: add missing UDP get/setsockopt option
linux-user: Support F_ADD_SEALS and F_GET_SEALS fcntls

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'remotes/cohuck-gitlab/tags/s390x-20210121' into staging

s390x updates:
- headers update to Linux 5.11-rc2
- fix tcg emulation for some instructions that

Merge remote-tracking branch 'remotes/cohuck-gitlab/tags/s390x-20210121' into staging

s390x updates:
- headers update to Linux 5.11-rc2
- fix tcg emulation for some instructions that are generated by
clang Linux kernel builds
- vfio-ccw: wire up the device unplug notification mechanism
- fix a gcc 11 warning

# gpg: Signature made Thu 21 Jan 2021 12:08:12 GMT
# gpg: using RSA key C3D0D66DC3624FF6A8C018CEDECF6B93C6F02FAF
# gpg: issuer "cohuck@redhat.com"
# gpg: Good signature from "Cornelia Huck <conny@cornelia-huck.de>" [unknown]
# gpg: aka "Cornelia Huck <huckc@linux.vnet.ibm.com>" [full]
# gpg: aka "Cornelia Huck <cornelia.huck@de.ibm.com>" [full]
# gpg: aka "Cornelia Huck <cohuck@kernel.org>" [unknown]
# gpg: aka "Cornelia Huck <cohuck@redhat.com>" [unknown]
# Primary key fingerprint: C3D0 D66D C362 4FF6 A8C0 18CE DECF 6B93 C6F0 2FAF

* remotes/cohuck-gitlab/tags/s390x-20210121:
s390x: Use strpadcpy for copying vm name
vfio-ccw: Connect the device request notifier
Update linux headers to 5.11-rc2
update-linux-headers: Include const.h
s390x/tcg: Ignore register content if b1/b2 is zero when handling EXECUTE
tests/tcg/s390x: Fix EXRL tests
s390x/tcg: Don't ignore content in r0 when not specified via "b" or "x"
s390x/tcg: Fix RISBHG
s390x/tcg: Fix ALGSI

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server uses iothreads
- iotest refactorings in preparation for rewriting ./check to be more
flexible, and preparing for more nbd server reconnect features

# gpg: Signature made Thu 21 Jan 2021 02:28:19 GMT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-01-20:
iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
iotests.py: fix qemu_tool_pipe_and_status()
iotests/264: fix style
iotests: define group in each iotest
iotests/294: add shebang line
iotests: make tests executable
iotests: fix some whitespaces in test output files
iotests/303: use dot slash for qcow2.py running
iotests/277: use dot slash for nbd-fault-injector.py running
nbd/server: Quiesce coroutines on context switch
block: Honor blk_set_aio_context() context requirements
qemu-nbd: Fix a memleak in nbd_client_thread()
qemu-nbd: Fix a memleak in qemu_nbd_client_list()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/bonzini-gitlab/tags/for-upstream' into staging

meson fixes:
* fix PIE default for Windows
* tweak conditions for requiring SDL2 and pixman
*

Merge remote-tracking branch 'remotes/bonzini-gitlab/tags/for-upstream' into staging

meson fixes:
* fix PIE default for Windows
* tweak conditions for requiring SDL2 and pixman
* move NSIS building to Meson
* fix creation of systemtap files
* add --ninja option for Windows builds
* other fixes for stupid bugs and for Windows paths

meson related:
* forward port of dtrace fix for MacOS
* change colorout option to auto

# gpg: Signature made Thu 27 Aug 2020 17:58:21 BST
# gpg: using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg: issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg: aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4 E2F7 7E15 100C CD36 69B1
# Subkey fingerprint: F133 3857 4B66 2389 866C 7682 BFFB D25F 78C7 AE83

* remotes/bonzini-gitlab/tags/for-upstream:
ninjatool: Fixes E$$: in generated Makefile.ninja
meson: set colorout to auto
meson: Mingw64 gcc doesn't recognize system include_type for sdl2
meson: fix relpath failure on Win32
ninjatool: quote dollars in variables
meson: don't require CONFIG_VTE for the GTK UI
meson: cleanup xkbcommon detection
configure: add --ninja option
meson: skip SDL2 detection if --disable-system
meson: move pixman detection to meson
configure: default to PIE disabled on Windows platforms
meson: Fix chardev-baum.so name
meson: Fix meson build with --enable-libdaxctl
scripts/qemu-version.sh: Add missing space before ']'
meson: Don't make object files for dtrace on macOS
meson: Build qemu-nbd on macOS again
trace: fix creation of systemtap files

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'remotes/hdeller/tags/target-hppa-v3-pull-request' into staging

artist out of bounds fixes

# gpg: Signature made Wed 26 Aug 2020 22:09:55 BST
# gpg:

Merge remote-tracking branch 'remotes/hdeller/tags/target-hppa-v3-pull-request' into staging

artist out of bounds fixes

# gpg: Signature made Wed 26 Aug 2020 22:09:55 BST
# gpg: using EDDSA key BCE9123E1AD29F07C049BBDEF712B510A23A0F5F
# gpg: Good signature from "Helge Deller <deller@gmx.de>" [unknown]
# gpg: aka "Helge Deller <deller@kernel.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 4544 8228 2CD9 10DB EF3D 25F8 3E5F 3D04 A7A2 4603
# Subkey fingerprint: BCE9 123E 1AD2 9F07 C049 BBDE F712 B510 A23A 0F5F

* remotes/hdeller/tags/target-hppa-v3-pull-request:
hw/display/artist: Fix invalidation of lines near screen border
hw/display/artist: Fix invalidation of lines in artist_draw_line()
hw/display/artist: Unbreak size mismatch memory accesses
hw/display/artist: Prevent out of VRAM buffer accesses
Revert "hw/display/artist: Avoid drawing line when nothing to display"
hw/display/artist: Refactor artist_rop8() to avoid buffer over-run
hw/display/artist: Check offset in draw_line to avoid buffer over-run
hw/hppa/lasi: Don't abort on invalid IMR value
hw/display/artist.c: fix out of bounds check
hw/hppa: Implement proper SeaBIOS version check
seabios-hppa: Update to SeaBIOS hppa version 1
hw/hppa: Sync hppa_hardware.h file with SeaBIOS sources

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-08-26' into staging

Block patches:
- qcow2 subclusters (extended L2 entries)

# gpg: Signature made Wed 26 Aug 2020 08:37:04 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-08-26: (34 commits)
iotests: Add tests for qcow2 images with extended L2 entries
qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters
qcow2: Allow preallocation and backing files if extended_l2 is set
qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
qcow2: Add prealloc field to QCowL2Meta
qcow2: Add subcluster support to qcow2_measure()
qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()
qcow2: Add subcluster support to handle_alloc_space()
qcow2: Clear the L2 bitmap when allocating a compressed cluster
qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
qcow2: Add subcluster support to check_refcounts_l2()
qcow2: Add subcluster support to discard_in_l2_slice()
qcow2: Add subcluster support to zero_in_l2_slice()
qcow2: Add subcluster support to qcow2_get_host_offset()
qcow2: Add subcluster support to calculate_l2_meta()
qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
qcow2: Add cluster type parameter to qcow2_get_host_offset()
qcow2: Add qcow2_cluster_is_allocated()
qcow2: Add qcow2_get_subcluster_range_type()
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Add tests for qcow2 images with extended L2 entries

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <e6dd0429cafe84ca

iotests: Add tests for qcow2 images with extended L2 entries

Signed-off-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <e6dd0429cafe84ca603179c298a8703bddca2904.1594396418.git.berto@igalia.com>
[mreitz: Use env in shebang line]
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...