Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server use

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server uses iothreads
- iotest refactorings in preparation for rewriting ./check to be more
flexible, and preparing for more nbd server reconnect features

# gpg: Signature made Thu 21 Jan 2021 02:28:19 GMT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-01-20:
iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
iotests.py: fix qemu_tool_pipe_and_status()
iotests/264: fix style
iotests: define group in each iotest
iotests/294: add shebang line
iotests: make tests executable
iotests: fix some whitespaces in test output files
iotests/303: use dot slash for qcow2.py running
iotests/277: use dot slash for nbd-fault-injector.py running
nbd/server: Quiesce coroutines on context switch
block: Honor blk_set_aio_context() context requirements
qemu-nbd: Fix a memleak in nbd_client_thread()
qemu-nbd: Fix a memleak in qemu_nbd_client_list()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' grou

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Support for FUSE exports
- Fix deadlock in bdrv_co_yield_to_drain()
- Use lock guard macros
- Some

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Support for FUSE exports
- Fix deadlock in bdrv_co_yield_to_drain()
- Use lock guard macros
- Some preparational patches for 64 bit block layer
- file-posix: Fix request extension to INT64_MAX in raw_do_pwrite_zeroes()

# gpg: Signature made Fri 11 Dec 2020 17:06:19 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (34 commits)
block: Fix deadlock in bdrv_co_yield_to_drain()
block: Fix locking in qmp_block_resize()
block: Simplify qmp_block_resize() error paths
block: introduce BDRV_MAX_LENGTH
block/io: bdrv_check_byte_request(): drop bdrv_is_inserted()
block/io: bdrv_refresh_limits(): use ERRP_GUARD
block/file-posix: fix workaround in raw_do_pwrite_zeroes()
can-host: Fix crash when 'canbus' property is not set
iotests/221: Discard image before qemu-img map
file-posix: check the use_lock before setting the file lock
iotests/308: Add test for FUSE exports
iotests: Enable fuse for many tests
iotests: Allow testing FUSE exports
iotests: Give access to the qemu-storage-daemon
storage-daemon: Call bdrv_close_all() on exit
iotests/287: Clean up subshell test image
iotests: Let _make_test_img guess $TEST_IMG_FILE
iotests: Restrict some Python tests to file
iotests/091: Use _cleanup_qemu instad of "wait"
iotests: Derive image names from $TEST_IMG
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Enable fuse for many tests

Many tests (that do not support generic protocols) can run just fine
with FUSE-exported images, so allow them to. Note that this is no
attempt at being definitel

iotests: Enable fuse for many tests

Many tests (that do not support generic protocols) can run just fine
with FUSE-exported images, so allow them to. Note that this is no
attempt at being definitely complete. There are some tests that might
be modified to run on FUSE, but this patch still skips them. This patch
only tries to pick the rather low-hanging fruits.

Note that 221 and 250 only pass when .lseek is correctly implemented,
which is only possible with a libfuse that is 3.8 or newer.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20201027190600.192171-20-mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-01-06' into staging

Block patches:
- Minor fixes and tests from the freeze period (too minor to be included
in 4.2)
- Allow many

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-01-06' into staging

Block patches:
- Minor fixes and tests from the freeze period (too minor to be included
in 4.2)
- Allow many bash iotests to test qcow2's external data file feature
- Add compress filter driver
- Fix Python iotests after 6f6e1698a6
- Fix for the backup job

# gpg: Signature made Mon 06 Jan 2020 14:33:06 GMT
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-01-06: (34 commits)
backup-top: Begin drain earlier
tests/qemu-iotests: Update tests to recent desugarized -accel option
tests/qemu-iotests: add case to write compressed data of multiple clusters
qcow2: Allow writing compressed data of multiple clusters
block: introduce compress filter driver
iotests: Allow check -o data_file
iotests: Disable data_file where it cannot be used
iotests: Make 198 work with data_file
iotests: Make 137 work with data_file
iotests: Make 110 work with data_file
iotests: Make 091 work with data_file
iotests: Avoid cp/mv of test images
iotests: Use _rm_test_img for deleting test images
iotests: Avoid qemu-img create
iotests: Drop IMGOPTS use in 267
iotests: Replace IMGOPTS='' by --no-opts
iotests: Replace IMGOPTS= by -o
iotests: Inject space into -ocompat=0.10 in 051
iotests: Add -o and --no-opts to _make_test_img
iotests: Let _make_test_img parse its parameters
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Disable data_file where it cannot be used

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20191107163708.833192-22-mreitz@redhat.com
[mreitz: Also disable 273]
Signed-off-by: Max R

iotests: Disable data_file where it cannot be used

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-id: 20191107163708.833192-22-mreitz@redhat.com
[mreitz: Also disable 273]
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

iotests: Replace IMGOPTS= by -o

Tests should not overwrite all user-supplied image options, but only add
to it (which will effectively overwrite conflicting values). Accomplish
this by passing opti

iotests: Replace IMGOPTS= by -o

Tests should not overwrite all user-supplied image options, but only add
to it (which will effectively overwrite conflicting values). Accomplish
this by passing options to _make_test_img via -o instead of $IMGOPTS.

For some tests, there is no functional change because they already only
appended options to IMGOPTS. For these, this patch is just a
simplification.

For others, this is a change, so they now heed user-specified $IMGOPTS.
Some of those tests do not work with all image options, though, so we
need to disable them accordingly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-id: 20191107163708.833192-12-mreitz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/stsquad/tags/pull-tesing-and-misc-191219-1' into staging

Various testing and logging updates

- test tci with Travis
- enable multiarch testing in Travis

Merge remote-tracking branch 'remotes/stsquad/tags/pull-tesing-and-misc-191219-1' into staging

Various testing and logging updates

- test tci with Travis
- enable multiarch testing in Travis
- default to out-of-tree builds
- make changing logfile safe via RCU
- remove redundant tests
- remove gtester test from docker
- convert DEBUG_MMAP to tracepoints
- remove hand rolled glob function
- trigger tcg re-configure when needed

# gpg: Signature made Thu 19 Dec 2019 08:24:08 GMT
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-tesing-and-misc-191219-1: (25 commits)
tests/tcg: ensure we re-configure if configure.sh is updated
trace: replace hand-crafted pattern_glob with g_pattern_match_simple
linux-user: convert target_munmap debug to a tracepoint
linux-user: log page table changes under -d page
linux-user: add target_mmap_complete tracepoint
linux-user: convert target_mmap debug to tracepoint
linux-user: convert target_mprotect debug to tracepoint
travis.yml: Remove the redundant clang-with-MAIN_SOFTMMU_TARGETS entry
docker: gtester is no longer used
Added tests for close and change of logfile.
Add use of RCU for qemu_logfile.
qemu_log_lock/unlock now preserves the qemu_logfile handle.
Add a mutex to guarantee single writer to qemu_logfile handle.
Cleaned up flow of code in qemu_set_log(), to simplify and clarify.
Fix double free issue in qemu_set_log_filename().
ci: build out-of-tree
travis.yml: Enable builds on arm64, ppc64le and s390x
tests/test-util-filemonitor: Skip test on non-x86 Travis containers
tests/hd-geo-test: Skip test when images can not be created
iotests: Skip test 079 if it is not possible to create large files
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Provide a function for checking the creation of huge files

Some tests create huge (but sparse) files, and to be able to run those
tests in certain limited environments (like CI containers),

iotests: Provide a function for checking the creation of huge files

Some tests create huge (but sparse) files, and to be able to run those
tests in certain limited environments (like CI containers), we have to
check for the possibility to create such files first. Thus let's introduce
a common function to check for large files, and replace the already
existing checks in the iotests 005 and 220 with this function.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20191204154618.23560-2-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

iotests: Provide a function for checking the creation of huge files

Some tests create huge (but sparse) files, and to be able to run those
tests in certain limited environments (like CI containers),

iotests: Provide a function for checking the creation of huge files

Some tests create huge (but sparse) files, and to be able to run those
tests in certain limited environments (like CI containers), we have to
check for the possibility to create such files first. Thus let's introduce
a common function to check for large files, and replace the already
existing checks in the iotests 005 and 220 with this function.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qcow2: Support for external data files
- qcow2: Default to 4KB for the qcow2 cache entry size
- Ap

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qcow2: Support for external data files
- qcow2: Default to 4KB for the qcow2 cache entry size
- Apply block driver whitelist for -drive format=help
- Several qemu-iotests improvements

# gpg: Signature made Fri 08 Mar 2019 12:54:27 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (33 commits)
qcow2 spec: Describe string header extensions
qemu-iotests: Add dependency to qemu-nbd tool
ahci-test: Add dependency to qemu-img tool
qemu-iotests: amend with external data file
qemu-iotests: General tests for qcow2 with external data file
qemu-iotests: Preallocation with external data file
qcow2: Implement data-file-raw create option
qcow2: Store data file name in the image
qcow2: Creating images with external data file
qcow2: Add basic data-file infrastructure
qcow2: Support external data file in qemu-img check
qcow2: Return error for snapshot operation with data file
qcow2: External file I/O
qcow2: Prepare qcow2_co_block_status() for data file
qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()
qcow2: Don't assume 0 is an invalid cluster offset
qcow2: Prepare count_contiguous_clusters() for external data file
qcow2: Prepare qcow2_get_cluster_type() for external data file
qcow2: Pass bs to qcow2_get_cluster_type()
qcow2: Basic definitions for external data files
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-iotests: Improve portability by searching bash in the $PATH

Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang

qemu-iotests: Improve portability by searching bash in the $PATH

Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.

Patch created mechanically by running:

$ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \
| while read f; do \
sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \
done

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Fix shared permission locks after reopen
- block: Fix error path for failed .bdrv_reop

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- file-posix: Fix shared permission locks after reopen
- block: Fix error path for failed .bdrv_reopen_prepare
- qcow2: Catch invalid allocations when the image becomes too large
- vvfat/fdc/nvme: Fix segfaults and leaks

# gpg: Signature made Mon 19 Nov 2018 14:28:18 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
iotests: Test file-posix locking and reopen
file-posix: Fix shared locks on reopen commit
block: Always abort reopen after prepare succeeded
iotests: Add new test 220 for max compressed cluster offset
qcow2: Don't allow overflow during cluster allocation
qcow2: Document some maximum size constraints
vvfat: Fix memory leak
fdc: fix segfault in fdctrl_stop_transfer() when DMA is disabled
nvme: fix oob access issue(CVE-2018-16847)

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Add new test 220 for max compressed cluster offset

If you have a capable file system (tmpfs is good, ext4 not so much;
run ./check with TEST_DIR pointing to a good location so as not
to ski

iotests: Add new test 220 for max compressed cluster offset

If you have a capable file system (tmpfs is good, ext4 not so much;
run ./check with TEST_DIR pointing to a good location so as not
to skip the test), it's actually possible to create a qcow2 file
that expands to a sparse 512T image with just over 38M of content.
The test is not the world's fastest (qemu crawling through 256M
bits of refcount table to find the next cluster to allocate takes
several seconds, as does qemu-img check reporting millions of
leaked clusters); but it DOES catch the problem that the previous
patch just fixed where writing a compressed cluster to a full
image ended up overwriting the wrong cluster.

Suggested-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...