Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server use

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server uses iothreads
- iotest refactorings in preparation for rewriting ./check to be more
flexible, and preparing for more nbd server reconnect features

# gpg: Signature made Thu 21 Jan 2021 02:28:19 GMT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-01-20:
iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
iotests.py: fix qemu_tool_pipe_and_status()
iotests/264: fix style
iotests: define group in each iotest
iotests/294: add shebang line
iotests: make tests executable
iotests: fix some whitespaces in test output files
iotests/303: use dot slash for qcow2.py running
iotests/277: use dot slash for nbd-fault-injector.py running
nbd/server: Quiesce coroutines on context switch
block: Honor blk_set_aio_context() context requirements
qemu-nbd: Fix a memleak in nbd_client_thread()
qemu-nbd: Fix a memleak in qemu_nbd_client_list()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' grou

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qcow2: Support for external data files
- qcow2: Default to 4KB for the qcow2 cache entry size
- Ap

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- qcow2: Support for external data files
- qcow2: Default to 4KB for the qcow2 cache entry size
- Apply block driver whitelist for -drive format=help
- Several qemu-iotests improvements

# gpg: Signature made Fri 08 Mar 2019 12:54:27 GMT
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (33 commits)
qcow2 spec: Describe string header extensions
qemu-iotests: Add dependency to qemu-nbd tool
ahci-test: Add dependency to qemu-img tool
qemu-iotests: amend with external data file
qemu-iotests: General tests for qcow2 with external data file
qemu-iotests: Preallocation with external data file
qcow2: Implement data-file-raw create option
qcow2: Store data file name in the image
qcow2: Creating images with external data file
qcow2: Add basic data-file infrastructure
qcow2: Support external data file in qemu-img check
qcow2: Return error for snapshot operation with data file
qcow2: External file I/O
qcow2: Prepare qcow2_co_block_status() for data file
qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset()
qcow2: Don't assume 0 is an invalid cluster offset
qcow2: Prepare count_contiguous_clusters() for external data file
qcow2: Prepare qcow2_get_cluster_type() for external data file
qcow2: Pass bs to qcow2_get_cluster_type()
qcow2: Basic definitions for external data files
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-iotests: Improve portability by searching bash in the $PATH

Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang

qemu-iotests: Improve portability by searching bash in the $PATH

Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.

Patch created mechanically by running:

$ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \
| while read f; do \
sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \
done

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2018-11-19' into staging

nbd patches for 2018-11-19

Add iotest coverage for NBD connections using TLS, including
a couple of code fixes tha

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2018-11-19' into staging

nbd patches for 2018-11-19

Add iotest coverage for NBD connections using TLS, including
a couple of code fixes that it pointed out

- Mao Zhongyi: 0/3 Do some cleaning work in qemu-iotests
- Daniel P. Berrangé: io: return 0 for EOF in TLS session read after shutdown
- Daniel P. Berrangé: 0/6 Misc fixes to NBD
- Eric Blake: iotests: Drop use of bash keyword 'function'

# gpg: Signature made Mon 19 Nov 2018 17:43:32 GMT
# gpg: using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>"
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>"
# gpg: aka "[jpeg image of size 6874]"
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2018-11-19:
iotests: Drop use of bash keyword 'function'
iotests: Also test I/O over NBD TLS
tests: exercise NBD server in TLS mode
tests: add iotests helpers for dealing with TLS certificates
tests: check if qemu-nbd is still alive before waiting
tests: pull qemu-nbd iotest helpers into common.nbd file
io: return 0 for EOF in TLS session read after shutdown
nbd/server: Ignore write errors when replying to NBD_OPT_ABORT
nbd: fix whitespace in server error message
qemu-iotests: Modern shell scripting (use $() instead of ``)
qemu-iotests: convert `pwd` and $(pwd) to $PWD
qemu-iotests: remove unused variable 'here'

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Drop use of bash keyword 'function'

Bash allows functions to be declared with or without the leading
keyword 'function'; but including the keyword does not comply with
POSIX syntax, and is

iotests: Drop use of bash keyword 'function'

Bash allows functions to be declared with or without the leading
keyword 'function'; but including the keyword does not comply with
POSIX syntax, and is confusing to ksh users where the use of the
keyword changes the scoping rules for functions. Stick to the
POSIX form through iotests.

Done mechanically with:
sed -i 's/^function //' $(git ls-files tests/qemu-iotests)

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20181116215002.2124581-1-eblake@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

show more ...

qemu-iotests: remove unused variable 'here'

Running
git grep '\$here' tests/qemu-iotests

has 0 hits, which means we are setting a variable that has
no use. It appears that commit e8f8624d removed

qemu-iotests: remove unused variable 'here'

Running
git grep '\$here' tests/qemu-iotests

has 0 hits, which means we are setting a variable that has
no use. It appears that commit e8f8624d removed the last
use. So execute the following cmd to remove all of
the 'here=...' lines as dead code.

sed -i '/^here=/d' $(git grep -l '^here=' tests/qemu-iotests)

Cc: kwolf@redhat.com
Cc: mreitz@redhat.com
Cc: eblake@redhat.com
Suggested-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Mao Zhongyi <maozhongyi@cmss.chinamobile.com>
Message-Id: <20181024094051.4470-3-maozhongyi@cmss.chinamobile.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[eblake: touch up commit message, reorder series, rebase to master]
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Generic background jobs
- qemu-iotests fixes for NFS and the 'migration' group
- sheepdog: Minor c

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- Generic background jobs
- qemu-iotests fixes for NFS and the 'migration' group
- sheepdog: Minor code simplification

# gpg: Signature made Wed 23 May 2018 13:33:49 BST
# gpg: using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (46 commits)
qemu-iotests: Test job-* with block jobs
iotests: Move qmp_to_opts() to VM
blockjob: Remove BlockJob.driver
job: Add query-jobs QMP command
job: Add lifecycle QMP commands
job: Add JOB_STATUS_CHANGE QMP event
job: Introduce qapi/job.json
job: Move progress fields to Job
job: Add job_transition_to_ready()
job: Add job_is_ready()
job: Add job_dismiss()
job: Add job_yield()
block: Cancel job in bdrv_close_all() callers
job: Move completion and cancellation to Job
job: Move transactions to Job
job: Switch transactions to JobTxn
job: Move job_finish_sync() to Job
job: Move .complete callback to Job
job: Add job_drain()
job: Convert block_job_cancel_async() to Job
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-iotests: 086 doesn't work with NFS

The reference output file only works for file. 'qemu-img convert -p'
makes a lot more progress updates for NFS than for file, so disable the
test for NFS.

Si

qemu-iotests: 086 doesn't work with NFS

The reference output file only works for file. 'qemu-img convert -p'
makes a lot more progress updates for NFS than for file, so disable the
test for NFS.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches for 2.6.0-rc3

# gpg: Signature made Fri 15 Apr 2016 17:02:23 BST using RSA key ID C88F2FD6
# gpg: Goo

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches for 2.6.0-rc3

# gpg: Signature made Fri 15 Apr 2016 17:02:23 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream:
nbd: Don't kill server on client that doesn't request TLS
nbd: fix assert() on qemu-nbd stop
nbd: Don't fail handshake on NBD_OPT_LIST descriptions
qemu-iotests: 041: More robust assertion on quorum node
qemu-iotests: place valgrind log file in scratch dir
qemu-iotests: tests: do not set unused tmp variable
qemu-iotests: common.rc: drop unused _do()
qemu-iotests: drop unused _within_tolerance() filter
Fix pflash migration
block: Don't ignore flags in blk_{,co,aio}_write_zeroes()
block/vpc: update comments to be compliant w/coding guidelines
block/vpc: set errp in vpc_open
block/vpc: make checks on max table size a bit more lax
block/vpc: Use the correct max sector count for VHD images
block/vpc: use current_size field for XenConverter VHD images
vpc: use current_size field for XenServer VHD images
block/vpc: set errp in vpc_create
block: Fix blk_aio_write_zeroes()
qemu-io: Support 'aio_write -z'

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Merge remote-tracking branch 'mreitz/tags/pull-block-for-kevin-2016-04-15' into queue-block

Block patches for 2.6.0-rc3.

# gpg: Signature made Fri Apr 15 17:57:30 2016 CEST using RSA key ID E838ACA

Merge remote-tracking branch 'mreitz/tags/pull-block-for-kevin-2016-04-15' into queue-block

Block patches for 2.6.0-rc3.

# gpg: Signature made Fri Apr 15 17:57:30 2016 CEST using RSA key ID E838ACAD
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"

* mreitz/tags/pull-block-for-kevin-2016-04-15:
nbd: Don't kill server on client that doesn't request TLS
nbd: fix assert() on qemu-nbd stop
nbd: Don't fail handshake on NBD_OPT_LIST descriptions
qemu-iotests: 041: More robust assertion on quorum node
qemu-iotests: place valgrind log file in scratch dir
qemu-iotests: tests: do not set unused tmp variable
qemu-iotests: common.rc: drop unused _do()
qemu-iotests: drop unused _within_tolerance() filter

Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qemu-iotests: tests: do not set unused tmp variable

The previous commit removed the last usage of ${tmp} inside the tests
themselves; the only remaining users are sourced by check. So we can now
dro

qemu-iotests: tests: do not set unused tmp variable

The previous commit removed the last usage of ${tmp} inside the tests
themselves; the only remaining users are sourced by check. So we can now
drop this variable from the tests.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Reviewed-by: Bo Tu <tubo@linux.vnet.ibm.com>
Message-id: 1460472980-26319-4-git-send-email-silbe@linux.vnet.ibm.com
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-for-peter-2016-01-07' into staging

Block patches from 2015-12-23 until 2016-01-07.

# gpg: Signature made Thu 07 Jan 2016 22:46:08 GMT

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-for-peter-2016-01-07' into staging

Block patches from 2015-12-23 until 2016-01-07.

# gpg: Signature made Thu 07 Jan 2016 22:46:08 GMT using RSA key ID E838ACAD
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>"

* remotes/maxreitz/tags/pull-block-for-peter-2016-01-07: (21 commits)
iotests: Add test cases for blockdev-mirror
qmp: Add blockdev-mirror command
block: Add check on mirror target
block: Extract blockdev part of qmp_drive_mirror
block: Rename BLOCK_OP_TYPE_MIRROR to BLOCK_OP_TYPE_MIRROR_SOURCE
qemu-iotests: s390x: fix test 051
iotests: 095: Filter _img_info output
iotests: 095: Use TEST_IMG override instead of "mv"
iotests: 050: Use TEST_IMG override instead of "mv"
iotests: 038: Use TEST_IMG override instead of "mv"
iotests: 037: Use TEST_IMG override instead of "mv"
iotests: 034: Use TEST_IMG override instead of "mv"
iotests: 028: Use TEST_IMG override instead of "mv"
iotests: 024: Use TEST_IMG override instead of "mv"
iotests: 020: Use TEST_IMG override instead of "mv"
iotests: 019: Use TEST_IMG override instead of "mv"
iotests: 018: Use TEST_IMG override instead of "mv"
block/qapi: Clear err for further error
block: use drained section in bdrv_close
qemu-iotests: make check-block.sh work on out-of-tree builds
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: 086: Add raw format

Raw is as qualified as qcow2 for this test case, add it for more
coverage.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1450851979-15580-1-git-send-email-famz

iotests: 086: Add raw format

Raw is as qualified as qcow2 for this test case, add it for more
coverage.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1450851979-15580-1-git-send-email-famz@redhat.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Thu 23 Oct 2014 18:56:05 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Ke

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Thu 23 Oct 2014 18:56:05 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (32 commits)
qemu-img: Print error if check failed
block: char devices on FreeBSD are not behind a pager
iotests: Add test for qcow2 L1 table update
qcow2: Do not overflow when writing an L1 sector
iotests: Add test for map commands
qemu-io: Respect early image end for map
block: Respect underlying file's EOF
docs/qcow2: Limit refcount_order to [0, 6]
docs/qcow2: Correct refcount_block_entries
qcow2: Drop REFCOUNT_SHIFT
iotests: Add test for potentially damaging repairs
iotests: Fix test outputs
qcow2: Clean up after refcount rebuild
qcow2: Rebuild refcount structure during check
qcow2: Do not perform potentially damaging repairs
qcow2: Fix refcount blocks beyond image end
qcow2: Reuse refcount table in calculate_refcounts()
qcow2: Let inc_refcounts() resize the reftable
qcow2: Let inc_refcounts() return -errno
qcow2: Split fail code in L1 and L2 checks
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: qemu-iotests change _supported_proto to file once more.

In preparation to possible automatic regression and performance
testing for the block layer I found that the iotests don't work
for all

block: qemu-iotests change _supported_proto to file once more.

In preparation to possible automatic regression and performance
testing for the block layer I found that the iotests don't work
for all protocols anymore.

In commit 1f7bf7d0 I started to change supported protocols from
generic to file for various tests. Unfortunately, some tests
added in the meantime again carry generic protocol altough they
can only work with file because they require local file access.

The other way around for some tests that only support file I added
NFS protocol after confirming they work.

Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Benoît Canet <benoit.canet@nodalink.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Wed 30 Apr 2014 19:19:32 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Ke

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Wed 30 Apr 2014 19:19:32 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (31 commits)
curl: Fix hang reading from slow connections
curl: Ensure all informationals are checked for completion
curl: Eliminate unnecessary use of curl_multi_socket_all
curl: Remove unnecessary explicit calls to internal event handler
curl: Remove erroneous sleep waiting for curl completion
curl: Fix return from curl_read_cb with invalid state
curl: Remove unnecessary use of goto
curl: Fix long line
block/vdi: Error out immediately in vdi_create()
block/bochs: Fix error handling for seek_to_sector()
qcow2: Check min_size in qcow2_grow_l1_table()
qcow2: Catch bdrv_getlength() error
block: Use correct width in format strings
qcow2: Avoid overflow in alloc_clusters_noref()
block: Use error_abort in bdrv_image_info_specific_dump()
block: Fix open_flags in bdrv_reopen()
Revert "block: another bdrv_append fix"
block: Unlink temporary files in raw-posix/win32
block: Remove BDRV_O_COPY_ON_READ for bs->file
block: Create bdrv_backing_flags()
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: qemu-iotests: make test 019 and 086 work with spaced pathnames

Both tests 019 and 086 need proper quotations to work with pathnames
that contain spaces.

Reviewed-by: Benoit Canet <benoit@irq

block: qemu-iotests: make test 019 and 086 work with spaced pathnames

Both tests 019 and 086 need proper quotations to work with pathnames
that contain spaces.

Reviewed-by: Benoit Canet <benoit@irqsave.net>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Jeff Cody <jcody@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Fri 07 Mar 2014 13:30:04 GMT using RSA key ID C88F2FD6
# gpg: Good signature from "Ke

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Fri 07 Mar 2014 13:30:04 GMT using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream:
block: qemu-iotests 085 - live snapshots tests
hw/ide/ahci.h: Avoid shifting left into sign bit
block: Fix error path segfault in bdrv_open()
qemu-iotests: Test a few blockdev-add error cases
blockdev: Fix NULL pointer dereference in blockdev-add
blockdev: Fail blockdev-add with encrypted images
block/raw-win32: Strip "file:" prefix on creation
block/raw-win32: Implement bdrv_parse_filename()
block/raw-posix: Strip "file:" prefix on creation
block/raw-posix: Implement bdrv_parse_filename()
block: Keep "filename" option after parsing
block: mirror - remove code cruft that has no function
block: make bdrv_swap rebuild the bs graph node list field.
block: Fix bs->request_alignment assertion for bs->sg=1
iscsi: Use bs->sg for everything else than disks
qemu-iotests: Test progress output for conversion
qemu-img convert: Fix progress output
gluster: Remove unused defines and header include
gluster: Change licence to GPLv2+

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qemu-iotests: Test progress output for conversion

Signed-off-by: Kevin Wolf <kwolf@redhat.com>