Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration bl

Merge remote-tracking branch 'remotes/hreitz-gitlab/tags/pull-block-2022-02-01' into staging

Block patches:
- Add support to the iotests to test qcow2's zstd compression mode
- Fix post-migration block node permissions
- iotests fixes (051 and mirror-ready-cancel-error)
- Remove an outdated comment

# gpg: Signature made Tue 01 Feb 2022 13:34:54 GMT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF

* remotes/hreitz-gitlab/tags/pull-block-2022-02-01: (24 commits)
block.h: remove outdated comment
iotests/migration-permissions: New test
block-backend: Retain permissions after migration
iotests: declare lack of support for compresion_type in IMGOPTS
iotest 214: explicit compression type
iotests 60: more accurate set dirty bit in qcow2 header
iotests: bash tests: filter compression type
iotest 39: use _qcow2_dump_header
iotests: massive use _qcow2_dump_header
iotests/common.rc: introduce _qcow2_dump_header helper
qcow2: simple case support for downgrading of qcow2 images with zstd
iotest 302: use img_info_log() helper
iotests.py: filter compression type out
iotests.py: filter out successful output of qemu-img create
iotest 065: explicit compression type
iotest 303: explicit compression type
iotests.py: rewrite default luks support in qemu_img
iotests: drop qemu_img_verbose() helper
iotests.py: qemu_img*("create"): support IMGOPTS='compression_type=zstd'
iotests: specify some unsupported_imgopts for python iotests
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: specify some unsupported_imgopts for python iotests

We are going to support IMGOPTS for python iotests. Still some iotests
will not work with common IMGOPTS used with bash iotests like
spec

iotests: specify some unsupported_imgopts for python iotests

We are going to support IMGOPTS for python iotests. Still some iotests
will not work with common IMGOPTS used with bash iotests like
specifying refcount_bits and compat qcow2 options. So we
should define corresponding unsupported_imgopts for now.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20211223160144.1097696-4-vsementsov@virtuozzo.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/dep-many-pull-request' into staging

Remove many old deprecated features

The following features have been deprecated for well over the 2
re

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/dep-many-pull-request' into staging

Remove many old deprecated features

The following features have been deprecated for well over the 2
release cycle we promise

``-drive file=json:{...{'driver':'file'}}`` (since 3.0)
``-vnc acl`` (since 4.0.0)
``-mon ...,control=readline,pretty=on|off`` (since 4.1)
``migrate_set_downtime`` and ``migrate_set_speed`` (since 2.8.0)
``query-named-block-nodes`` result ``encryption_key_missing`` (since 2.10.0)
``query-block`` result ``inserted.encryption_key_missing`` (since 2.10.0)
``migrate-set-cache-size`` and ``query-migrate-cache-size`` (since 2.11.0)
``query-named-block-nodes`` and ``query-block`` result dirty-bitmaps[i].status (since 4.0)
``query-cpus`` (since 2.12.0)
``query-cpus-fast`` ``arch`` output member (since 3.0.0)
``query-events`` (since 4.0)
chardev client socket with ``wait`` option (since 4.0)
``acl_show``, ``acl_reset``, ``acl_policy``, ``acl_add``, ``acl_remove`` (since 4.0.0)
``ide-drive`` (since 4.2)
``scsi-disk`` (since 4.2)

# gpg: Signature made Thu 18 Mar 2021 09:23:39 GMT
# gpg: using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg: aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF

* remotes/berrange-gitlab/tags/dep-many-pull-request:
block: remove support for using "file" driver with block/char devices
block: remove 'dirty-bitmaps' field from 'BlockInfo' struct
block: remove dirty bitmaps 'status' field
block: remove 'encryption_key_missing' flag from QAPI
hw/scsi: remove 'scsi-disk' device
hw/ide: remove 'ide-drive' device
chardev: reject use of 'wait' flag for socket client chardevs
machine: remove 'arch' field from 'query-cpus-fast' QMP command
machine: remove 'query-cpus' QMP command
migrate: remove QMP/HMP commands for speed, downtime and cache size
monitor: remove 'query-events' QMP command
monitor: raise error when 'pretty' option is used with HMP
ui, monitor: remove deprecated VNC ACL option and HMP commands

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: remove 'dirty-bitmaps' field from 'BlockInfo' struct

The same data is available in the 'BlockDeviceInfo' struct.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-o

block: remove 'dirty-bitmaps' field from 'BlockInfo' struct

The same data is available in the 'BlockDeviceInfo' struct.

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server use

Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2021-01-20' into staging

nbd patches for 2021-01-20

- minor resource leak fixes in qemu-nbd
- ensure proper aio context when nbd server uses iothreads
- iotest refactorings in preparation for rewriting ./check to be more
flexible, and preparing for more nbd server reconnect features

# gpg: Signature made Thu 21 Jan 2021 02:28:19 GMT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2021-01-20:
iotests.py: qemu_io(): reuse qemu_tool_pipe_and_status()
iotests.py: fix qemu_tool_pipe_and_status()
iotests/264: fix style
iotests: define group in each iotest
iotests/294: add shebang line
iotests: make tests executable
iotests: fix some whitespaces in test output files
iotests/303: use dot slash for qcow2.py running
iotests/277: use dot slash for nbd-fault-injector.py running
nbd/server: Quiesce coroutines on context switch
block: Honor blk_set_aio_context() context requirements
qemu-nbd: Fix a memleak in nbd_client_thread()
qemu-nbd: Fix a memleak in qemu_nbd_client_list()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' grou

iotests: define group in each iotest

We are going to drop group file. Define group in tests as a preparatory
step.

The patch is generated by

cd tests/qemu-iotests

grep '^[0-9]\{3\} ' group | while read line; do
file=$(awk '{print $1}' <<< "$line");
groups=$(sed -e 's/^... //' <<< "$line");
awk "NR==2{print \"# group: $groups\"}1" $file > tmp;
cat tmp > $file;
done

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20210116134424.82867-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-05' into staging

Block patches:
- Asynchronous copying for block-copy (i.e., the backup job)
- Allow resizing of qcow2 images w

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-05-05' into staging

Block patches:
- Asynchronous copying for block-copy (i.e., the backup job)
- Allow resizing of qcow2 images when they have internal snapshots
- iotests: Logging improvements for Python tests
- iotest 153 fix, and block comment cleanups

# gpg: Signature made Tue 05 May 2020 13:56:58 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-05-05: (24 commits)
block/block-copy: use aio-task-pool API
block/block-copy: refactor task creation
block/block-copy: add state pointer to BlockCopyTask
block/block-copy: alloc task on each iteration
block/block-copy: rename in-flight requests to tasks
Fix iotest 153
block: Comment cleanups
qcow2: Tweak comment about bitmaps vs. resize
qcow2: Allow resize of images with internal snapshots
block: Add blk_new_with_bs() helper
iotests: use python logging for iotests.log()
iotests: Mark verify functions as private
iotest 258: use script_main
iotests: add script_initialize
iotests: add hmp helper with logging
iotests: limit line length to 79 chars
iotests: touch up log function signature
iotests: drop pre-Python 3.4 compatibility code
iotests: alphabetize standard imports
iotests: add pylintrc file
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: add script_initialize

Like script_main, but doesn't require a single point of entry.
Replace all existing initialization sections with this drop-in replacement.

This brings debug support t

iotests: add script_initialize

Like script_main, but doesn't require a single point of entry.
Replace all existing initialization sections with this drop-in replacement.

This brings debug support to all existing script-style iotests.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200331000014.11581-12-jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
[mreitz: Give 274 the same treatment]
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/philmd-gitlab/tags/python-next-20200207' into staging

- Python 3 cleanups:
. Remove text about Python 2 in qemu-deprecated (Thomas)
. Remove shebang header

Merge remote-tracking branch 'remotes/philmd-gitlab/tags/python-next-20200207' into staging

- Python 3 cleanups:
. Remove text about Python 2 in qemu-deprecated (Thomas)
. Remove shebang header (Paolo, Philippe)
. scripts/checkpatch.pl now allows Python 3 interpreter (Philippe)
. Explicit usage of Python 3 interpreter in scripts (Philippe)
. Fix Python scripts permissions (Paolo, Philippe)
. Drop 'from __future__ import print_function' (Paolo)
. Specify minimum python requirements in ReadTheDocs configuration (Alex)
- Test UNIX/EXEC transports with migration (Oksana)
- Added extract_from_rpm helper, improved extract_from_deb (Liam)
- Allow to use other serial consoles than default one (Philippe)
- Various improvements in QEMUMonitorProtocol (Wainer)
- Fix kvm_available() on ppc64le (Wainer)

# gpg: Signature made Fri 07 Feb 2020 15:01:56 GMT
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/python-next-20200207: (46 commits)
.readthedocs.yml: specify some minimum python requirements
drop "from __future__ import print_function"
make all Python scripts executable
scripts/signrom: remove Python 2 support, add shebang
tests/qemu-iotests/check: Only check for Python 3 interpreter
scripts: Explicit usage of Python 3 (scripts without __main__)
tests/qemu-iotests: Explicit usage of Python3 (scripts without __main__)
tests/vm: Remove shebang header
tests/acceptance: Remove shebang header
scripts/tracetool: Remove shebang header
scripts/minikconf: Explicit usage of Python 3
scripts: Explicit usage of Python 3 (scripts with __main__)
tests: Explicit usage of Python 3
tests/qemu-iotests: Explicit usage of Python 3 (scripts with __main__)
tests/qemu-iotests/check: Allow use of python3 interpreter
scripts/checkpatch.pl: Only allow Python 3 interpreter
tests/acceptance/migration: Default to -nodefaults
tests/acceptance/migration: Add the 'migration' tag
tests/acceptance/migration: Test EXEC transport when migrating
tests/acceptance/migration: Test UNIX transport when migrating
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

tests/qemu-iotests: Explicit usage of Python3 (scripts without __main__)

Use the program search path to find the Python 3 interpreter.

Patch created mechanically by running:

$ sed -i "s,^#\!/usr

tests/qemu-iotests: Explicit usage of Python3 (scripts without __main__)

Use the program search path to find the Python 3 interpreter.

Patch created mechanically by running:

$ sed -i "s,^#\!/usr/bin/\(env\ \)\?python$,#\!/usr/bin/env python3," \
$(git grep -lF '#!/usr/bin/env python' \
| xargs grep -L 'if __name__.*__main__')

Reported-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20200130163232.10446-11-philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

Rebase notes:

011/36:[0003] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
016/36:[--

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

Rebase notes:

011/36:[0003] [FC] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
016/36:[----] [-C] 'iotests: Add virtio-scsi device helper'
017/36:[0002] [FC] 'iotests: add test 257 for bitmap-mode backups'
030/36:[0011] [FC] 'block/backup: teach TOP to never copy unallocated regions'
032/36:[0018] [FC] 'iotests/257: test traditional sync modes'

11: A new hbitmap call was added late in 4.1, changed to
bdrv_dirty_bitmap_next_zero.
16: Context-only (self.has_quit is new context in 040)
17: Removed 'auto' to follow upstream trends in iotest fashion
30: Handled explicitly on-list with R-B from Max.
32: Fix capitalization in test, as mentioned on-list.

# gpg: Signature made Sat 17 Aug 2019 00:12:13 BST
# gpg: using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F 18F2 88A9 064D 1835 61EB
# Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76 CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request: (36 commits)
tests/test-hbitmap: test next_zero and _next_dirty_area after truncate
block/backup: refactor write_flags
block/backup: deal with zero detection
qapi: add dirty-bitmaps to query-named-block-nodes result
iotests/257: test traditional sync modes
block/backup: support bitmap sync modes for non-bitmap backups
block/backup: teach TOP to never copy unallocated regions
block/backup: add backup_is_cluster_allocated
block/backup: centralize copy_bitmap initialization
block/backup: improve sync=bitmap work estimates
iotests/257: test API failures
block/backup: hoist bitmap check into QMP interface
iotests/257: Refactor backup helpers
iotests/257: add EmulatedBitmap class
iotests/257: add Pattern class
iotests: test bitmap moving inside 254
qapi: implement block-dirty-bitmap-remove transaction action
blockdev: reduce aio_context locked sections in bitmap add/remove
block/backup: loosen restriction on readonly bitmaps
iotests: add test 257 for bitmap-mode backups
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: test bitmap moving inside 254

Test persistent bitmap copying with and without removal of original
bitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-

iotests: test bitmap moving inside 254

Test persistent bitmap copying with and without removal of original
bitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-id: 20190708220502.12977-4-jsnow@redhat.com
[Edited comment "bitmap1" --> "bitmap2" as per review. --js]
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-06-14' into staging

Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
context to newly added nod

Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-06-14' into staging

Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
context to newly added nodes
- Add salvaging mode to qemu-img convert
- Minor fixes to tests, documentation, and for less Valgrind annoyance

# gpg: Signature made Fri 14 Jun 2019 14:38:11 BST
# gpg: using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg: issuer "mreitz@redhat.com"
# gpg: Good signature from "Max Reitz <mreitz@redhat.com>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1 1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-06-14:
iotests: Test qemu-img convert -C --salvage
iotests: Test qemu-img convert --salvage
blkdebug: Inject errors on .bdrv_co_block_status()
blkdebug: Add "none" event
blkdebug: Add @iotype error option
qemu-img: Add salvaging mode to convert
qemu-img: Move quiet into ImgConvertState
blockdev: Overlays are not snapshots
qapi/block-core: Overlays are not snapshots
qemu-img: Fix options leakage in img_rebase()
iotests: restrict 254 to support only qcow2
hw/block/fdc: floppy command FIFO memory initialization
iotests: Fix intermittent failure in 219
iotests: Filter 175's allocation information
event_match: always match on None value
iotests: add iotest 256 for testing blockdev-backup across iothread contexts
iotests.py: rewrite run_job to be pickier
QEMUMachine: add events_wait method
iotests.py: do not use infinite waits
blockdev-backup: don't check aio_context too early

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: restrict 254 to support only qcow2

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge)

iotests: restrict 254 to support only qcow2

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge).

The test aim is to check block-dirty-bitmap-merge between different
nodes functionality, no needs to check all formats. So, let's just drop
support for anything except qcow2.

Reported-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20190605155405.104384-1-vsementsov@virtuozzo.com
Signed-off-by: Max Reitz <mreitz@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

# gpg: Signature made Wed 29 May 2019 00:58:33 BST
# gpg: using RSA key F9B7ABDBBCAC

Merge remote-tracking branch 'remotes/jnsnow/tags/bitmaps-pull-request' into staging

Pull request

# gpg: Signature made Wed 29 May 2019 00:58:33 BST
# gpg: using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F 18F2 88A9 064D 1835 61EB
# Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76 CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request:
iotests: test external snapshot with bitmap copying
qapi: support external bitmaps in block-dirty-bitmap-merge
migration/dirty-bitmaps: change bitmap enumeration method

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: test external snapshot with bitmap copying

This test shows that external snapshots and incremental backups are
friends.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.co

iotests: test external snapshot with bitmap copying

This test shows that external snapshots and incremental backups are
friends.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20190517152111.206494-3-vsementsov@virtuozzo.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...