Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-08-21' into staging

bitmaps patches for 2020-08-21

- Andrey Shinkevich: Enhance qcow2.py for iotest inspection of qcow2 images
- M

Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-08-21' into staging

bitmaps patches for 2020-08-21

- Andrey Shinkevich: Enhance qcow2.py for iotest inspection of qcow2 images
- Max Reitz: Add block-bitmap-mapping migration parameter

# gpg: Signature made Fri 21 Aug 2020 15:05:07 BST
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-bitmaps-2020-08-21:
iotests: Test node/bitmap aliases during migration
iotests.py: Let wait_migration() return on failure
migration: Add block-bitmap-mapping parameter
iotests: dump QCOW2 header in JSON in #303
qcow2_format.py: support dumping metadata in JSON format
qcow2_format.py: collect fields to dump in JSON format
qcow2.py: Introduce '-j' key to dump in JSON format
qcow2_format.py: Dump bitmap table serialized entries
qcow2_format.py: pass cluster size to substructures
qcow2_format.py: Dump bitmap directory information
qcow2_format.py: dump bitmap flags in human readable way.
qcow2_format.py: change Qcow2BitmapExt initialization method
qcow2_format.py: make printable data an extension class member
iotests: add test for QCOW2 header dump

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2_format.py: support dumping metadata in JSON format

Implementation of dumping QCOW2 image metadata.
The sample output:
{
"Header_extensions": [
{
"name": "Feature table"

qcow2_format.py: support dumping metadata in JSON format

Implementation of dumping QCOW2 image metadata.
The sample output:
{
"Header_extensions": [
{
"name": "Feature table",
"magic": 1745090647,
"length": 192,
"data_str": "<binary>"
},
{
"name": "Bitmaps",
"magic": 595929205,
"length": 24,
"data": {
"nb_bitmaps": 2,
"reserved32": 0,
"bitmap_directory_size": 64,
"bitmap_directory_offset": 1048576,
"bitmap_directory": [
{
"name": "bitmap-1",
"bitmap_table_offset": 589824,
"bitmap_table_size": 1,
"flags": 2,
"type": 1,
"granularity_bits": 15,
"name_size": 8,
"extra_data_size": 0,
"bitmap_table": [
{
"type": "serialized",
"offset": 655360
},
...

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-11-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: collect fields to dump in JSON format

As __dict__ is being extended with class members we do not want to
print, add the to_json() method to classes that returns a json-dumpable
obje

qcow2_format.py: collect fields to dump in JSON format

As __dict__ is being extended with class members we do not want to
print, add the to_json() method to classes that returns a json-dumpable
object with desired fields and their values. Extend it in subclass when
necessary to print the final dictionary in the JSON output which
follows.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <1596742557-320265-10-git-send-email-andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2.py: Introduce '-j' key to dump in JSON format

Add the command key to the qcow2.py arguments list to dump QCOW2
metadata in JSON format. Here is the suggested way to do that. The
implementation

qcow2.py: Introduce '-j' key to dump in JSON format

Add the command key to the qcow2.py arguments list to dump QCOW2
metadata in JSON format. Here is the suggested way to do that. The
implementation of the dump in JSON format is in the patch that follows.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-9-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: Dump bitmap table serialized entries

Add bitmap table information to the QCOW2 metadata dump.

Bitmap name bitmap-1
...
Bitmap table type size off

qcow2_format.py: Dump bitmap table serialized entries

Add bitmap table information to the QCOW2 metadata dump.

Bitmap name bitmap-1
...
Bitmap table type size offset
0 serialized 65536 10092544
1 all-zeroes 0 0
2 all-zeroes 0 0

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-8-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: pass cluster size to substructures

The cluster size of an image is the QcowHeader class member and may be
obtained by dependent extension structures such as Qcow2BitmapExt for
furth

qcow2_format.py: pass cluster size to substructures

The cluster size of an image is the QcowHeader class member and may be
obtained by dependent extension structures such as Qcow2BitmapExt for
further bitmap table details print.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-7-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: Dump bitmap directory information

Read and dump entries from the bitmap directory of QCOW2 image.

Header extension:
magic 0x23852875 (Bitmaps)
...
Bitmap name

qcow2_format.py: Dump bitmap directory information

Read and dump entries from the bitmap directory of QCOW2 image.

Header extension:
magic 0x23852875 (Bitmaps)
...
Bitmap name bitmap-1
bitmap_table_offset 0xf0000
bitmap_table_size 1
flags 0x2 (['auto'])
type 1
granularity_bits 16
name_size 8
extra_data_size 0

Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-6-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: dump bitmap flags in human readable way.

Introduce the class BitmapFlags that parses a bitmap flags mask.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Sign

qcow2_format.py: dump bitmap flags in human readable way.

Introduce the class BitmapFlags that parses a bitmap flags mask.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-5-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: change Qcow2BitmapExt initialization method

There are two ways to initialize a class derived from Qcow2Struct:
1. Pass a block of binary data to the constructor.
2. Pass the file de

qcow2_format.py: change Qcow2BitmapExt initialization method

There are two ways to initialize a class derived from Qcow2Struct:
1. Pass a block of binary data to the constructor.
2. Pass the file descriptor to allow reading the file from constructor.
Let's change the Qcow2BitmapExt initialization method from 1 to 2 to
support a scattered reading in the initialization chain.
The implementation comes with the patch that follows.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-4-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: make printable data an extension class member

Let us differ binary data type from string one for the extension data
variable and keep the string as the QcowHeaderExtension class mem

qcow2_format.py: make printable data an extension class member

Let us differ binary data type from string one for the extension data
variable and keep the string as the QcowHeaderExtension class member.

Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <1596742557-320265-3-git-send-email-andrey.shinkevich@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- enhance handling of size-related BlockConf properties
- nvme: small fixes, refactoring and cleanup

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block layer patches:

- enhance handling of size-related BlockConf properties
- nvme: small fixes, refactoring and cleanups
- virtio-blk: On restart, process queued requests in the proper context
- icount: make dma reads deterministic
- iotests: Some fixes for rarely run cases
- .gitignore: Ignore storage-daemon files
- Minor code cleanups

# gpg: Signature made Wed 17 Jun 2020 15:47:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream: (43 commits)
iotests: Add copyright line in qcow2.py
iotests/{190,291}: compat=0.10 is unsupported
iotests/229: data_file is unsupported
iotests/292: data_file is unsupported
iotests/041: Skip test_small_target for qed
iotests.py: Add skip_for_formats() decorator
block: lift blocksize property limit to 2 MiB
qdev-properties: add getter for size32 and blocksize
block: make BlockConf size props 32bit and accept size suffixes
qdev-properties: make blocksize accept size suffixes
qdev-properties: add size32 property type
qdev-properties: blocksize: use same limits in code and description
block: consolidate blocksize properties consistency checks
virtio-blk: store opt_io_size with correct size
.gitignore: Ignore storage-daemon files
hw/block/nvme: verify msix_init_exclusive_bar() return value
hw/block/nvme: add msix_qsize parameter
hw/block/nvme: Verify msix_vector_use() returned value
hw/block/nvme: factor out controller identify setup
hw/block/nvme: do cmb/pmr init as part of pci init
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

iotests: Add copyright line in qcow2.py

The file qcow2.py was originally contributed in 2012 by Kevin Wolf,
but was not given traditional boilerplate headers at the time. The
missing license was ju

iotests: Add copyright line in qcow2.py

The file qcow2.py was originally contributed in 2012 by Kevin Wolf,
but was not given traditional boilerplate headers at the time. The
missing license was just rectified (commit 16306a7b39) using the
project-default GPLv2+, but as Vladimir is not at Red Hat, he did not
add a Copyright line. All earlier contributions have come from CC'd
authors, where all but Stefan used a Red Hat address at the time of
the contribution, and that copyright carries over to the split to
qcow2_format.py (d5262c7124).

CC: Kevin Wolf <kwolf@redhat.com>
CC: Stefan Hajnoczi <stefanha@redhat.com>
CC: Eduardo Habkost <ehabkost@redhat.com>
CC: Max Reitz <mreitz@redhat.com>
CC: Philippe Mathieu-Daudé <philmd@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200609205944.3549240-1-eblake@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-06-09' into staging

bitmaps patches for 2020-06-09

- documenation fix
- various improvements to qcow2.py program used in iotests

Merge remote-tracking branch 'remotes/ericb/tags/pull-bitmaps-2020-06-09' into staging

bitmaps patches for 2020-06-09

- documenation fix
- various improvements to qcow2.py program used in iotests

# gpg: Signature made Tue 09 Jun 2020 21:50:35 BST
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2 F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-bitmaps-2020-06-09:
iotests: Fix 291 across more file systems
qcow2_format.py: dump bitmaps header extension
qcow2: QcowHeaderExtension print names for extension magics
qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct
qcow2_format.py: QcowHeaderExtension: add dump method
qcow2_format.py: add field-formatting class
qcow2_format.py: separate generic functionality of structure classes
qcow2_format.py: use strings to specify c-type of struct fields
qcow2_format.py: use modern string formatting
qcow2_format.py: use tuples instead of lists for fields
qcow2_format.py: drop new line printing at end of dump()
qcow2.py: move qcow2 format classes to separate module
qcow2.py: add licensing blurb
qcow2.py: python style fixes
qemu-img: Fix doc typo for 'bitmap' subcommand

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qcow2_format.py: dump bitmaps header extension

Add class for bitmap extension and dump its fields. Further work is to
dump bitmap directory.

Test new functionality inside 291 iotest.

Signed-off-by

qcow2_format.py: dump bitmaps header extension

Add class for bitmap extension and dump its fields. Further work is to
dump bitmap directory.

Test new functionality inside 291 iotest.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-14-vsementsov@virtuozzo.com>
[eblake: fix iotest output]
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2: QcowHeaderExtension print names for extension magics

Suggested-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
M

qcow2: QcowHeaderExtension print names for extension magics

Suggested-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-13-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct

Only two fields we can parse by generic code, but that is better than
nothing. Keep further refactoring of variable-length fie

qcow2_format: refactor QcowHeaderExtension as a subclass of Qcow2Struct

Only two fields we can parse by generic code, but that is better than
nothing. Keep further refactoring of variable-length fields for another
day.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-12-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: QcowHeaderExtension: add dump method

Obviously, for-loop body in dump_extensions should be the dump method
of extension.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@vir

qcow2_format.py: QcowHeaderExtension: add dump method

Obviously, for-loop body in dump_extensions should be the dump method
of extension.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-11-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: add field-formatting class

Allow formatter class in structure definition instead of hacking with
'mask'. This will simplify further introduction of new formatters.

Signed-off-by: V

qcow2_format.py: add field-formatting class

Allow formatter class in structure definition instead of hacking with
'mask'. This will simplify further introduction of new formatters.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-10-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: separate generic functionality of structure classes

We are going to introduce more Qcow2 structure types, defined like
QcowHeader. Move generic functionality into base class to be r

qcow2_format.py: separate generic functionality of structure classes

We are going to introduce more Qcow2 structure types, defined like
QcowHeader. Move generic functionality into base class to be reused for
further structure classes.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-9-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: use strings to specify c-type of struct fields

We are going to move field-parsing to super-class, this will be simpler
with simple string specifiers instead of variables.

For some

qcow2_format.py: use strings to specify c-type of struct fields

We are going to move field-parsing to super-class, this will be simpler
with simple string specifiers instead of variables.

For some reason, python doesn't allow the definition of ctypes variable
in the class alongside fields: it would not be available then for use
by the 'for' operator. Don't worry: ctypes will be moved to metaclass
soon.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-8-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: use modern string formatting

Use .format and f-strings instead of old %style. Also, the file uses
both '' and "" quotes, for consistency let's use '', except for cases
when we need

qcow2_format.py: use modern string formatting

Use .format and f-strings instead of old %style. Also, the file uses
both '' and "" quotes, for consistency let's use '', except for cases
when we need '' inside the string (use "" to avoid extra escaping).

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-7-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: use tuples instead of lists for fields

No need in lists: it's a constant variable.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinke

qcow2_format.py: use tuples instead of lists for fields

No need in lists: it's a constant variable.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-6-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2_format.py: drop new line printing at end of dump()

This will simplify further conversion. To compensate, print this empty
line directly in cmd_dump_header().

Signed-off-by: Vladimir Sementsov

qcow2_format.py: drop new line printing at end of dump()

This will simplify further conversion. To compensate, print this empty
line directly in cmd_dump_header().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Message-Id: <20200606081806.23897-5-vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...

qcow2.py: move qcow2 format classes to separate module

We are going to enhance qcow2 format parsing by adding more structure
classes. Let's split format parsing from utility code.

Signed-off-by: Vl

qcow2.py: move qcow2 format classes to separate module

We are going to enhance qcow2 format parsing by adding more structure
classes. Let's split format parsing from utility code.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200606081806.23897-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>

show more ...