Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Pull request

Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.

# gpg: Signature made Tue 05 Nov 2

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Pull request

Let's get the image fuzzer Python 3 changes merged in QEMU 4.2.

# gpg: Signature made Tue 05 Nov 2019 15:43:16 GMT
# gpg: using RSA key 8695A8BFD3F97CDAAC35775A9CA4ABB381AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>" [full]
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>" [full]
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35 775A 9CA4 ABB3 81AB 73C8

* remotes/stefanha/tags/block-pull-request:
image-fuzzer: Use OSerror.strerror instead of tuple subscript
image-fuzzer: Use errors parameter of subprocess.Popen()
image-fuzzer: Run using python3
image-fuzzer: Encode file name and file format to bytes
image-fuzzer: Use bytes constant for field values
image-fuzzer: Return bytes objects on string fuzzing functions
image-fuzzer: Use %r for all fiels at Field.__repr__()
image-fuzzer: Use io.StringIO
image-fuzzer: Explicitly use integer division operator
image-fuzzer: Write bytes instead of string to image file
image-fuzzer: Open image files in binary mode

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

image-fuzzer: Use OSerror.strerror instead of tuple subscript

OSError can't be used like a tuple on Python 3, so change the
code to use `e.sterror` instead of `e[1]`.

Reported-by: John Snow <jsnow@

image-fuzzer: Use OSerror.strerror instead of tuple subscript

OSError can't be used like a tuple on Python 3, so change the
code to use `e.sterror` instead of `e[1]`.

Reported-by: John Snow <jsnow@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20191021214117.18091-1-ehabkost@redhat.com
Message-Id: <20191021214117.18091-1-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

image-fuzzer: Use errors parameter of subprocess.Popen()

Instead of manually encoding stderr and stdout output, use
`errors` parameter of subprocess.Popen(). This will make
process.communicate() re

image-fuzzer: Use errors parameter of subprocess.Popen()

Instead of manually encoding stderr and stdout output, use
`errors` parameter of subprocess.Popen(). This will make
process.communicate() return unicode strings instead of bytes
objects.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-11-ehabkost@redhat.com
Message-Id: <20191016192430.25098-11-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

image-fuzzer: Run using python3

image-fuzzer is now supposed to be ready to run using Python 3.
Remove the __future__ imports and change the interpreter line to
"#!/usr/bin/env python3".

Signed-off

image-fuzzer: Run using python3

image-fuzzer is now supposed to be ready to run using Python 3.
Remove the __future__ imports and change the interpreter line to
"#!/usr/bin/env python3".

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-10-ehabkost@redhat.com
Message-Id: <20191016192430.25098-10-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

image-fuzzer: Use io.StringIO

StringIO.StringIO is not available on Python 3, but io.StringIO
is available on both Python 2 and 3. io.StringIO is slightly
different from the Python 2 StringIO modul

image-fuzzer: Use io.StringIO

StringIO.StringIO is not available on Python 3, but io.StringIO
is available on both Python 2 and 3. io.StringIO is slightly
different from the Python 2 StringIO module, though, so we need
bytes coming from subprocess.Popen() to be explicitly decoded.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20191016192430.25098-5-ehabkost@redhat.com
Message-Id: <20191016192430.25098-5-ehabkost@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ehabkost/tags/python-next-pull-request' into staging

Python queue, 2018-06-11

* Make code compatible with Python 3 using 'futurize --stage1'
* Require Python >

Merge remote-tracking branch 'remotes/ehabkost/tags/python-next-pull-request' into staging

Python queue, 2018-06-11

* Make code compatible with Python 3 using 'futurize --stage1'
* Require Python >= 2.7 and remove Python 2.6 compatibility
modules

# gpg: Signature made Mon 11 Jun 2018 18:41:26 BST
# gpg: using RSA key 2807936F984DC5A6
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>"
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6

* remotes/ehabkost/tags/python-next-pull-request:
python: Remove scripts/ordereddict.py
python: Remove scripts/argparse.py
configure: Require Python 2.7 or newer
python: futurize -f lib2to3.fixes.fix_numliterals
python: futurize -f lib2to3.fixes.fix_except
python: futurize -f lib2to3.fixes.fix_renames
python: futurize -f lib2to3.fixes.fix_tuple_params
python: futurize -f lib2to3.fixes.fix_reduce
python: futurize -f lib2to3.fixes.fix_standarderror
python: futurize -f lib2to3.fixes.fix_has_key
python: futurize -f libfuturize.fixes.fix_next_call
python: futurize -f libfuturize.fixes.fix_absolute_import
python: futurize -f libfuturize.fixes.fix_print_with_import

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

python: futurize -f lib2to3.fixes.fix_renames

Change sys.maxint to sys.maxsize.

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -

python: futurize -f lib2to3.fixes.fix_renames

Change sys.maxint to sys.maxsize.

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \
sort -u | grep -v README.sh4)
$ futurize -w -f lib2to3.fixes.fix_renames $py

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20180608122952.2009-9-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

python: futurize -f libfuturize.fixes.fix_next_call

Change obj.next() calls to next(obj).

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^#!.*python';find -nam

python: futurize -f libfuturize.fixes.fix_next_call

Change obj.next() calls to next(obj).

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \
sort -u | grep -v README.sh4)
$ futurize -w -f libfuturize.fixes.fix_next_call $py

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20180608122952.2009-4-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

python: futurize -f libfuturize.fixes.fix_print_with_import

Change all Python code to use print as a function.

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^

python: futurize -f libfuturize.fixes.fix_print_with_import

Change all Python code to use print as a function.

This is necessary for Python 3 compatibility.

Done using:

$ py=$( (g grep -l -E '^#!.*python';find -name '*.py' -printf '%P\n';) | \
sort -u | grep -v README.sh4)
$ futurize -w -f libfuturize.fixes.fix_print_with_import $py

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20180608122952.2009-2-ehabkost@redhat.com>
[ehabkost: fixup tests/docker/docker.py]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-02-09' into staging

QAPI patches for 2016-02-09

# gpg: Signature made Tue 09 Feb 2016 10:55:51 GMT using RSA key ID EB918653
# gpg:

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-02-09' into staging

QAPI patches for 2016-02-09

# gpg: Signature made Tue 09 Feb 2016 10:55:51 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2016-02-09: (31 commits)
qapi: Add missing JSON files in build dependencies
qapi: Fix compilation failure on MIPS and SPARC
qmp: Don't abuse stack to track qmp-output root
qmp: Fix reference-counting of qnull on empty output visit
qapi: Drop unused error argument for list and implicit struct
qapi: Tighten qmp_input_end_list()
qapi: Drop unused 'kind' for struct/enum visit
qapi: Swap 'name' in visit_* callbacks to match public API
qom: Swap 'name' next to visitor in ObjectPropertyAccessor
qapi: Swap visit_* arguments for consistent 'name' placement
qom: Use typedef for Visitor
qapi: Don't cast Enum* to int*
qapi: Consolidate visitor small integer callbacks
qapi: Make all visitors supply uint64 callbacks
qapi: Prefer type_int64 over type_int in visitors
qapi-visit: Kill unused visit_end_union()
qapi: Track all failures between visit_start/stop
qapi: Improve generated event use of qapi visitor
balloon: Improve use of qapi visitor
vl: Ensure qapi visitor properly ends struct visit
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

tests: Use Python 2.6 "except E as ..." syntax

PEP 8 calls for it, because it's forward compatible with Python 3.
Supported since Python 2.6, which we require (commit fec2103).

Signed-off-by: Marku

tests: Use Python 2.6 "except E as ..." syntax

PEP 8 calls for it, because it's forward compatible with Python 3.
Supported since Python 2.6, which we require (commit fec2103).

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <1450425164-24969-5-git-send-email-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/mjt/tags/pull-trivial-patches-2015-09-11' into staging

trivial patches for 2015-09-11

# gpg: Signature made Fri 11 Sep 2015 12:02:43 BST using RSA key ID A4C3D

Merge remote-tracking branch 'remotes/mjt/tags/pull-trivial-patches-2015-09-11' into staging

trivial patches for 2015-09-11

# gpg: Signature made Fri 11 Sep 2015 12:02:43 BST using RSA key ID A4C3D7DB
# gpg: Good signature from "Michael Tokarev <mjt@tls.msk.ru>"
# gpg: aka "Michael Tokarev <mjt@corpit.ru>"
# gpg: aka "Michael Tokarev <mjt@debian.org>"

* remotes/mjt/tags/pull-trivial-patches-2015-09-11: (26 commits)
virtio-vga: enable for i386
hw/arm/spitz: Remove meaningless blank Property
hw/gpio/zaurus: Remove meaningless blank Property
hw/virtio/virtio-pci: Remove meaningless blank Property
hw/s390x/s390-virtio-bus: Remove meaningless blank Property
typofixes - v4
qapi-schema: remove legacy<> from doc
disas/microblaze: Remove unused code
help: dd missing newline
Target-ppc: Remove unnecessary variable
baum: Fix build with debugging enabled
linux-user: Fix warnings caused by missing 'static' attribute
opts: produce valid command line in qemu_opts_print
docs: fix a qga/qapi-schema.json comment
trivial: remove trailing newline from error_report
maint: avoid useless "if (foo) free(foo)" pattern
maint: avoid useless "if (foo) free(foo)" pattern
maint: remove unused include for strings.h
maint: remove unused include for signal.h
maint: remove unused include for dirent.h
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

typofixes - v4

Signed-off-by: Veres Lajos <vlajos@gmail.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

# gpg: Signature made Mon 22 Sep 2014 12:41:59 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan H

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

# gpg: Signature made Mon 22 Sep 2014 12:41:59 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/block-pull-request: (59 commits)
block: Always compile virtio-blk dataplane
vring: Better error handling if num is too large
virtio: Import virtio_vring.h
async: aio_context_new(): Handle event_notifier_init failure
block: vhdx - fix reading beyond pointer during image creation
block: delete cow block driver
block/archipelago: Fix typo in qemu_archipelago_truncate()
ahci: Add test_identify case to ahci-test.
ahci: Add test_hba_enable to ahci-test.
ahci: Add test_hba_spec to ahci-test.
ahci: properly shadow the TFD register
ahci: add test_pci_enable to ahci-test.
ahci: Add test_pci_spec to ahci-test.
ahci: MSI capability should be at 0x80, not 0x50.
ahci: Adding basic functionality qtest.
layout: Add generators for refcount table and blocks
fuzz: Add fuzzing functions for entries of refcount table and blocks
docs: List all image elements currently supported by the fuzzer
qapi/block-core: Add "new" qcow2 options
qcow2: Add overlap-check.template option
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

block: delete cow block driver

This patch removes support for the cow file format.

Normally we do not break backwards compatibility but in this case there
is no impact and it is the most logical op

block: delete cow block driver

This patch removes support for the cow file format.

Normally we do not break backwards compatibility but in this case there
is no impact and it is the most logical option. Extraordinary claims
require extraordinary evidence so I will show why removing the cow block
driver is the right thing to do.

The cow file format is the disk image format for Usermode Linux, a way
of running a Linux system in userspace. The performance of UML was
never great and it was hacky, but it enjoyed some popularity before
hardware virtualization support became mainstream.

QEMU's block/cow.c is supposed to read this image file format.
Unfortunately the file format was underspecified:

1. Earlier Linux versions used the MAXPATHLEN constant for the backing
filename field. The value of MAXPATHLEN can change, so Linux
switched to a 4096 literal but QEMU has a 1024 literal.

2. Padding was not used on the header struct (both in the Linux kernel
and in QEMU) so the struct layout varied across architectures. In
particular, i386 and x86_64 were different due to int64_t alignment
differences. Linux now uses __attribute__((packed)), QEMU does not.

Therefore:

1. QEMU cow images do not conform to the Linux cow image file format.

2. cow images cannot be shared between different host architectures.

This means QEMU cow images are useless and QEMU has not had bug reports
from users actually hitting these issues.

Let's get rid of this thing, it serves no purpose and no one will be
affected.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 1410877464-20481-1-git-send-email-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

image-fuzzer: Trivial readability and formatting improvements

Signed-off-by: Maria Kustova <maria.k@catit.be>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Fri 22 Aug 2014 14:47:53 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Ke

Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging

Block patches

# gpg: Signature made Fri 22 Aug 2014 14:47:53 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (29 commits)
qemu-img: Allow cache mode specification for amend
qemu-img: Allow source cache mode specification
vmdk: Use bdrv_nb_sectors() where sectors, not bytes are wanted
blkdebug: Delete BH in bdrv_aio_cancel
qemu-iotests: add test case 101 for short file I/O
raw-posix: fix O_DIRECT short reads
block/iscsi: fix memory corruption on iscsi resize
block/vvfat.c: remove debugging code to reinit stderr if NULL
iotests: Add test for image filename construction
quorum: Implement bdrv_refresh_filename()
nbd: Implement bdrv_refresh_filename()
blkverify: Implement bdrv_refresh_filename()
blkdebug: Implement bdrv_refresh_filename()
block: Add bdrv_refresh_filename()
virtio-blk: fix reference a pointer which might be freed
virtio-blk: allow block_resize with dataplane
block: acquire AioContext in qmp_block_resize()
qemu-iotests: Fix 028 reference output for qed
test-coroutine: test cost introduced by coroutine
iotests: Add test for qcow2's cache options
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

runner: Kill a program under test by time-out

If a program under test get frozen, the test should finish and report about its
failure.
In such cases the runner waits for 10 minutes until the program

runner: Kill a program under test by time-out

If a program under test get frozen, the test should finish and report about its
failure.
In such cases the runner waits for 10 minutes until the program ends its
execution. After this time-out the program will be terminated and the test will
be marked as failed.

For current limitation of test image size to 10 MB as a maximum an execution of
each command takes about several seconds in general, so 10 minutes is enough to
discriminate freeze, but not drastically increase an overall test duration.

Signed-off-by: Maria Kustova <maria.k@catit.be>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

runner: Add an argument for test duration

After the specified duration the runner stops executing new tests, but it
doesn't interrupt running ones.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-o

runner: Add an argument for test duration

After the specified duration the runner stops executing new tests, but it
doesn't interrupt running ones.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Maria Kustova <maria.k@catit.be>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Fri 15 Aug 2014 18:04:23 BST using RSA key ID 81AB73C8
# gpg: Good sign

Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging

Block pull request

# gpg: Signature made Fri 15 Aug 2014 18:04:23 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/block-pull-request: (55 commits)
qcow2: fix new_blocks double-free in alloc_refcount_block()
image-fuzzer: Reduce number of generator functions in __init__
image-fuzzer: Add generators of L1/L2 tables
image-fuzzer: Add fuzzing functions for L1/L2 table entries
docs: Expand the list of supported image elements with L1/L2 tables
image-fuzzer: Public API for image-fuzzer/runner/runner.py
image-fuzzer: Generator of fuzzed qcow2 images
image-fuzzer: Fuzzing functions for qcow2 images
image-fuzzer: Tool for fuzz tests execution
docs: Specification for the image fuzzer
ide: only constrain read/write requests to drive size, not other types
virtio-blk: Correct bug in support for flexible descriptor layout
libqos: Change free function called in malloc
libqos: Correct mask to align size to PAGE_SIZE in malloc-pc
libqtest: add QTEST_LOG for debugging qtest testcases
ide: Fix segfault when flushing a device that doesn't exist
qemu-options: add missing -drive discard option to cmdline help
parallels: 2TB+ parallels images support
parallels: split check for parallels format in parallels_open
parallels: replace tabs with spaces in block/parallels.c
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

image-fuzzer: Tool for fuzz tests execution

The purpose of the test runner is to prepare the test environment (e.g. create
a work directory, a test image, etc), execute a program under test with
par

image-fuzzer: Tool for fuzz tests execution

The purpose of the test runner is to prepare the test environment (e.g. create
a work directory, a test image, etc), execute a program under test with
parameters, indicate a test failure if the program was killed during the test
execution and collect core dumps, logs and other test artifacts.

The test runner doesn't depend on an image format, so it can be used with any
external image generator.

[Fixed path to qcow2 format module "qcow2" instead of "../qcow2" since
runner.py is no longer in a sub-directory.
--Stefan]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Maria Kustova <maria.k@catit.be>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...