scripts: drop comment about autogenerated CPU API file

The RST doc include can't be made to skip the comment indicating the CPU
CSV file is auto-generated when importing it. This comment line was
pr

scripts: drop comment about autogenerated CPU API file

The RST doc include can't be made to skip the comment indicating the CPU
CSV file is auto-generated when importing it. This comment line was
previously manually removed from the generated output that was committed.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

scripts/clean-includes: Update exclude list

Update the exclude list to exclude some more files which don't follow our
standard #include policy.

Signed-off-by: Peter Maydell <peter.maydell@linaro.or

scripts/clean-includes: Update exclude list

Update the exclude list to exclude some more files which don't follow our
standard #include policy.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>

show more ...

analyze-migration.py: Remove trick on parsing ramblocks

RAM_SAVE_FLAG_MEM_SIZE contains the total length of ramblock idstr to know
whether scanning of ramblocks is complete. Drop the trick.

Review

analyze-migration.py: Remove trick on parsing ramblocks

RAM_SAVE_FLAG_MEM_SIZE contains the total length of ramblock idstr to know
whether scanning of ramblocks is complete. Drop the trick.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240117075848.139045-4-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>

show more ...

qapi: Fix dangling references to docs/devel/qapi-code-gen.txt

Conversion of docs/devel/qapi-code-gen.txt to ReST left several
dangling references behind. Fix them to point to
docs/devel/qapi-code-g

qapi: Fix dangling references to docs/devel/qapi-code-gen.txt

Conversion of docs/devel/qapi-code-gen.txt to ReST left several
dangling references behind. Fix them to point to
docs/devel/qapi-code-gen.rst.

Fixes: f7aa076dbdfc (docs: convert qapi-code-gen.txt to ReST)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240120095327.666239-4-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

Merge tag 'pull-testing-updates-120124-2' of https://gitlab.com/stsquad/qemu into staging

testing and misc updates

- add LE microblaze test to avocado
- use modern snapshot=on to avoid trashing

Merge tag 'pull-testing-updates-120124-2' of https://gitlab.com/stsquad/qemu into staging

testing and misc updates

- add LE microblaze test to avocado
- use modern snapshot=on to avoid trashing disk image
- use plain bool for fe_is_open
- various updates to qtest timeouts
- enable meson test timeouts
- tweak the readthedocs environment
- partially revert un-flaking x86_64

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmWhPccACgkQ+9DbCVqe
# KkS5agf+OoW6HOitt34YeL6cGGtIKaxbta+Fs6jq+ucIbN63TmLTuKrmPiRNxjuo
# Fj2Qvh9R7Tl7Q/a7ZAym0Fze7GtsvvsidkiQS4pmi9vYuJrhS734CxXHT8JS6zJr
# ymQ0nGZODg1cVB4oAR9sXo/OwEQdDTSgKp8wdNr930fxYwokUKBUgcOqElu3SWHv
# duSYDuaflnP5B8ZGbb1ZnOlwS9lZIHTwjZyN5J1YtxF0T8Ez4A+xseEOpQ/00MoE
# Ecjdp3ELCxzOI+1U33Yni7ol//fxQpRKi+xf2fGIxhuSA3i32rmY5NWTvl7VwuS1
# gXryjX2rukSujySP3vkdtTp0dmkbpg==
# =ZuDd
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 12 Jan 2024 13:25:27 GMT
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44

* tag 'pull-testing-updates-120124-2' of https://gitlab.com/stsquad/qemu: (22 commits)
tests/avocado: partially revert unmasking of replay_linux tests
readthodocs: fully specify a build environment
mtest2make: stop disabling meson test timeouts
tests/fp: Bump fp-test-mulAdd test timeout to 3 minutes
tests/unit: Bump test-crypto-block test timeout to 5 minutes
tests/unit: Bump test-aio-multithread test timeout to 2 minutes
tests/qtest: Bump the device-introspect-test timeout to 12 minutes
qtest: bump bios-table-test timeout to 9 minutes
qtest: bump aspeed_smc-test timeout to 6 minutes
qtest: bump qos-test timeout to 2 minutes
qtest: bump boot-serial-test timeout to 3 minutes
qtest: bump prom-env-test timeout to 6 minutes
qtest: bump pxe-test timeout to 10 minutes
qtest: bump test-hmp timeout to 4 minutes
qtest: bump npcm7xx_pwm-test timeout to 5 minutes
qtest: bump qom-test timeout to 15 minutes
qtest: bump migration-test timeout to 8 minutes
qtest: bump min meson timeout to 60 seconds
chardev: use bool for fe_is_open
gitlab: include microblazeel in testing
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab

mtest2make: stop disabling meson test timeouts

The mtest2make.py script passes the arg '-t 0' to 'meson test' which
disables all test timeouts. This is a major source of pain when running
in GitLab CI and a test gets stuck. It will stall until GitLab kills the
CI job. This leaves us with little easily consumable information about
the stalled test. The TAP format doesn't show the test name until it is
completed, and TAP output from multiple tests it interleaved. So we
have to analyse the log to figure out what tests had un-finished TAP
output present and thus infer which test case caused the hang. This is
very time consuming and error prone.

By allowing meson to kill stalled tests, we get a direct display of what
test program got stuck, which lets us more directly focus in on what
specific test case within the test program hung.

The other issue with disabling meson test timeouts by default is that it
makes it more likely that maintainers inadvertantly introduce slowdowns.
For example the recent-ish change that accidentally made migrate-test
take 15-20 minutes instead of around 1 minute.

The main risk of this change is that the individual test timeouts might
be too short to allow completion in high load scenarios. Thus, there is
likely to be some short term pain where we have to bump the timeouts for
certain tests to make them reliable enough. The preceeding few patches
raised the timeouts for all failures that were immediately apparent
in GitLab CI.

Even with the possible short term instability, this should still be a
net win for debuggability of failed CI pipelines over the long term.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230717182859.707658-13-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20231215070357.10888-17-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

scripts/checkpatch: Support codespell checking

Add two spelling check options (--codespell and --codespellfile) to
enhance spelling check through dictionary, which copied the Linux
kernel's implemen

scripts/checkpatch: Support codespell checking

Add two spelling check options (--codespell and --codespellfile) to
enhance spelling check through dictionary, which copied the Linux
kernel's implementation in checkpatch.pl.

This check uses the dictionary at "/usr/share/codespell/dictionary.txt"
by default, if there is no dictionary specified under this path, it
will look for the dictionary of python3's codespell (This requires user
to add python3's path in environment variable $PATH, and to install
codespell by "pip install codespell").

Tested-by: Yongwei Ma <yongwei.ma@intel.com>
Tested-by: Samuel Tardieu <sam@rfc1149.net>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
Message-ID: <20240105083848.267192-1-zhao1.liu@linux.intel.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

linux-headers: riscv: add ptrace.h

KVM vector support for RISC-V requires the linux-header ptrace.h.

Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
Acked-by: Alistair Francis <a

linux-headers: riscv: add ptrace.h

KVM vector support for RISC-V requires the linux-header ptrace.h.

Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
Acked-by: Alistair Francis <alistair.francis@wdc.com>
Message-ID: <20231218204321.75757-3-dbarboza@ventanamicro.com>
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>

show more ...

scripts/replay_dump: track total number of instructions

This will help in tracking where we are in the stream when debugging.

Reviewed-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: P

scripts/replay_dump: track total number of instructions

This will help in tracking where we are in the stream when debugging.

Reviewed-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231211091346.14616-6-alex.bennee@linaro.org>

show more ...

scripts/replay-dump: update to latest format

To help debugging replay logs I've implemented decode_plain and
decode_char_write as well as put in a new table for the current format
of log.

Reviewed-

scripts/replay-dump: update to latest format

To help debugging replay logs I've implemented decode_plain and
decode_char_write as well as put in a new table for the current format
of log.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231211091346.14616-5-alex.bennee@linaro.org>

show more ...

block-coroutine-wrapper: use qemu_get_current_aio_context()

Use qemu_get_current_aio_context() in mixed wrappers and coroutine
wrappers so that code runs in the caller's AioContext instead of moving

block-coroutine-wrapper: use qemu_get_current_aio_context()

Use qemu_get_current_aio_context() in mixed wrappers and coroutine
wrappers so that code runs in the caller's AioContext instead of moving
to the BlockDriverState's AioContext. This change is necessary for the
multi-queue block layer where any thread can call into the block layer.

Most wrappers are IO_CODE where it's safe to use the current AioContext
nowadays. BlockDrivers and the core block layer use their own locks and
no longer depend on the AioContext lock for thread-safety.

The bdrv_create() wrapper invokes GLOBAL_STATE code. Using the current
AioContext is safe because this code is only called with the BQL held
from the main loop thread.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230912231037.826804-6-stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: remove AioContext locking

This is the big patch that removes
aio_context_acquire()/aio_context_release() from the block layer and
affected block layer users.

There isn't a clean way to split

block: remove AioContext locking

This is the big patch that removes
aio_context_acquire()/aio_context_release() from the block layer and
affected block layer users.

There isn't a clean way to split this patch and the reviewers are likely
the same group of people, so I decided to do it in one patch.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Paul Durrant <paul@xen.org>
Message-ID: <20231205182011.1976568-7-stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

graph-lock: remove AioContext locking

Stop acquiring/releasing the AioContext lock in
bdrv_graph_wrlock()/bdrv_graph_unlock() since the lock no longer has any
effect.

The distinction between bdrv_g

graph-lock: remove AioContext locking

Stop acquiring/releasing the AioContext lock in
bdrv_graph_wrlock()/bdrv_graph_unlock() since the lock no longer has any
effect.

The distinction between bdrv_graph_wrunlock() and
bdrv_graph_wrunlock_ctx() becomes meaningless and they can be collapsed
into one function.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231205182011.1976568-6-stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

tests/qtest/migration-test: Fix analyze-migration.py for s390x

The migration stream on s390x contains data for the storage_attributes
which the analyze-migration.py cannot handle yet. Add the basic

tests/qtest/migration-test: Fix analyze-migration.py for s390x

The migration stream on s390x contains data for the storage_attributes
which the analyze-migration.py cannot handle yet. Add the basic code
for handling this, so we can re-enable the check in the migration-test.

Message-ID: <20231120113951.162090-1-thuth@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

scripts: adjust url to Coverity tools

The URL to the Coverity tools download has changed; the old one points
to an obsolete version that is not supported anymore. Adjust to point
to the correct and

scripts: adjust url to Coverity tools

The URL to the Coverity tools download has changed; the old one points
to an obsolete version that is not supported anymore. Adjust to point
to the correct and supported tools.

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

coverity: physmem: use simple assertions instead of modelling

Unfortunately Coverity doesn't follow the logic aroung "len" and "l"
variables in stacks finishing with flatview_{read,write}_continue()

coverity: physmem: use simple assertions instead of modelling

Unfortunately Coverity doesn't follow the logic aroung "len" and "l"
variables in stacks finishing with flatview_{read,write}_continue() and
generate a lot of OVERRUN false-positives. When small buffer (2 or 4
bytes) is passed to mem read/write path, Coverity assumes the worst
case of sz=8 in stn_he_p()/ldn_he_p() (defined in
include/qemu/bswap.h), and reports buffer overrun.

To silence these false-positives we have model functions, which hide
real logic from Coverity.

However, it turned out that these new two assertions are enough to
quiet Coverity.

Assertions are better than hiding the logic, so let's drop the
modelling and move to assertions for memory r/w call stacks.

After patch, the sequence

cov-make-library --output-file /tmp/master.xmldb \
scripts/coverity-scan/model.c
cov-build --dir ~/covtmp/master make -j9
cov-analyze --user-model-file /tmp/master.xmldb \
--dir ~/covtmp/master --all --strip-path "$(pwd)
cov-format-errors --dir ~/covtmp/master \
--html-output ~/covtmp/master_html_report

Generate for me the same big set of CIDs excepept for 6 disappeared (so
it becomes even better).

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Acked-by: David Hildenbrand <david@redhat.com>
Message-ID: <20231005140326.332830-1-vsementsov@yandex-team.ru>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

block: Fix deadlocks in bdrv_graph_wrunlock()

bdrv_graph_wrunlock() calls aio_poll(), which may run callbacks that
have a nested event loop. Nested event loops can depend on other
iothreads making p

block: Fix deadlocks in bdrv_graph_wrunlock()

bdrv_graph_wrunlock() calls aio_poll(), which may run callbacks that
have a nested event loop. Nested event loops can depend on other
iothreads making progress, so in order to allow them to make progress it
must not hold the AioContext lock of another thread while calling
aio_poll().

This introduces a @bs parameter to bdrv_graph_wrunlock() whose
AioContext is temporarily dropped (which matches bdrv_graph_wrlock()),
and a bdrv_graph_wrunlock_ctx() that can be used if the BlockDriverState
doesn't necessarily exist any more when unlocking.

This also requires a change to bdrv_schedule_unref(), which was relying
on the incorrectly taken lock. It needs to take the lock itself now.
While this is a separate bug, it can't be fixed a separate patch because
otherwise the intermediate state would either deadlock or try to release
a lock that we don't even hold.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231115172012.112727-3-kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[kwolf: Fixed up bdrv_schedule_unref()]
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

qapi: Fix QAPISchemaEntity.__repr__()

I messed it up on merge. It's a debugging aid, so no impact on build.

Fixes: e307a8174bb8 (qapi: provide a friendly string representation of QAPI classes)
Sig

qapi: Fix QAPISchemaEntity.__repr__()

I messed it up on merge. It's a debugging aid, so no impact on build.

Fixes: e307a8174bb8 (qapi: provide a friendly string representation of QAPI classes)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20231024104841.1569250-1-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

show more ...

Merge tag 'tracing-pull-request' of https://gitlab.com/stefanha/qemu into staging

Pull request

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCAAdFiEEhpWov9P5fNqsNXdanKSrs4Grc8gFAmVMhGAACgkQnKSrs4Gr
#

Merge tag 'tracing-pull-request' of https://gitlab.com/stefanha/qemu into staging

Pull request

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCAAdFiEEhpWov9P5fNqsNXdanKSrs4Grc8gFAmVMhGAACgkQnKSrs4Gr
# c8iBoQgAgWjGzp++3Iik0kdMObRlXVpTWwrlYohPg+5NanQB1LifYMt/2MTB70OA
# TUcMcZwzHCUSrtqp+IwU9m5YBZlk+NTQIUn3j5r6Rdv3TnHElBJgpiNWRqpQoDo6
# ZPqSymoyOh2pWtyKQtPQWhx1SsUeF5dEsbOy7fqZi7wLEUqBM1+9LV9C/Y6pIAa7
# KRSnbyNPviyV5B4qX1gaH17DKhj6QyRF/secv8TkKFlfr/6GelNySxEj9+9Om7qO
# R1NT09ZjjMXnNqA9cRzz00fP7bTP36WF065jkYc1oGAo9VwhWAfysju1qbSN2whe
# 4CwucP9lSPuJ+g5tv+7nfX+JfY8+Gw==
# =jAvl
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 09 Nov 2023 15:04:00 HKT
# gpg: using RSA key 8695A8BFD3F97CDAAC35775A9CA4ABB381AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>" [ultimate]
# gpg: aka "Stefan Hajnoczi <stefanha@gmail.com>" [ultimate]
# Primary key fingerprint: 8695 A8BF D3F9 7CDA AC35 775A 9CA4 ABB3 81AB 73C8

* tag 'tracing-pull-request' of https://gitlab.com/stefanha/qemu:
tracetool: avoid invalid escape in Python string

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

tracetool: avoid invalid escape in Python string

This is an error in Python 3.12; fix it by using a raw string literal.

Cc: <qemu-stable@nongnu.org>
Signed-off-by: Marc-André Lureau <marcandre.lur

tracetool: avoid invalid escape in Python string

This is an error in Python 3.12; fix it by using a raw string literal.

Cc: <qemu-stable@nongnu.org>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20231108105649.60453-1-marcandre.lureau@redhat.com>

show more ...

gdbstub: Add num_regs member to GDBFeature

Currently the number of registers exposed to GDB is written as magic
numbers in code. Derive the number of registers GDB actually see from
XML files to rep

gdbstub: Add num_regs member to GDBFeature

Currently the number of registers exposed to GDB is written as magic
numbers in code. Derive the number of registers GDB actually see from
XML files to replace the magic numbers in code later.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231025093128.33116-2-akihiko.odaki@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231106185112.2755262-9-alex.bennee@linaro.org>

show more ...

scripts/cpu-x86-uarch-abi.py: Fix parameter error of cmd

When run this script, there's the error:

python3 scripts/cpu-x86-uarch-abi.py /tmp/qmp
Traceback (most recent call last):
File "/path-to-q

scripts/cpu-x86-uarch-abi.py: Fix parameter error of cmd

When run this script, there's the error:

python3 scripts/cpu-x86-uarch-abi.py /tmp/qmp
Traceback (most recent call last):
File "/path-to-qemu/qemu/scripts/cpu-x86-uarch-abi.py", line 96, in <module>
cpu = shell.cmd("query-cpu-model-expansion",
TypeError: QEMUMonitorProtocol.cmd() takes 2 positional arguments but 3 were given

Commit 7f521b023bc28 ("scripts/cpu-x86-uarch-abi.py: use .command()
instead of .cmd()") converts the the original .cmd() to .command()
(which was later renamed to "cmd" to replace the original one).

But the new .cmd() only accepts typing.Mapping as the parameter instead
of typing.Dict (see _qmp.execute()).

Change the paremeters of "query-cpu-model-expansion" to typing.Mapping
format to fix this error.

Fixes: 7f521b023bc28 ("scripts/cpu-x86-uarch-abi.py: use .command() instead of .cmd()")

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

build-sys: add a "pixman" feature

For now, pixman is mandatory, but we set config_host.h and Kconfig.
Once compilation is fixed, "pixman" will become actually optional.

Signed-off-by: Marc-André Lu

build-sys: add a "pixman" feature

For now, pixman is mandatory, but we set config_host.h and Kconfig.
Once compilation is fixed, "pixman" will become actually optional.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>

show more ...

Add Hyper-V Dynamic Memory Protocol driver (hv-balloon) base

This driver is like virtio-balloon on steroids: it allows both changing the
guest memory allocation via ballooning and (in the next patch

Add Hyper-V Dynamic Memory Protocol driver (hv-balloon) base

This driver is like virtio-balloon on steroids: it allows both changing the
guest memory allocation via ballooning and (in the next patch) inserting
pieces of extra RAM into it on demand from a provided memory backend.

The actual resizing is done via ballooning interface (for example, via
the "balloon" HMP command).
This includes resizing the guest past its boot size - that is, hot-adding
additional memory in granularity limited only by the guest alignment
requirements, as provided by the next patch.

In contrast with ACPI DIMM hotplug where one can only request to unplug a
whole DIMM stick this driver allows removing memory from guest in single
page (4k) units via ballooning.

After a VM reboot the guest is back to its original (boot) size.

In the future, the guest boot memory size might be changed on reboot
instead, taking into account the effective size that VM had before that
reboot (much like Hyper-V does).

For performance reasons, the guest-released memory is tracked in a few
range trees, as a series of (start, count) ranges.
Each time a new page range is inserted into such tree its neighbors are
checked as candidates for possible merging with it.

Besides performance reasons, the Dynamic Memory protocol itself uses page
ranges as the data structure in its messages, so relevant pages need to be
merged into such ranges anyway.

One has to be careful when tracking the guest-released pages, since the
guest can maliciously report returning pages outside its current address
space, which later clash with the address range of newly added memory.
Similarly, the guest can report freeing the same page twice.

The above design results in much better ballooning performance than when
using virtio-balloon with the same guest: 230 GB / minute with this driver
versus 70 GB / minute with virtio-balloon.

During a ballooning operation most of time is spent waiting for the guest
to come up with newly freed page ranges, processing the received ranges on
the host side (in QEMU and KVM) is nearly instantaneous.

The unballoon operation is also pretty much instantaneous:
thanks to the merging of the ballooned out page ranges 200 GB of memory can
be returned to the guest in about 1 second.
With virtio-balloon this operation takes about 2.5 minutes.

These tests were done against a Windows Server 2019 guest running on a
Xeon E5-2699, after dirtying the whole memory inside guest before each
balloon operation.

Using a range tree instead of a bitmap to track the removed memory also
means that the solution scales well with the guest size: even a 1 TB range
takes just a few bytes of such metadata.

Since the required GTree operations aren't present in every Glib version
a check for them was added to the meson build script, together with new
"--enable-hv-balloon" and "--disable-hv-balloon" configure arguments.
If these GTree operations are missing in the system's Glib version this
driver will be skipped during QEMU build.

An optional "status-report=on" device parameter requests memory status
events from the guest (typically sent every second), which allow the host
to learn both the guest memory available and the guest memory in use
counts.

Following commits will add support for their external emission as
"HV_BALLOON_STATUS_REPORT" QMP events.

The driver is named hv-balloon since the Linux kernel client driver for
the Dynamic Memory Protocol is named as such and to follow the naming
pattern established by the virtio-balloon driver.
The whole protocol runs over Hyper-V VMBus.

The driver was tested against Windows Server 2012 R2, Windows Server 2016
and Windows Server 2019 guests and obeys the guest alignment requirements
reported to the host via DM_CAPABILITIES_REPORT message.

Acked-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>

show more ...