migration: Allow migrate-recover to run multiple times

Previously migration didn't have an easy way to cleanup the listening
transport, migrate recovery only allows to execute once. That's done wit

migration: Allow migrate-recover to run multiple times

Previously migration didn't have an easy way to cleanup the listening
transport, migrate recovery only allows to execute once. That's done with a
trick flag in postcopy_recover_triggered.

Now the facility is already there.

Drop postcopy_recover_triggered and instead allows a new migrate-recover to
release the previous listener transport.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220331150857.74406-8-peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream' into staging

Block layer patches

- qemu-storage-daemon: Add --daemonize
- Fix x-blockdev-amend and block node activation code w

Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream' into staging

Block layer patches

- qemu-storage-daemon: Add --daemonize
- Fix x-blockdev-amend and block node activation code which incorrectly
executed code in the iothread that must run in the main thread.
- Add macros for coroutine-safe TLS variables (required for correctness
with LTO)
- Fix crashes with concurrent I/O and bdrv_refresh_limits()
- Split block APIs in global state and I/O
- iotests: Don't refuse to run at all without GNU sed, just skip tests
that need it

# gpg: Signature made Fri 04 Mar 2022 17:18:31 GMT
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6

* remotes/kwolf-gitlab/tags/for-upstream: (50 commits)
block/amend: Keep strong reference to BDS
block/amend: Always call .bdrv_amend_clean()
tests/qemu-iotests: Rework the checks and spots using GNU sed
iotests/graph-changes-while-io: New test
iotests: Allow using QMP with the QSD
block: Make bdrv_refresh_limits() non-recursive
job.h: assertions in the callers of JobDriver function pointers
job.h: split function pointers in JobDriver
block-backend-common.h: split function pointers in BlockDevOps
block_int-common.h: assertions in the callers of BdrvChildClass function pointers
block_int-common.h: split function pointers in BdrvChildClass
block_int-common.h: assertions in the callers of BlockDriver function pointers
block_int-common.h: split function pointers in BlockDriver
block/coroutines: I/O and "I/O or GS" API
block/copy-before-write.h: global state API + assertions
include/block/snapshot: global state API + assertions
assertions for blockdev.h global state API
include/sysemu/blockdev.h: global state API
assertions for blockjob.h global state API
include/block/blockjob.h: global state API
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

include/block/snapshot: global state API + assertions

Snapshots run also under the BQL, so they all are
in the global state API. The aiocontext lock that they hold
is currently an overkill and in fu

include/block/snapshot: global state API + assertions

Snapshots run also under the BQL, so they all are
in the global state API. The aiocontext lock that they hold
is currently an overkill and in future could be removed.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Message-Id: <20220303151616.325444-23-eesposit@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

block: rename bdrv_invalidate_cache_all, blk_invalidate_cache and test_sync_op_invalidate_cache

Following the bdrv_activate renaming, change also the name
of the respective callers.

bdrv_invalidate

block: rename bdrv_invalidate_cache_all, blk_invalidate_cache and test_sync_op_invalidate_cache

Following the bdrv_activate renaming, change also the name
of the respective callers.

bdrv_invalidate_cache_all -> bdrv_activate_all
blk_invalidate_cache -> blk_activate
test_sync_op_invalidate_cache -> test_sync_op_activate

No functional change intended.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220209105452.1694545-5-eesposit@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20220302b' into staging

Migration/HMP/Virtio pull 2022-03-02

A bit of a mix this time:
* Minor fixes from myself, Hanna,

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20220302b' into staging

Migration/HMP/Virtio pull 2022-03-02

A bit of a mix this time:
* Minor fixes from myself, Hanna, and Jack
* VNC password rework by Stefan and Fabian
* Postcopy changes from Peter X that are
the start of a larger series to come
* Removing the prehistoic load_state_old
code from Peter M

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# gpg: Signature made Wed 02 Mar 2022 18:25:12 GMT
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert-gitlab/tags/pull-migration-20220302b:
migration: Remove load_state_old and minimum_version_id_old
tests: Pass in MigrateStart** into test_migrate_start()
migration: Add migration_incoming_transport_cleanup()
migration: postcopy_pause_fault_thread() never fails
migration: Enlarge postcopy recovery to capture !-EIO too
migration: Move static var in ram_block_from_stream() into global
migration: Add postcopy_thread_create()
migration: Dump ramblock and offset too when non-same-page detected
migration: Introduce postcopy channels on dest node
migration: Tracepoint change in postcopy-run bottom half
migration: Finer grained tracepoints for POSTCOPY_LISTEN
migration: Dump sub-cmd name in loadvm_process_command tp
migration/rdma: set the REUSEADDR option for destination
qapi/monitor: allow VNC display id in set/expire_password
qapi/monitor: refactor set/expire_password with enums
monitor/hmp: add support for flag argument with value
virtiofsd: Let meson check for statx.stx_mnt_id
clock-vmstate: Add missing END_OF_LIST

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Add postcopy_thread_create()

Postcopy create threads. A common manner is we init a sem and use it to sync
with the thread. Namely, we have fault_thread_sem and listen_thread_sem and
they

migration: Add postcopy_thread_create()

Postcopy create threads. A common manner is we init a sem and use it to sync
with the thread. Namely, we have fault_thread_sem and listen_thread_sem and
they're only used for this.

Make it a shared infrastructure so it's easier to create yet another thread.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220301083925.33483-7-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: Introduce postcopy channels on dest node

Postcopy handles huge pages in a special way that currently we can only have
one "channel" to transfer the page.

It's because when we install pag

migration: Introduce postcopy channels on dest node

Postcopy handles huge pages in a special way that currently we can only have
one "channel" to transfer the page.

It's because when we install pages using UFFDIO_COPY, we need to have the whole
huge page ready, it also means we need to have a temp huge page when trying to
receive the whole content of the page.

Currently all maintainance around this tmp page is global: firstly we'll
allocate a temp huge page, then we maintain its status mostly within
ram_load_postcopy().

To enable multiple channels for postcopy, the first thing we need to do is to
prepare N temp huge pages as caching, one for each channel.

Meanwhile we need to maintain the tmp huge page status per-channel too.

To give some example, some local variables maintained in ram_load_postcopy()
are listed; they are responsible for maintaining temp huge page status:

- all_zero: this keeps whether this huge page contains all zeros
- target_pages: this counts how many target pages have been copied
- host_page: this keeps the host ptr for the page to install

Move all these fields to be together with the temp huge pages to form a new
structure called PostcopyTmpPage. Then for each (future) postcopy channel, we
need one structure to keep the state around.

For vanilla postcopy, obviously there's only one channel. It contains both
precopy and postcopy pages.

This patch teaches the dest migration node to start realize the possible number
of postcopy channels by introducing the "postcopy_channels" variable. Its
value is calculated when setup postcopy on dest node (during POSTCOPY_LISTEN
phase).

Vanilla postcopy will have channels=1, but when postcopy-preempt capability is
enabled (in the future), we will boost it to 2 because even during partial
sending of a precopy huge page we still want to preempt it and start sending
the postcopy requested page right away (so we start to keep two temp huge
pages; more if we want to enable multifd). In this patch there's a TODO marked
for that; so far the channels is always set to 1.

We need to send one "host huge page" on one channel only and we cannot split
them, because otherwise the data upon the same huge page can locate on more
than one channel so we need more complicated logic to manage. One temp host
huge page for each channel will be enough for us for now.

Postcopy will still always use the index=0 huge page even after this patch.
However it prepares for the latter patches where it can start to use multiple
channels (which needs src intervention, because only src knows which channel we
should use).

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220301083925.33483-5-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
dgilbert: Fixed up long line

show more ...

migration: Tracepoint change in postcopy-run bottom half

Remove the old two tracepoints and they're even near each other:

trace_loadvm_postcopy_handle_run_cpu_sync()
trace_loadvm_postcopy_h

migration: Tracepoint change in postcopy-run bottom half

Remove the old two tracepoints and they're even near each other:

trace_loadvm_postcopy_handle_run_cpu_sync()
trace_loadvm_postcopy_handle_run_vmstart()

Add trace_loadvm_postcopy_handle_run_bh() with a finer granule trace.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220301083925.33483-4-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: Finer grained tracepoints for POSTCOPY_LISTEN

The enablement of postcopy listening has a few steps, add a few tracepoints to
be there ready for some basic measurements for them.

Reviewed

migration: Finer grained tracepoints for POSTCOPY_LISTEN

The enablement of postcopy listening has a few steps, add a few tracepoints to
be there ready for some basic measurements for them.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220301083925.33483-3-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: Dump sub-cmd name in loadvm_process_command tp

It'll be easier to read the name rather than index of sub-cmd when debugging.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Sig

migration: Dump sub-cmd name in loadvm_process_command tp

It'll be easier to read the name rather than index of sub-cmd when debugging.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20220301083925.33483-2-peterx@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/quintela-gitlab/tags/migration-20220128-pull-request' into staging

Migration Pull request (Take 2)

Hi

This time I have disabled vmstate canary patches form Da

Merge remote-tracking branch 'remotes/quintela-gitlab/tags/migration-20220128-pull-request' into staging

Migration Pull request (Take 2)

Hi

This time I have disabled vmstate canary patches form Dave Gilbert.

Let's see if it works.

Later, Juan.

# gpg: Signature made Fri 28 Jan 2022 18:30:25 GMT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/quintela-gitlab/tags/migration-20220128-pull-request: (36 commits)
migration: Move temp page setup and cleanup into separate functions
migration: Simplify unqueue_page()
migration: Add postcopy_has_request()
migration: Enable UFFD_FEATURE_THREAD_ID even without blocktime feat
migration: No off-by-one for pss->page update in host page size
migration: Tally pre-copy, downtime and post-copy bytes independently
migration: Introduce ram_transferred_add()
migration: Don't return for postcopy_send_discard_bm_ram()
migration: Drop return code for disgard ram process
migration: Do chunk page in postcopy_each_ram_send_discard()
migration: Drop postcopy_chunk_hostpages()
migration: Don't return for postcopy_chunk_hostpages()
migration: Drop dead code of ram_debug_dump_bitmap()
migration/ram: clean up unused comment.
migration: Report the error returned when save_live_iterate fails
migration/migration.c: Remove the MIGRATION_STATUS_ACTIVE when migration finished
migration/migration.c: Avoid COLO boot in postcopy migration
migration/migration.c: Add missed default error handler for migration state
Remove unnecessary minimum_version_id_old fields
multifd: Rename pages_used to normal_pages
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Report the error returned when save_live_iterate fails

Should qemu_savevm_state_iterate() encounter a failure when calling a
particular save_live_iterate function, report the error code r

migration: Report the error returned when save_live_iterate fails

Should qemu_savevm_state_iterate() encounter a failure when calling a
particular save_live_iterate function, report the error code returned
by the function.

Signed-off-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge tag 'migration-20211214-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request

Hi

This are the reviewed patches for the freeze period:

- colo: fix/optimi

Merge tag 'migration-20211214-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request

Hi

This are the reviewed patches for the freeze period:

- colo: fix/optimize several things (rao, chen)
- shutdown qio channels correctly when an error happens (li)
- serveral multifd patches for the zero series (me)

Please apply.

Thanks, Juan.

# gpg: Signature made Wed 15 Dec 2021 02:32:09 AM PST
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]

* tag 'migration-20211214-pull-request' of https://gitlab.com/juan.quintela/qemu:
multifd: Make zlib compression method not use iovs
multifd: Make zstd compression method not use iovs
COLO: Move some trace code behind qemu_mutex_unlock_iothread()
multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
multifd: Fill offset and block for reception
multifd: remove used parameter from send_recv_pages() method
multifd: remove used parameter from send_prepare() method
multifd: The variable is only used inside the loop
multifd: Add missing documention
multifd: Rename used field to num
migration: Never call twice qemu_target_page_size()
multifd: Delete useless operation
dump: Remove is_zero_page()
migration: Remove is_zero_range()
migration/colo: Optimize COLO primary node start code path
Fixed a QEMU hang when guest poweroff in COLO mode
migration/colo: More accurate update checkpoint time
migration/ram.c: Remove the qemu_mutex_lock in colo_flush_ram_cache.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration: Never call twice qemu_target_page_size()

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211102-pull-request' into staging

Migration Pull request

Hi

This are the pending migration patches on the list:
- Provide an err

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211102-pull-request' into staging

Migration Pull request

Hi

This are the pending migration patches on the list:
- Provide an error message for migration_cancel by Laurent
- Don't dump colo cache when a guest core is requested by Lukas
- Initialise Compression_conters for new migration by Yuxiating
On top of that I added another missing initialization
- Colo optimizations and crash improvements by Rao.

Please, apply.

# gpg: Signature made Wed 03 Nov 2021 04:45:35 AM EDT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]

* remotes/juanquintela/tags/migration-20211102-pull-request:
Optimized the function of fill_connection_key.
colo: Don't dump colo cache if dump-guest-core=off
Changed the last-mode to none of first start COLO
Removed the qemu_fclose() in colo_process_incoming_thread
colo: fixed 'Segmentation fault' when the simplex mode PVM poweroff
Fixed SVM hang when do failover before PVM crash
Fixed qemu crash when guest power off in COLO mode
Some minor optimizations for COLO
migration: Zero migration compression counters
migration: initialise compression_counters for a new migration
migration: provide an error message to migration_cancel()

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration: Zero migration compression counters

Based on previous patch from yuxiating <yuxiating@huawei.com>

Signed-off-by: Juan Quintela <quintela@redhat.com>

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20210726a' into staging

Migration fixes 2021-07-26

Peter's fix for a bunch of races
-> Seem to fix the occasional crash se

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20210726a' into staging

Migration fixes 2021-07-26

Peter's fix for a bunch of races
-> Seem to fix the occasional crash seen by Peter

Wei's fix for migration with free page hinting
-> Bug has been around for a while, but makes a huge difference

My fix for OpenBSD test corner case

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# gpg: Signature made Mon 26 Jul 2021 13:42:16 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert-gitlab/tags/pull-migration-20210726a:
migration: clear the memory region dirty bitmap when skipping free pages
migration: Move the yank unregister of channel_close out
migration: Teach QEMUFile to be QIOChannel-aware
migration: Introduce migration_ioc_[un]register_yank()
migration: Make from_dst_file accesses thread-safe
migration: Fix missing join() of rp_thread
tests/qtest/migration-test.c: use 127.0.0.1 instead of 0

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Move the yank unregister of channel_close out

It's efficient, but hackish to call yank unregister calls in channel_close(),
especially it'll be hard to debug when qemu crashed with some y

migration: Move the yank unregister of channel_close out

It's efficient, but hackish to call yank unregister calls in channel_close(),
especially it'll be hard to debug when qemu crashed with some yank function
leaked.

Remove that hack, but instead explicitly unregister yank functions at the
places where needed, they are:

(on src)
- migrate_fd_cleanup
- postcopy_pause

(on dst)
- migration_incoming_state_destroy
- postcopy_pause_incoming

Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20210722175841.938739-6-peterx@redhat.com>
Reviewed-by: Lukas Straub <lukasstraub2@web.de>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: Teach QEMUFile to be QIOChannel-aware

migration uses QIOChannel typed qemufiles. In follow up patches, we'll need
the capability to identify this fact, so that we can get the backing QIO

migration: Teach QEMUFile to be QIOChannel-aware

migration uses QIOChannel typed qemufiles. In follow up patches, we'll need
the capability to identify this fact, so that we can get the backing QIOChannel
from a QEMUFile.

We can also define types for QEMUFile but so far since we only need to be able
to identify QIOChannel, introduce a boolean which is simpler.

Introduce another helper qemu_file_get_ioc() to return the ioc backend of a
qemufile if has_ioc is set.

No functional change.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20210722175841.938739-5-peterx@redhat.com>
Reviewed-by: Lukas Straub <lukasstraub2@web.de>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging

Merge misc patches

# gpg: Signature made Mon 14 Jun 2021 15:14:48 BST
# gpg: using RS

Merge remote-tracking branch 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging

Merge misc patches

# gpg: Signature made Mon 14 Jun 2021 15:14:48 BST
# gpg: using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg: aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF

* remotes/berrange-gitlab/tags/misc-fixes-pull-request:
usb/dev-mtp: use GDateTime for formatting timestamp for objects
block: use GDateTime for formatting timestamp when dumping snapshot info
migration: use GDateTime for formatting timestamp in snapshot names
block: remove duplicate trace.h include
block: add trace point when fdatasync fails
block: preserve errno from fdatasync failures
softmmu: add trace point when bdrv_flush_all fails
migration: add trace point when vm_stop_force_state fails
sasl: remove comment about obsolete kerberos versions
docs: recommend SCRAM-SHA-256 SASL mech instead of SHA-1 variant
docs: document usage of the authorization framework
docs: document how to pass secret data to QEMU
docs: add table of contents to QAPI references

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: use GDateTime for formatting timestamp in snapshot names

The GDateTime APIs provided by GLib avoid portability pitfalls, such
as some platforms where 'struct timeval.tv_sec' field is stil

migration: use GDateTime for formatting timestamp in snapshot names

The GDateTime APIs provided by GLib avoid portability pitfalls, such
as some platforms where 'struct timeval.tv_sec' field is still 'long'
instead of 'time_t'. When combined with automatic cleanup, GDateTime
often results in simpler code too.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20210208a' into staging

Migration pull 2021-02-08

v2
Dropped vmstate: Fix memory leak in vmstate_handle_alloc
Broke on Power

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20210208a' into staging

Migration pull 2021-02-08

v2
Dropped vmstate: Fix memory leak in vmstate_handle_alloc
Broke on Power
Added migration: only check page size match if RAM postcopy is enabled

# gpg: Signature made Mon 08 Feb 2021 11:28:14 GMT
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20210208a: (27 commits)
migration: only check page size match if RAM postcopy is enabled
migration: introduce snapshot-{save, load, delete} QMP commands
iotests: fix loading of common.config from tests/ subdir
iotests: add support for capturing and matching QMP events
migration: introduce a delete_snapshot wrapper
migration: wire up support for snapshot device selection
migration: control whether snapshots are ovewritten
block: rename and alter bdrv_all_find_snapshot semantics
block: allow specifying name of block device for vmstate storage
block: add ability to specify list of blockdevs during snapshot
migration: stop returning errno from load_snapshot()
migration: Make save_snapshot() return bool, not 0/-1
block: push error reporting into bdrv_all_*_snapshot functions
migration: Display the migration blockers
migration: Add blocker information
migration: Fix a few absurdly defective error messages
migration: Fix cache_init()'s "Failed to allocate" error messages
migration: Clean up signed vs. unsigned XBZRLE cache-size
migration: Fix migrate-set-parameters argument validation
migration: introduce 'userfaultfd-wrlat.py' script
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: introduce snapshot-{save, load, delete} QMP commands

savevm, loadvm and delvm are some of the few HMP commands that have never
been converted to use QMP. The reasons for the lack of conve

migration: introduce snapshot-{save, load, delete} QMP commands

savevm, loadvm and delvm are some of the few HMP commands that have never
been converted to use QMP. The reasons for the lack of conversion are
that they blocked execution of the event thread, and the semantics
around choice of disks were ill-defined.

Despite this downside, however, libvirt and applications using libvirt
have used these commands for as long as QMP has existed, via the
"human-monitor-command" passthrough command. IOW, while it is clearly
desirable to be able to fix the problems, they are not a blocker to
all real world usage.

Meanwhile there is a need for other features which involve adding new
parameters to the commands. This is possible with HMP passthrough, but
it provides no reliable way for apps to introspect features, so using
QAPI modelling is highly desirable.

This patch thus introduces new snapshot-{load,save,delete} commands to
QMP that are intended to replace the old HMP counterparts. The new
commands are given different names, because they will be using the new
QEMU job framework and thus will have diverging behaviour from the HMP
originals. It would thus be misleading to keep the same name.

While this design uses the generic job framework, the current impl is
still blocking. The intention that the blocking problem is fixed later.
None the less applications using these new commands should assume that
they are asynchronous and thus wait for the job status change event to
indicate completion.

In addition to using the job framework, the new commands require the
caller to be explicit about all the block device nodes used in the
snapshot operations, with no built-in default heuristics in use.

Note that the existing "query-named-block-nodes" can be used to query
what snapshots currently exist for block nodes.

Acked-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210204124834.774401-13-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
dgilbert: removed tests for now, the output ordering isn't
deterministic

show more ...

migration: introduce a delete_snapshot wrapper

Make snapshot deletion consistent with the snapshot save
and load commands by using a wrapper around the blockdev
layer. The main difference is that we

migration: introduce a delete_snapshot wrapper

Make snapshot deletion consistent with the snapshot save
and load commands by using a wrapper around the blockdev
layer. The main difference is that we get upfront validation
of the passed in device list (if any).

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210204124834.774401-10-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: wire up support for snapshot device selection

Modify load_snapshot/save_snapshot to accept the device list and vmstate
node name parameters previously added to the block layer.

Reviewed-

migration: wire up support for snapshot device selection

Modify load_snapshot/save_snapshot to accept the device list and vmstate
node name parameters previously added to the block layer.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210204124834.774401-9-berrange@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...