Merge tag 'migration-20241030-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull request for softfreeze

v2:
- Patch "migration: Move cpu-throttle.c from system to migration

Merge tag 'migration-20241030-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull request for softfreeze

v2:
- Patch "migration: Move cpu-throttle.c from system to migration",
fix build on MacOS, and subject spelling

NOTE: checkpatch.pl could report a false positive on this branch:

WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#21:
{include/sysemu => migration}/cpu-throttle.h | 0

That's covered by "F: migration/" entry.

Changelog:

- Peter's cleanup patch on migrate_fd_cleanup()
- Peter's cleanup patch to introduce thread name macros
- Hanna's error path fix for vmstate subsection save()s
- Hyman's auto converge enhancement on background dirty sync
- Peter's additional tracepoints for save state entries
- Thomas's build fix for OpenBSD in dirtyrate.c
- Peter's deprecation of query-migrationthreads command
- Peter's cleanup/fixes from the "export misc.h" series
- Maciej's two small patches from multifd+vfio series

# -----BEGIN PGP SIGNATURE-----
#
# iIgEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZyTbVRIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wan3wD+L4TVNDc34Hy4mvWu7u1lCOePX0GBdUEc
# oEeBGblwbrcBAIR8d+5z9O5YcWH1coozG1aUC4qCtSHHk5TGbJk4/UUD
# =XB5Q
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 01 Nov 2024 13:44:53 GMT
# gpg: using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg: issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [marginal]
# gpg: aka "Peter Xu <peterx@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D D1A9 3B5F CCCD F3AB D706

* tag 'migration-20241030-pull-request' of https://gitlab.com/peterx/qemu:
migration/multifd: Zero p->flags before starting filling a packet
migration/ram: Add load start trace event
migration: Drop migration_is_idle()
migration: Drop migration_is_setup_or_active()
migration: Unexport ram_mig_init()
migration: Unexport dirty_bitmap_mig_init()
migration: Take migration object refcount earlier for threads
migration: Deprecate query-migrationthreads command
migration/dirtyrate: Silence warning about strcpy() on OpenBSD
tests/migration: Add case for periodic ramblock dirty sync
migration: Support periodic RAMBlock dirty bitmap sync
migration: Remove "rs" parameter in migration_bitmap_sync_precopy
migration: Move cpu-throttle.c from system to migration
migration: Stop CPU throttling conditionally
accel/tcg/icount-common: Remove the reference to the unused header file
migration: Ensure vmstate_save() sets errp
migration: Put thread names together with macros
migration: Cleanup migrate_fd_cleanup() on accessing to_dst_file

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Unexport ram_mig_init()

It's only used within migration/.

Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/2024

migration: Unexport ram_mig_init()

It's only used within migration/.

Reviewed-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20241024213056.1395400-4-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>

show more ...

Merge tag 'migration-20240423-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull for 9.1

- Het's new test cases for "channels"
- Het's fix for a typo for vsock parsing
- Ce

Merge tag 'migration-20240423-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull for 9.1

- Het's new test cases for "channels"
- Het's fix for a typo for vsock parsing
- Cedric's VFIO error report series
- Cedric's one more patch for dirty-bitmap error reports
- Zhijian's rdma deprecation patch
- Yuan's zeropage optimization to fix double faults on anon mem
- Zhijian's COLO fix on a crash

# -----BEGIN PGP SIGNATURE-----
#
# iIgEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZig4HxIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wbQiwD/V5nSJzSuAG4Ra1Fjo+LRG2TT6qk8eNCi
# fIytehSw6cYA/0wqarxOF0tr7ikeyhtG3w4xFf44kk6KcPkoVSl1tqoL
# =pJmQ
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 23 Apr 2024 03:37:19 PM PDT
# gpg: using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg: issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [unknown]
# gpg: aka "Peter Xu <peterx@redhat.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D D1A9 3B5F CCCD F3AB D706

* tag 'migration-20240423-pull-request' of https://gitlab.com/peterx/qemu: (26 commits)
migration/colo: Fix bdrv_graph_rdlock_main_loop: Assertion `!qemu_in_coroutine()' failed.
migration/multifd: solve zero page causing multiple page faults
migration: Add Error** argument to add_bitmaps_to_list()
migration: Modify ram_init_bitmaps() to report dirty tracking errors
migration: Add Error** argument to xbzrle_init()
migration: Add Error** argument to ram_state_init()
memory: Add Error** argument to the global_dirty_log routines
migration: Introduce ram_bitmaps_destroy()
memory: Add Error** argument to .log_global_start() handler
migration: Add Error** argument to .load_setup() handler
migration: Add Error** argument to .save_setup() handler
migration: Add Error** argument to qemu_savevm_state_setup()
migration: Add Error** argument to vmstate_save()
migration: Always report an error in ram_save_setup()
migration: Always report an error in block_save_setup()
vfio: Always report an error in vfio_save_setup()
s390/stattrib: Add Error** argument to set_migrationmode() handler
tests/qtest/migration: Fix typo for vsock in SocketAddress_to_str
tests/qtest/migration: Add negative tests to validate migration QAPIs
tests/qtest/migration: Add multifd_tcp_plain test using list of channels instead of uri
...

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration/multifd: solve zero page causing multiple page faults

Implemented recvbitmap tracking of received pages in multifd.

If the zero page appears for the first time in the recvbitmap, this
pag

migration/multifd: solve zero page causing multiple page faults

Implemented recvbitmap tracking of received pages in multifd.

If the zero page appears for the first time in the recvbitmap, this
page is not checked and set.

If the zero page has already appeared in the recvbitmap, there is no
need to check the data but directly set the data to 0, because it is
unlikely that the zero page will be migrated multiple times.

Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240401154110.2028453-2-yuan1.liu@intel.com
[peterx: touch up the comment, as the bitmap is used outside postcopy now]
Signed-off-by: Peter Xu <peterx@redhat.com>

show more ...

Merge tag 'migration-20240311-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull request

- Avihai's fix to allow vmstate iterators to not starve for VFIO
- Maksim's fix on

Merge tag 'migration-20240311-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull request

- Avihai's fix to allow vmstate iterators to not starve for VFIO
- Maksim's fix on additional check on precopy load error
- Fabiano's fix on fdatasync() hang in mapped-ram
- Jonathan's fix on vring cached access over MMIO regions
- Cedric's cleanup patches 1-4 out of his error report series
- Yu's fix for RDMA migration (which used to be broken even for 8.2)
- Anthony's small cleanup/fix on err message
- Steve's patches on privatize migration.h
- Xiang's patchset to enable zero page detections in multifd threads

# -----BEGIN PGP SIGNATURE-----
#
# iIgEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZe9+uBIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wamaQD/SvmpMEcuRndT9LPSxzXowAGDZTBpYUfv
# 5XAbx80dS9IBAO8PJJgQJIBHBeacyLBjHP9CsdVtgw5/VW+wCsbfV4AB
# =xavb
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 11 Mar 2024 21:59:20 GMT
# gpg: using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg: issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [marginal]
# gpg: aka "Peter Xu <peterx@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D D1A9 3B5F CCCD F3AB D706

* tag 'migration-20240311-pull-request' of https://gitlab.com/peterx/qemu: (34 commits)
migration/multifd: Add new migration test cases for legacy zero page checking.
migration/multifd: Enable multifd zero page checking by default.
migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.
migration/multifd: Implement zero page transmission on the multifd thread.
migration/multifd: Add new migration option zero-page-detection.
migration/multifd: Allow clearing of the file_bmap from multifd
migration/multifd: Allow zero pages in file migration
migration: purge MigrationState from public interface
migration: delete unused accessors
migration: privatize colo interfaces
migration: migration_file_set_error
migration: migration_is_device
migration: migration_thread_is_self
migration: export vcpu_dirty_limit_period
migration: export migration_is_running
migration: export migration_is_active
migration: export migration_is_setup_or_active
migration: remove migration.h references
migration: export fewer options
migration: Fix format in error message
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration/multifd: Allow clearing of the file_bmap from multifd

We currently only need to clear the mapped-ram file bitmap from the
migration thread during save_zero_page.

We're about to add suppor

migration/multifd: Allow clearing of the file_bmap from multifd

We currently only need to clear the mapped-ram file bitmap from the
migration thread during save_zero_page.

We're about to add support for zero page detection on the multifd
thread, so allow ramblock_set_file_bmap_atomic() to also clear the
bits.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240311180015.3359271-3-hao.xiang@linux.dev
Signed-off-by: Peter Xu <peterx@redhat.com>

show more ...

Merge tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu into staging

Migartion pull request for 20240304

- Bryan's fix on multifd compression level API
- Fabiano's mapped-ram seri

Merge tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu into staging

Migartion pull request for 20240304

- Bryan's fix on multifd compression level API
- Fabiano's mapped-ram series (base + multifd only)
- Steve's amend on cpr document in qapi/

# -----BEGIN PGP SIGNATURE-----
#
# iIgEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZeUjKhIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wbv5QD/ZexBUsmZA5qyxgGvZ2yvlUBEGNOvtmKY
# kRdiYPU7khMA/0N43rn4LcqKCoq4+T+EAnYizGjIyhH/7BRUyn4DUxgO
# =AeEn
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 04 Mar 2024 01:26:02 GMT
# gpg: using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg: issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [marginal]
# gpg: aka "Peter Xu <peterx@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D D1A9 3B5F CCCD F3AB D706

* tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu: (27 commits)
migration/multifd: Document two places for mapped-ram
tests/qtest/migration: Add a multifd + mapped-ram migration test
migration/multifd: Add mapped-ram support to fd: URI
migration/multifd: Support incoming mapped-ram stream format
migration/multifd: Support outgoing mapped-ram stream format
migration/multifd: Prepare multifd sync for mapped-ram migration
migration/multifd: Add incoming QIOChannelFile support
migration/multifd: Add outgoing QIOChannelFile support
migration/multifd: Add a wrapper for channels_created
migration/multifd: Allow receiving pages without packets
migration/multifd: Allow multifd without packets
migration/multifd: Decouple recv method from pages
migration/multifd: Rename MultiFDSend|RecvParams::data to compress_data
tests/qtest/migration: Add tests for mapped-ram file-based migration
migration/ram: Add incoming 'mapped-ram' migration
migration/ram: Add outgoing 'mapped-ram' migration
migration: Add mapped-ram URI compatibility check
migration/ram: Introduce 'mapped-ram' migration capability
migration/qemu-file: add utility methods for working with seekable channels
io: fsync before closing a file channel
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

# Conflicts:
# migration/ram.c

show more ...

migration/multifd: Support outgoing mapped-ram stream format

The new mapped-ram stream format uses a file transport and puts ram
pages in the migration file at their respective offsets and can be
do

migration/multifd: Support outgoing mapped-ram stream format

The new mapped-ram stream format uses a file transport and puts ram
pages in the migration file at their respective offsets and can be
done in parallel by using the pwritev system call which takes iovecs
and an offset.

Add support to enabling the new format along with multifd to make use
of the threading and page handling already in place.

This requires multifd to stop sending headers and leaving the stream
format to the mapped-ram code. When it comes time to write the data, we
need to call a version of qio_channel_write that can take an offset.

Usage on HMP is:

(qemu) stop
(qemu) migrate_set_capability multifd on
(qemu) migrate_set_capability mapped-ram on
(qemu) migrate_set_parameter max-bandwidth 0
(qemu) migrate_set_parameter multifd-channels 8
(qemu) migrate file:migfile

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-21-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>

show more ...

Merge tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231102)

Hi

In this pull request:

- migration reboot mode (steve)
* I

Merge tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231102)

Hi

In this pull request:

- migration reboot mode (steve)
* I disabled the test because our CI don't like programs using so
much shared memory. Searching for a fix.
- test for postcopy recover (fabiano)
- MigrateAddress QAPI (het)
- better return path error handling (peter)
- traces for downtime (peter)
- vmstate_register() check for duplicates (juan)
thomas find better solutions for s390x and ipmi.
now also works on s390x

Please, apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmVDipMACgkQ9IfvGFhy
# 1yNYnQ/9E5Cywsoqljqa/9FiKBSII2qMrmkfu6JLKqePnsh5pFZiukbudYRuJCCe
# ZTDEmD0NmKRJbDx2xRU1qx/e6gKJy+gz37KP89Buuh/WwZHPboPYtxQpGvCSiH26
# J3i+1+TgaqmkLzcO35wa8tp6gneQclWeAwKgMvdb4cm2pJEhgWRKI62ccyLzxeve
# UCzFQn60t55ETyVZGnRD4YwdTQvGKH+DPlyTuJOLR3DePuvZd8EdH+ypvB4RLAy7
# 3+CuQOxmF5LRXPbpJuAeOsudbmhhHzrO/yL7ZmsiKQTthsJv+SzC1bO94jhQrawZ
# Q7GCii5KpGq0KnRTRKZRGk6XKwxcYRduXMX3R5tXuVmDmCZsjhXzziU8yEdftph8
# 5TJdk1o0Gb043EFu81mrsQYS+9yJqe6sy6m3PTJaec54cAty5ln+c17WOvpAOaSV
# +1phe05ftuVPmQ3KWhbIR/tCmavNLwEZxpVIfyaKJx04bFbtQ9gRpRyURORX4KXc
# s4WXvNirQEohxYBnP4TPvA09xBTW3V08pk/wRDwt0YDXnLiqCltOuxD8r05K8K4B
# MkCLcWj0g7he2tBkF60oz1KSIE0oTB81um9AzLIv5F2YSYLaJM5BIcoC437MR2f4
# MOR7drR1fP5GsRu/SeU5BWvhVq3IvdOxR7G2MLNRJJvl7ZtGXDc=
# =uaqL
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 02 Nov 2023 19:40:03 HKT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu: (40 commits)
migration: modify test_multifd_tcp_none() to use new QAPI syntax.
migration: Implement MigrateChannelList to hmp migration flow.
migration: Implement MigrateChannelList to qmp migration flow.
migration: modify migration_channels_and_uri_compatible() for new QAPI syntax
migration: New migrate and migrate-incoming argument 'channels'
migration: Convert the file backend to the new QAPI syntax
migration: convert exec backend to accept MigrateAddress.
migration: convert rdma backend to accept MigrateAddress
migration: convert socket backend to accept MigrateAddress
migration: convert migration 'uri' into 'MigrateAddress'
migration: New QAPI type 'MigrateAddress'
migration: Change ram_dirty_bitmap_reload() retval to bool
tests/migration-test: Add a test for postcopy hangs during RECOVER
migration: Allow network to fail even during recovery
migration: Refactor error handling in source return path
tests/qtest: migration: add reboot mode test
cpr: reboot mode
cpr: relax vhost migration blockers
cpr: relax blockdev migration blockers
migration: per-mode blockers
...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

migration: Change ram_dirty_bitmap_reload() retval to bool

Now we have a Error** passed into the return path thread stack, which is
even clearer than an int retval. Change ram_dirty_bitmap_reload()

migration: Change ram_dirty_bitmap_reload() retval to bool

Now we have a Error** passed into the return path thread stack, which is
even clearer than an int retval. Change ram_dirty_bitmap_reload() and the
callers to use a bool instead to replace errnos.

Suggested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-5-peterx@redhat.com>

show more ...

migration: Refactor error handling in source return path

rp_state.error was a boolean used to show error happened in return path
thread. That's not only duplicating error reporting (migrate_set_err

migration: Refactor error handling in source return path

rp_state.error was a boolean used to show error happened in return path
thread. That's not only duplicating error reporting (migrate_set_error),
but also not good enough in that we only do error_report() and set it to
true, we never can keep a history of the exact error and show it in
query-migrate.

To make this better, a few things done:

- Use error_setg() rather than error_report() across the whole lifecycle
of return path thread, keeping the error in an Error*.

- With above, no need to have mark_source_rp_bad(), remove it, alongside
with rp_state.error itself.

- Use migrate_set_error() to apply that captured error to the global
migration object when error occured in this thread.

- Do the same when detected qemufile error in source return path

We need to re-export qemu_file_get_error_obj() to do the last one.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-2-peterx@redhat.com>

show more ...

Merge tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231031)

Hi

This is repeat of the Migration PULL for 20231020.
- I remove

Merge tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231031)

Hi

This is repeat of the Migration PULL for 20231020.
- I removed vmstate_register(big problems with s390x)
- I added yet more countes (juan)

CI: https://gitlab.com/juan.quintela/qemu/-/pipelines/1055797950

Please apply.

Thanks, Juan.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmVAwmYACgkQ9IfvGFhy
# 1yPJ9g//f8Up+5Az0DmJMWwRe+08vLa3ZRCSh5aCRJguFVfMZSVxRNuoikQ/C/Gz
# 1ePB+Q8H0NcP86FF7pifhtLU0uE9L4At4Z+vOQP1+n67p7aush050kKQxyDYIfO2
# 3tO2HkfHvC/R3S5FtqQtE1Y0/MpHdj1vgV9bNidPorA6EZ01KEEfWw3soptuD14I
# LPvXA8BG5mOvB7R55MymTAej3ZDmOUQlZotsE2KmlkOfzYoqTtApkLtW03/WH8b8
# fAYJ0ghYpesRTO1rF61n1peLMUr+/HRLqGJmhLDSEZZlB5tnUYeiLR9dRJ1/1+o2
# zNjLr6X2hnia6Kb0UibRoAcyyy8lSLp79Zt5nhDneuTSQxeYhNh6EecxAzKvd/02
# vfE/reOEkZn7KzYH/MvlD5P6XmwrT5aV9cqmyC/8BkNnipHAtJ2Av1H4ONdnahuK
# hOhLRAGE7SINtgo8jdauQNor1QAsIX19nvYk9p7ta5VAysrDSbuD+9Yq7HtUErlP
# 585z5BPGfaP2GwIXPNJNcqXwPh0InInGASqEWmYSlu8GF3Ic0KNWWrC5bwSn7tHL
# I7qaMrCHxvWGYx6cRzzp08EqCcbOQCixrPyk8g6o3SgXHrTGKthzjPG5bLe+QXpv
# P2gblC7Fo3sUo89IwVjsRMO3nU9wBfb9skE7iZM06SILO7QD3u8=
# =r1DI
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 31 Oct 2023 18:01:26 JST
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu: (38 commits)
qemu-file: Make qemu_fflush() return errors
migration: Remove transferred atomic counter
migration: Use migration_transferred_bytes()
qemu-file: Simplify qemu_file_get_error()
migration: migration_rate_limit_reset() don't need the QEMUFile
migration: migration_transferred_bytes() don't need the QEMUFile
qemu-file: Remove _noflush from qemu_file_transferred_noflush()
qemu_file: Remove unused qemu_file_transferred()
migration: Use the number of transferred bytes directly
qemu_file: total_transferred is not used anymore
qemu_file: Use a stat64 for qemu_file_transferred
qemu-file: Don't increment qemu_file_transferred at qemu_file_fill_buffer
migration: Stop migration immediately in RDMA error paths
migration: Deprecate old compression method
migration: Deprecate block migration
migration: migrate 'blk' command option is deprecated.
migration: migrate 'inc' command option is deprecated.
qemu-iotests: Filter warnings about block migration being deprecated
migration: set file error on subsection loading
migration: rename vmstate_save_needed->vmstate_section_needed
...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

migration: Create compress_update_rates()

So we can move more compression_counters stuff to ram-compress.c.
Create compression_counters struct to add the stuff that was on
MigrationState.

Reviewed-

migration: Create compress_update_rates()

So we can move more compression_counters stuff to ram-compress.c.
Create compression_counters struct to add the stuff that was on
MigrationState.

Reviewed-by: Lukas Straub <lukasstraub2@web.de>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231019110724.15324-8-quintela@redhat.com>

show more ...

migration: Rename ram_handle_compressed() to ram_handle_zero()

Now that we know it only handles zero, we can remove the ch parameter.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Peter

migration: Rename ram_handle_compressed() to ram_handle_zero()

Now that we know it only handles zero, we can remove the ch parameter.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231019085259.13307-3-quintela@redhat.com>

show more ...

Merge tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Memory device cleanups (especially around machin

Merge tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Memory device cleanups (especially around machine initialization)
- "x-ignore-shared" migration support for virtio-mem
- Add an abstract virtio-md-pci device as a common parent for
virtio-mem-pci and virtio-pmem-pci (virtio based memory devices)
- Device unplug support for virtio-mem-pci

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAmSuYAQRHGRhdmlkQHJl
# ZGhhdC5jb20ACgkQTd4Q9wD/g1od9A/9HXT8IqKGup9is7P/mpobPWXczRGZ5sEg
# /q21PzX6crr9aFa+fYRF/Dlm3G/cSMOVXFRKGz3royLjsvaEj/veEewfKF8KWbBf
# eIS9udQTOwoD2kAhcv3pm0SwSJoVizpw2z7IodGVKE6iZxTXsmDksqQuFbrvVLSh
# 2wtP4lizEXco/YsiCoAnStj2QtXBcHw7Ua7W2cDzxFmL+1pM5w3rjQ1ydCNz3bSG
# l4CXXs1i8OmOZbFN78F/E9SEkzQnAuHSO0Sc1aeAJkwVzOt2lj/YMgt0jHjAY0at
# pheWZ5pEE6hnQP740YXpt4Y6IIgO22pH23dLhq9A2reyRnwjt830uObHi3qAE8kB
# KR+ZQ+Z5bI6ZNB/EFiUsC1dFsr2fF20zQlO02MctyJ+lUG6p3gpvwsGScQxt+zdF
# QlkiSecGErYwC+nZ529SQB4gSEJTCjd/STDoidVYnZazdStaOaSyft02xRNzBPW/
# OnOY+6ZxZK6R11KfwGjnsftrovQIP3Pqi9TXGzW2xVlkWJHqlicy6G3ZfceTTlj9
# Gg2Ue694Wr1r4PDV2XlYcZ1IPLjSy5Msp5V2wERRrp3OItxnvegvTevQN7USEHC+
# BPGNMu11jriSY2pE5BSFN0hfGOvuvsk3GreLJiHFUXoje6gzAynuLjCN/CHdIVyK
# 5i0AwdZ+xcA=
# =ch6m
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 12 Jul 2023 09:10:44 AM BST
# gpg: using RSA key 1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A
# gpg: issuer "david@redhat.com"
# gpg: Good signature from "David Hildenbrand <david@redhat.com>" [unknown]
# gpg: aka "David Hildenbrand <davidhildenbrand@gmail.com>" [undefined]
# gpg: aka "David Hildenbrand <hildenbr@in.tum.de>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1BD9 CAAD 735C 4C3A 460D FCCA 4DDE 10F7 00FF 835A

* tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu: (21 commits)
virtio-mem-pci: Device unplug support
virtio-mem: Prepare for device unplug support
virtio-md-pci: Support unplug requests for compatible devices
virtio-md-pci: Handle unplug of virtio based memory devices
arm/virt: Use virtio-md-pci (un)plug functions
pc: Factor out (un)plug handling of virtio-md-pci devices
virtio-md-pci: New parent type for virtio-mem-pci and virtio-pmem-pci
virtio-mem: Support "x-ignore-shared" migration
migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()
virtio-mem: Skip most of virtio_mem_unplug_all() without plugged memory
softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping
memory-device: Track used region size in DeviceMemoryState
memory-device: Refactor memory_device_pre_plug()
hw/i386/pc: Remove PC_MACHINE_DEVMEM_REGION_SIZE
hw/i386/acpi-build: Rely on machine->device_memory when building SRAT
hw/i386/pc: Use machine_memory_devices_init()
hw/loongarch/virt: Use machine_memory_devices_init()
hw/ppc/spapr: Use machine_memory_devices_init()
hw/arm/virt: Use machine_memory_devices_init()
memory-device: Introduce machine_memory_devices_init()
...

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()

virtio-mem wants to know whether it should not mess with the RAMBlock
content (e.g., discard RAM, preallocate memory) on incom

migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()

virtio-mem wants to know whether it should not mess with the RAMBlock
content (e.g., discard RAM, preallocate memory) on incoming migration.

So let's expose that function as migrate_ram_is_ignored() in
migration/misc.h

Message-ID: <20230706075612.67404-4-david@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Tested-by: Mario Casquero <mcasquer@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>

show more ...

Merge tag 'migration-20230509-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20230509 vintage) take 2

Hi

In this take 2:
- Change uint -> uint32_t to f

Merge tag 'migration-20230509-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20230509 vintage) take 2

Hi

In this take 2:
- Change uint -> uint32_t to fix mingw32 compilation.

Please apply.
[take 1]
In this PULL request:
- 1st part of colo support for multifd (lukas)
- 1st part of disabling colo option (vladimir)

Please, apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmRb3dgACgkQ9IfvGFhy
# 1yNLBxAAwHiAOdSPS7TqJXH2/PkBKsd42XMtWzC9UowZ6SUdQi0Q2bQUBnygJ8BA
# 59yLOTPdwUhaPWk4KsyKM2znOCJ+f9MF5V4QXbyILf1WCAq6d+mtPwArnYF1TRwi
# XIewVDeRopdOO5lnWGcfAKZZ5WIDzA/bn6NiGLi+pQa5HGyk84Bk+tFa8kJI6xBL
# 5CWfhNTcxDNYRFg/z/9YVirkuxIXEEL6VEeRFV+pmFuj05q9bysWJkLFoEcFNawO
# gp1foHDkU7wHmHDJ3D4AVTm3TW641ft1wdlHIHZRoOiIIu3EUOoDEVVsaCfdxrY8
# pPJZ5m37wb52GIaCJmigG8rkHxIJ8xKLk4HKu4umDqFq5jZQ2krnnj7AkQhpp7p2
# aEIOXJQQq7XCsKpuvSUIexPv4gbN5SEYKi7XKoOPe3sZ03Rkn0I5xY3KSyMQMamP
# jtk8tNlRA+9Wug82eb/FtIKDj3//4SbuQOJEdRXjKJBldd3mtWTT/FRj/8oo96/p
# hmTu/cGDrP5qgtWpz0kKI/xaBf8at1nwpDgdEzOjRw4zf6xQHFjbXgJ7tQBH/JUI
# T3A9pdiXN6QdRupcWUSV0iJsfS/5i3mOUTA/C529qGXabSnZzfMK+unL/I8N02yt
# 83o7jSg22etMjaS1c+VuDmzKCAfuZloDZv2Bms/+yM/8k8Xe5S4=
# =vbqf
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 10 May 2023 07:09:28 PM BST
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [undefined]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20230509-pull-request' of https://gitlab.com/juan.quintela/qemu:
migration: block incoming colo when capability is disabled
migration: disallow change capabilities in COLO state
migration: process_incoming_migration_co: simplify code flow around ret
migration: drop colo_incoming_thread from MigrationIncomingState
build: move COLO under CONFIG_REPLICATION
colo: make colo_checkpoint_notify static and provide simpler API
block/meson.build: prefer positive condition for replication
multifd: Add the ramblock to MultiFDRecvParams
ram: Let colo_flush_ram_cache take the bitmap_mutex
ram: Add public helper to set colo bitmap

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

ram: Add public helper to set colo bitmap

The overhead of the mutex in non-multifd mode is negligible,
because in that case its just the single thread taking the mutex.

This will be used in the nex

ram: Add public helper to set colo bitmap

The overhead of the mutex in non-multifd mode is negligible,
because in that case its just the single thread taking the mutex.

This will be used in the next commits to add colo support to multifd.

Signed-off-by: Lukas Straub <lukasstraub2@web.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <22d83cb428f37929563155531bfb69fd8953cc61.1683572883.git.lukasstraub2@web.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge tag 'migration-20230428-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migraiton Pull request (20230428 take 2)

Hi

Dropped the compression cleanups to see if we find wha

Merge tag 'migration-20230428-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migraiton Pull request (20230428 take 2)

Hi

Dropped the compression cleanups to see if we find what is going on.

Please apply.

Later, Juan.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmRSKKQACgkQ9IfvGFhy
# 1yN1Tw//ZBExxagVY5Uy7oUp+YBBexK3PkcWrU5xNVIu1SiFr654AWWO6DUoMIde
# pb0EsM1rU/7iSPw3PYOPJDAO2eHmzeeLxGTCbR98R4YSHgr5WBHcRmzKnY5irKIm
# JDn4xxyp8+grCcMTi5nJ6CMVDvtL8CAA1fZfn2zzJimJkuSRkTAYgGw1fIXXut05
# 911eOccmA3mUFsU81bOXLvWhmrPCyy5m2vrLhIhwEJGTUdVdQlk72at7aWq1pu9D
# WIZJG6hI/eGZW5DX6Xbj7tfLW2MeqBFpFijuCNsxU4jAGSoBHR7And9YO/i7kDCp
# 3RlQyL/sVf6NZ7STEXrLEIh7g7nH05AF3ZuuwBn8TN2Vf9NkVPNFUygdmwukzHTj
# i8/VoC/gSoVfnA4JXVhnVtzHwRpgzP6rlKSeHEW1IMMnKC0xvmss6Mq7+z9dZxx+
# /XbxdjjwmsjnYUxH+BNwUVX2PjPguF5xib8M1QhsQNGOZ2Z+4RuWj4h/ItWsGqVI
# jjI3sVPY6cw8aiSXiSL2AO9Hde9KRIkcITby/zembSkvfsRR3kIWcMNJsteAP4UI
# t6MQSGsFRjMUKeIyArU/ES3iYvnBXi9LSwOR94S9KR1vEPxmhefoGu5wnwJ0is3C
# nAJPaHkQCY4JCCsyNO18hpSStOlNH300fIlMoB+ftcbn+fnpeOo=
# =iGGF
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 03 May 2023 10:25:56 AM BST
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [undefined]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20230428-pull-request' of https://gitlab.com/juan.quintela/qemu:
migration: Drop unused parameter for migration_tls_client_create()
migration: Drop unused parameter for migration_tls_get_creds()
migration/rdma: Unfold last user of acct_update_position()
migration/rdma: Split the zero page case from acct_update_position
migration: Rename RAMStats to MigrationAtomicStats
migration: Rename ram_counters to mig_stats
migration: Move ram_stats to its own file migration-stats.[ch]
multifd: We already account for this packet on the multifd thread

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration/rdma: Unfold last user of acct_update_position()

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Lukas Straub <lukasstraub2@web.de>

migration/rdma: Split the zero page case from acct_update_position

Now that we have atomic counters, we can do it on the place that we
need it, no need to do it inside ram.c.

Signed-off-by: Juan Qu

migration/rdma: Split the zero page case from acct_update_position

Now that we have atomic counters, we can do it on the place that we
need it, no need to do it inside ram.c.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Lukas Straub <lukasstraub2@web.de>

show more ...

migration: Move ram_stats to its own file migration-stats.[ch]

There is already include/qemu/stats.h, so stats.h was a bad idea.
We want this file to not depend on anything else, we will move all th

migration: Move ram_stats to its own file migration-stats.[ch]

There is already include/qemu/stats.h, so stats.h was a bad idea.
We want this file to not depend on anything else, we will move all the
migration counters/stats to this struct.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Lukas Straub <lukasstraub2@web.de>

show more ...

Merge tag 'migration-20230427-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20230427 edition)

Hi

Everything that has been reviewed:
- stat64_set() by

Merge tag 'migration-20230427-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20230427 edition)

Hi

Everything that has been reviewed:
- stat64_set() by paolo
- atomic_counters series fully reviewed (juan)
- move capabilities to options.c fully reviewed (juan)
- fix the channels_ready semaphore (juan)
- multifd flush optimization reviewed (juan)

Please, apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmRKi3YACgkQ9IfvGFhy
# 1yPtERAAgkBt4g3tpn90q+QeFLBPyo0KGxX4UiAmh8aiGRAJzmSJfjaQG3RgK0vM
# Iyat60Tx6lpHta7xRE+pDF7tXmoi0631B5FJjfuyTIFGWdrOpjGHihuxWdnIjK0H
# ZhR3LfQvHqybMfiyZN/7T5/qICB0zZ+erIeoaurRU3RVrO8yRPM+VBwsYr1RW1S4
# DxUeXEJF09tat7TgiQYp2w5cMPcQ2AvzgViR2r7yc0WjY2yUscyMjuPbHPqAsYWB
# UkbNuH7M/quvHqGvgxGYEQ+a6wyBzKlissjxWe8KTuSLNML2RVR6IrpuEPe6XyGe
# FWi+XRB5114EMy7YjjgTWzcLMC9EmhpzKTgJi0NEtL+cGRb9ZaSaSLE5yO7GLF9b
# efhCXT9jIef9/3Hi+Kr0juJ3nX6c7pINTAsx7wxKU0phz5kYx37IqIioMlVDh1ew
# ECpH+8Syrp/zYytc/HIpyOwIR9VdZrRi6jr2yqcc9lLLcuNUzl+E+9mAxWzO/gef
# +cYPnQZC0rjIzJSBZjq7TXYFNblD6HreONGL8UdFN0B7Js5mPu2PU3449jYNSOxi
# 1aj/eHNXEcqKcVZ3GDGgtus3ARvFsbEba0UZYIVboKp5i96f8O9YqKvRdBjAjpq1
# IDT/fvI2Fj1mjKi8KuAq8fK5utm+g4NR9WkQX0d661dQNZ3XqVc=
# =NE9v
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 27 Apr 2023 03:49:26 PM BST
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [undefined]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20230427-pull-request' of https://gitlab.com/juan.quintela/qemu:
migration: Make dirty_bytes_last_sync atomic
migration: Make dirty_pages_rate atomic
stat64: Add stat64_set() operation
multifd: Only flush once each full round of memory
multifd: Protect multifd_send_sync_main() calls
multifd: Create property multifd-flush-after-each-section
migration: Move migration_properties to options.c
migration: Create migrate_block_bitmap_mapping() function
migration: Create migrate_tls_hostname() function
migration: Create migrate_tls_authz() function
migration: Create migrate_tls_creds() function
migration: Remove MigrationState from block_cleanup_parameters()
migration: Move block_cleanup_parameters() to options.c
migration: Move migrate_set_block_incremental() to options.c
migration: Create migrate_downtime_limit() function
migration: Make all functions check have the same format
migration: Create migrate_params_init() function
multifd: Fix the number of channels ready

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration: Make dirty_bytes_last_sync atomic

As we set its value, it needs to be operated with atomics.
We rename it from remaining to better reflect its meaning.

Statistics always return the real

migration: Make dirty_bytes_last_sync atomic

As we set its value, it needs to be operated with atomics.
We rename it from remaining to better reflect its meaning.

Statistics always return the real reamaining bytes. This was used to
store how much pages where dirty on the previous generation, so we can
calculate the expected downtime as: dirty_bytes_last_sync /
current_bandwith.

If we use the actual remaining bytes, we would see a very small value
at the end of the iteration.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

---

I am open to use ram_bytes_remaining() in its only use and be more
"optimistic" about the downtime.

Don't use __nocheck() functions.
Use stat64_get() now that it exists.

show more ...

migration: Make dirty_pages_rate atomic

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Peter Xu <peterx@redhat.com>

--

migration: Make dirty_pages_rate atomic

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Peter Xu <peterx@redhat.com>

---

Don't use __nocheck() variants
Use stat64_get()

show more ...