Merge tag 'dirtylimit-dirtyrate-pull-request-20231010' of https://github.com/newfriday/qemu into staging

Dirtylimit and dirtyrate 20231010 patches PULL request

Dirty page rate measurement optimizat

Merge tag 'dirtylimit-dirtyrate-pull-request-20231010' of https://github.com/newfriday/qemu into staging

Dirtylimit and dirtyrate 20231010 patches PULL request

Dirty page rate measurement optimization.
Please apply, thanks, Yong.

# -----BEGIN PGP SIGNATURE-----
#
# iQJKBAABCAA0FiEEaF0CINwmSCgVLlfC3/Ij1rP+y5wFAmUklg0WHHlvbmcuaHVh
# bmdAc21hcnR4LmNvbQAKCRDf8iPWs/7LnIw6D/sECFML6dIDSckhKe1kRBT2oXRd
# lYz4/RRdxPJIJP0zS0yLYXd2d5vHzXdC3hETv7//QiWB2OP/UQnsZ70JCgF4DxIq
# bGL4BUHgaQmyUsyIQXceFznJJQOLs5DczDFJBR2zlQbu3YOAGeNJJmfxmVIEPfcL
# w5NK++g3nVZ3pLJBNblDBUwIW5uoOj6z85rCTc6pvddoTBcAqS0er1aTkuyx9jbB
# VsDdFNJumF6+VnE6QUITHX2knr3UmXc5dfXCJi4CLKRfx3nyK8vYydNawhPtobGD
# 0G5MZAPLO3JxdM67EccKj3I/kcAXU4iHu7rV5AscSI/rlfneGjfCup2Xd0we1GCR
# TD07AVDRuW+cS76nEtvDSRj6+8KarZEa3lVbvoPaXIazoHg3GjKylIMTAcGjFlzL
# AnGornOEZSfwNxT3BRvNHFdUNdA9ICZ90sEpWjeu80UNOT2JASOB6JE2VJBFnW81
# 4gaoIT74hpI8H2k/x3R8REPnw+SLMI+7VpcA2XcXuOQOdfk0+8zlvxPsJRBaKBzS
# d2es+CpUcmBxZdEQNOi905qxfCFLOhwcstJXyCvFQBp4f8l2SJfIE4liI29qpuma
# hubbOEo/EAVe8ywToHSYj2RU5hnj6gu0n3hvSeye76hS/K+bfvI+HZ3AX+rcRmP8
# vX6Vqs4wdNl5khEnNg==
# =ixu5
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 09 Oct 2023 20:08:45 EDT
# gpg: using RSA key 685D0220DC264828152E57C2DFF223D6B3FECB9C
# gpg: issuer "yong.huang@smartx.com"
# gpg: Good signature from "Yong Huang <yong.huang@smartx.com>" [unknown]
# gpg: WARNING: The key's User ID is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 685D 0220 DC26 4828 152E 57C2 DFF2 23D6 B3FE CB9C

* tag 'dirtylimit-dirtyrate-pull-request-20231010' of https://github.com/newfriday/qemu:
migration/dirtyrate: use QEMU_CLOCK_HOST to report start-time
migration/calc-dirty-rate: millisecond-granularity period

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

migration/calc-dirty-rate: millisecond-granularity period

This patch allows to measure dirty page rate for
sub-second intervals of time. An optional argument is
introduced -- calc-time-unit. For exa

migration/calc-dirty-rate: millisecond-granularity period

This patch allows to measure dirty page rate for
sub-second intervals of time. An optional argument is
introduced -- calc-time-unit. For example:
{"execute": "calc-dirty-rate", "arguments":
{"calc-time": 500, "calc-time-unit": "millisecond"} }

Millisecond granularity allows to make predictions whether
migration will succeed or not. To do this, calculate dirty
rate with calc-time set to max allowed downtime (e.g. 300ms),
convert measured rate into volume of dirtied memory,
and divide by network throughput. If the value is lower
than max allowed downtime, then migration will converge.

Measurement results for single thread randomly writing to
a 1/4/24GiB memory region:

+----------------+-----------------------------------------------+
| calc-time | dirty rate MiB/s |
| (milliseconds) +----------------+---------------+--------------+
| | theoretical | page-sampling | dirty-bitmap |
| | (at 3M wr/sec) | | |
+----------------+----------------+---------------+--------------+
| 1GiB |
+----------------+----------------+---------------+--------------+
| 100 | 6996 | 7100 | 3192 |
| 200 | 4606 | 4660 | 2655 |
| 300 | 3305 | 3280 | 2371 |
| 400 | 2534 | 2525 | 2154 |
| 500 | 2041 | 2044 | 1871 |
| 750 | 1365 | 1341 | 1358 |
| 1000 | 1024 | 1052 | 1025 |
| 1500 | 683 | 678 | 684 |
| 2000 | 512 | 507 | 513 |
+----------------+----------------+---------------+--------------+
| 4GiB |
+----------------+----------------+---------------+--------------+
| 100 | 10232 | 8880 | 4070 |
| 200 | 8954 | 8049 | 3195 |
| 300 | 7889 | 7193 | 2881 |
| 400 | 6996 | 6530 | 2700 |
| 500 | 6245 | 5772 | 2312 |
| 750 | 4829 | 4586 | 2465 |
| 1000 | 3865 | 3780 | 2178 |
| 1500 | 2694 | 2633 | 2004 |
| 2000 | 2041 | 2031 | 1789 |
+----------------+----------------+---------------+--------------+
| 24GiB |
+----------------+----------------+---------------+--------------+
| 100 | 11495 | 8640 | 5597 |
| 200 | 11226 | 8616 | 3527 |
| 300 | 10965 | 8386 | 2355 |
| 400 | 10713 | 8370 | 2179 |
| 500 | 10469 | 8196 | 2098 |
| 750 | 9890 | 7885 | 2556 |
| 1000 | 9354 | 7506 | 2084 |
| 1500 | 8397 | 6944 | 2075 |
| 2000 | 7574 | 6402 | 2062 |
+----------------+----------------+---------------+--------------+

Theoretical values are computed according to the following formula:
size * (1 - (1-(4096/size))^(time*wps)) / (time * 2^20),
where size is in bytes, time is in seconds, and wps is number of
writes per second.

Signed-off-by: Andrei Gudkov <gudkov.andrei@huawei.com>
Reviewed-by: Hyman Huang <yong.huang@smartx.com>
Message-Id: <d802e6b8053eb60fbec1a784cf86f67d9528e0a8.1693895970.git.gudkov.andrei@huawei.com>
Signed-off-by: Hyman Huang <yong.huang@smartx.com>

show more ...

Merge tag 'pull-migration-20220720c' of https://gitlab.com/dagrh/qemu into staging

Migration pull 2022-07-20

This replaces yesterdays pull and:
a) Fixes some test build errors without TLS
b) Re

Merge tag 'pull-migration-20220720c' of https://gitlab.com/dagrh/qemu into staging

Migration pull 2022-07-20

This replaces yesterdays pull and:
a) Fixes some test build errors without TLS
b) Reenabled the zlib acceleration on s390
now that we have Ilya's fix

Hyman's dirty page rate limit set
Ilya's fix for zlib vs migration
Peter's postcopy-preempt
Cleanup from Dan
zero-copy tidy ups from Leo
multifd doc fix from Juan
Revert disable of zlib acceleration on s390x

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# gpg: Signature made Wed 20 Jul 2022 12:18:56 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* tag 'pull-migration-20220720c' of https://gitlab.com/dagrh/qemu: (30 commits)
Revert "gitlab: disable accelerated zlib for s390x"
migration: Avoid false-positive on non-supported scenarios for zero-copy-send
multifd: Document the locking of MultiFD{Send/Recv}Params
migration/multifd: Report to user when zerocopy not working
Add dirty-sync-missed-zero-copy migration stat
QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
migration: remove unreachable code after reading data
tests: Add postcopy preempt tests
tests: Add postcopy tls recovery migration test
tests: Add postcopy tls migration test
tests: Move MigrateCommon upper
migration: Respect postcopy request order in preemption mode
migration: Enable TLS for preempt channel
migration: Export tls-[creds|hostname|authz] params to cmdline too
migration: Add helpers to detect TLS capability
migration: Add property x-postcopy-preempt-break-huge
migration: Create the postcopy preempt channel asynchronously
migration: Postcopy recover with preempt enabled
migration: Postcopy preemption enablement
migration: Postcopy preemption preparation on channel creation
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration/dirtyrate: Refactor dirty page rate calculation

abstract out dirty log change logic into function
global_dirty_log_change.

abstract out dirty page rate calculation logic via
dirty-ring in

migration/dirtyrate: Refactor dirty page rate calculation

abstract out dirty log change logic into function
global_dirty_log_change.

abstract out dirty page rate calculation logic via
dirty-ring into function vcpu_calculate_dirtyrate.

abstract out mathematical dirty page rate calculation
into do_calculate_dirtyrate, decouple it from DirtyStat.

rename set_sample_page_period to dirty_stat_wait, which
is well-understood and will be reused in dirtylimit.

handle cpu hotplug/unplug scenario during measurement of
dirty page rate.

export util functions outside migration.

Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
Reviewed-by: Peter Xu <peterx@redhat.com>
Message-Id: <7b6f6f4748d5b3d017b31a0429e630229ae97538.1656177590.git.huangy81@chinatelecom.cn>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211031-pull-request' into staging

Migration Pull request

Hi

this includes pending bits of migration patches.

- virtio-mem suppo

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211031-pull-request' into staging

Migration Pull request

Hi

this includes pending bits of migration patches.

- virtio-mem support by David Hildenbrand
- dirtyrate improvements by Hyman Huang
- fix rdma wrid by Li Zhijian
- dump-guest-memory fixes by Peter Xu

Pleas apply.

Thanks, Juan.

# gpg: Signature made Mon 01 Nov 2021 06:03:44 PM EDT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]

* remotes/juanquintela/tags/migration-20211031-pull-request:
migration/dirtyrate: implement dirty-bitmap dirtyrate calculation
memory: introduce total_dirty_pages to stat dirty pages
migration/ram: Handle RAMBlocks with a RamDiscardManager on background snapshots
migration/ram: Factor out populating pages readable in ram_block_populate_pages()
migration: Simplify alignment and alignment checks
migration/postcopy: Handle RAMBlocks with a RamDiscardManager on the destination
virtio-mem: Drop precopy notifier
migration/ram: Handle RAMBlocks with a RamDiscardManager on the migration source
virtio-mem: Implement replay_discarded RamDiscardManager callback
memory: Introduce replay_discarded callback for RamDiscardManager
dump-guest-memory: Block live migration
migration: Add migrate_add_blocker_internal()
migration: Make migration blocker work for snapshots too
migration/dirtyrate: implement dirty-ring dirtyrate calculation
migration/dirtyrate: move init step of calculation to main thread
migration/dirtyrate: adjust order of registering thread
migration/dirtyrate: introduce struct and adjust DirtyRateStat
memory: make global_dirty_tracking a bitmask
KVM: introduce dirty_pages and kvm_dirty_ring_enabled
migration/rdma: Fix out of order wrid

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

migration/dirtyrate: introduce struct and adjust DirtyRateStat

introduce "DirtyRateMeasureMode" to specify what method should be
used to calculate dirty rate, introduce "DirtyRateVcpu" to store
dirt

migration/dirtyrate: introduce struct and adjust DirtyRateStat

introduce "DirtyRateMeasureMode" to specify what method should be
used to calculate dirty rate, introduce "DirtyRateVcpu" to store
dirty rate for each vcpu.

use union to store stat data of specific mode

Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
Message-Id: <661c98c40f40e163aa58334337af8f3ddf41316a.1624040308.git.huangy81@chinatelecom.cn>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20210609a' into staging

Migration pull for 2021-06-09

Yank crash fix from Leo
RDMA fix from Li
mptcp support from me
dirty-

Merge remote-tracking branch 'remotes/dgilbert-gitlab/tags/pull-migration-20210609a' into staging

Migration pull for 2021-06-09

Yank crash fix from Leo
RDMA fix from Li
mptcp support from me
dirty-rate changes from Hyman and Peter

(Note I've switched to the gitlab I've been using for virtiofs pulls)

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

# gpg: Signature made Wed 09 Jun 2021 15:40:01 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert-gitlab/tags/pull-migration-20210609a:
hmp: Add "calc_dirty_rate" and "info dirty_rate" cmds
migration/dirtyrate: make sample page count configurable
sockets: Support multipath TCP
migration/socket: Close the listener at the end
migration: Add cleanup hook for inwards migration
io/net-listener: Call the notifier during finalize
channel-socket: Only set CLOEXEC if we have space for fds
migration/rdma: Fix cm event use after free
yank: Unregister function when using TLS migration

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration/dirtyrate: make sample page count configurable

introduce optional sample-pages argument in calc-dirty-rate,
making sample page count per GB configurable so that more
accurate dirtyrate can

migration/dirtyrate: make sample page count configurable

introduce optional sample-pages argument in calc-dirty-rate,
making sample page count per GB configurable so that more
accurate dirtyrate can be calculated.

Signed-off-by: Hyman Huang(黄勇) <huangy81@chinatelecom.cn>
Message-Id: <3103453a3b2796f929269c99a6ad81a9a7f1f405.1623027729.git.huangy81@chinatelecom.cn>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Wrapped a couple of long lines

show more ...

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20200925a' into staging

Migration and virtiofsd pull

Chuan Zheng's Dirtyrate and TLS changes, with small fixes from Dov and
Luaren

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20200925a' into staging

Migration and virtiofsd pull

Chuan Zheng's Dirtyrate and TLS changes, with small fixes from Dov and
Luarent.
Small virtiofs changes from Harry, Stefan, Vivek and Jiachen.
One HMP/monitor rework from me.

# gpg: Signature made Fri 25 Sep 2020 13:03:50 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20200925a: (26 commits)
virtiofsd: Add -o allow_direct_io|no_allow_direct_io options
virtiofsd: Used glib "shared" thread pool
virtiofsd: document cache=auto default
monitor: Use LOCK_GUARD macros
migration/tls: add trace points for multifd-tls
migration/tls: add support for multifd tls-handshake
migration/tls: extract cleanup function for common-use
migration/tls: add tls_hostname into MultiFDSendParams
migration/tls: extract migration_tls_client_create for common-use
migration/tls: save hostname into MigrationState
migration: increase max-bandwidth to 128 MiB/s (1 Gib/s)
migration: Truncate state file in xen-save-devices-state
migration/dirtyrate: Add trace_calls to make it easier to debug
migration/dirtyrate: Implement qmp_cal_dirty_rate()/qmp_get_dirty_rate() function
migration/dirtyrate: Implement calculate_dirtyrate() function
migration/dirtyrate: Implement set_sample_page_period() and is_sample_period_valid()
migration/dirtyrate: skip sampling ramblock with size below MIN_RAMBLOCK_SIZE
migration/dirtyrate: Compare page hash results for recorded sampled page
migration/dirtyrate: Record hash results for each sampled page
migration/dirtyrate: move RAMBLOCK_FOREACH_MIGRATABLE into ram.h
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration/dirtyrate: Implement set_sample_page_period() and is_sample_period_valid()

Implement is_sample_period_valid() to check if the sample period is vaild and
do set_sample_page_period() to slee

migration/dirtyrate: Implement set_sample_page_period() and is_sample_period_valid()

Implement is_sample_period_valid() to check if the sample period is vaild and
do set_sample_page_period() to sleep specific time between sample actions.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Message-Id: <1600237327-33618-10-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration/dirtyrate: skip sampling ramblock with size below MIN_RAMBLOCK_SIZE

In order to sample real RAM, skip ramblock with size below MIN_RAMBLOCK_SIZE
which is set as 128M.

Signed-off-by: Chuan

migration/dirtyrate: skip sampling ramblock with size below MIN_RAMBLOCK_SIZE

In order to sample real RAM, skip ramblock with size below MIN_RAMBLOCK_SIZE
which is set as 128M.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Message-Id: <1600237327-33618-9-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration/dirtyrate: Add dirtyrate statistics series functions

Add dirtyrate statistics functions to record/update dirtyrate info.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: Dr

migration/dirtyrate: Add dirtyrate statistics series functions

Add dirtyrate statistics functions to record/update dirtyrate info.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Message-Id: <1600237327-33618-5-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration/dirtyrate: Add RamblockDirtyInfo to store sampled page info

Add RamblockDirtyInfo to store sampled page info of each ramblock.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-

migration/dirtyrate: Add RamblockDirtyInfo to store sampled page info

Add RamblockDirtyInfo to store sampled page info of each ramblock.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Message-Id: <1600237327-33618-4-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration/dirtyrate: setup up query-dirtyrate framwork

Add get_dirtyrate_thread() functions to setup query-dirtyrate
framework.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Signed-off-by: Yan

migration/dirtyrate: setup up query-dirtyrate framwork

Add get_dirtyrate_thread() functions to setup query-dirtyrate
framework.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Signed-off-by: YanYing Zhuang <ann.zhuangyanying@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
Message-Id: <1600237327-33618-2-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...