Merge tag 'mem-2024-09-24' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Kconfig fix for virtio-based memory devices
- vi

Merge tag 'mem-2024-09-24' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Kconfig fix for virtio-based memory devices
- virtio-mem support for suspend+wake-up with plugged memory
- hostmem fix when specifying "merge=off"

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAmbyikMRHGRhdmlkQHJl
# ZGhhdC5jb20ACgkQTd4Q9wD/g1q6MBAAitNST73Shc+j327WvRLHQDkzkAlIYm+M
# E8NqtDiV11h7A0eNVu+5BkY/ejtY0Fduae3nxIkrHjK20eHHpiNPUp3hBNIhkKs3
# vlSaU8FLGdt58CteMGcLYsP2E32WNNTckaFGwGjDmyUEfk+Gug4r/rJAZXDfuuLV
# 083I0/MuUF+ozPA0c2MrOwhoBPerg3a5aflVpbgPwGNrT9BHMjo62Q5QzG3U7mxr
# HnlLAScSXsYg2z+d5XLXkKLAiZ4C7UN4vfUAOZwqkfs7IFUTtFO/ev6e7VZI747n
# XhAqOAKzLqPu7tBPZJIC6jwZAUIv5yM0/v5qhVvVVdu7H0ZMtSCXyvCVtnT25Rsn
# yiA+XvCOb7yQ3hRbBIi60IzjNYfWbvw+oTVIDfXkG35TeNf4ZdjWtAiUmw9s5U9Q
# z0tINsD7VlSkbh5h3PkFw1+xagIuJAVkp673HHTtQsg+xgYK2ur5jhhWJdJlnpzA
# 77CAu07UaqU39ssnC2zeGG1eNRA4uzjwQtREzqH2jMfkw/7UuUeXMF+v/fEuLn6w
# JneSMq/a0gmD42HNae0Y40cn2Akfj6+wFu1rW3djF8F6TeLUSssQhbQSHCMwGoOg
# qX7O/3SeSRzlnp3Zyx9Tr7s+BkMz0EGGDe17GQwTQUX2t5wR5iXoGqpKZgOBA8En
# 6uUIcjBUckc=
# =PExj
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 24 Sep 2024 10:45:39 BST
# gpg: using RSA key 1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A
# gpg: issuer "david@redhat.com"
# gpg: Good signature from "David Hildenbrand <david@redhat.com>" [marginal]
# gpg: aka "David Hildenbrand <davidhildenbrand@gmail.com>" [full]
# gpg: aka "David Hildenbrand <hildenbr@in.tum.de>" [unknown]
# Primary key fingerprint: 1BD9 CAAD 735C 4C3A 460D FCCA 4DDE 10F7 00FF 835A

* tag 'mem-2024-09-24' of https://github.com/davidhildenbrand/qemu:
hostmem: Apply merge property after the memory region is initialized
virtio-mem: Add support for suspend+wake-up with plugged memory
virtio-mem: Use new Resettable framework instead of LegacyReset
reset: Add RESET_TYPE_WAKEUP
reset: Use ResetType for qemu_devices_reset() and MachineClass::reset()
virtio: kconfig: memory devices are PCI only

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

virtio-mem: Use new Resettable framework instead of LegacyReset

LegacyReset does not pass ResetType to the reset callback method, which
the new Resettable framework uses. Due to this, virtio-mem can

virtio-mem: Use new Resettable framework instead of LegacyReset

LegacyReset does not pass ResetType to the reset callback method, which
the new Resettable framework uses. Due to this, virtio-mem cannot use
the new RESET_TYPE_WAKEUP to skip the reset during wake-up from a
suspended state.

This patch adds overrides Resettable interface methods in VirtIOMEMClass
to use the new Resettable framework and replaces
qemu_[un]register_reset() calls with qemu_[un]register_resettable().

Message-ID: <20240904103722.946194-4-jmarcin@redhat.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Juraj Marcin <jmarcin@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>

show more ...

Merge tag 'mem-2023-10-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Support memory devices with multiple memslots
-

Merge tag 'mem-2023-10-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Support memory devices with multiple memslots
- Support memory devices that dynamically consume memslots
- Support memory devices that can automatically decide on the number of
memslots to use
- virtio-mem support for exposing memory dynamically via multiple
memslots
- Some required cleanups/refactorings

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAmUn+XMRHGRhdmlkQHJl
# ZGhhdC5jb20ACgkQTd4Q9wD/g1qDHA//T01suTa+uzrcoJHoMWN11S47WnAmbuTo
# vVakucLBPMJAa9xZeCy3OavXaVGpHkw+t6g3OFknof0LfQ5/j9iE3Q1PxURN7g5j
# SJ2WJXCoceM6T4TMhPvVvgEaYjFmESqZB5FZgedMT0QRyhAxMuF9pCkWhk1O3OAV
# JqQKqLFiGcv60AEuBYGZGzgiOUv8EJ5gKwRF4VOdyHIxqZDw1aZXzlcd4TzFZBQ7
# rwW/3ef+sFmUJdmfrSrqcIlQSRrqZ2w95xATDzLTIEEUT3SWqh/E95EZWIz1M0oQ
# NgWgFiLCR1KOj7bWFhLXT7IfyLh0mEysD+P/hY6QwQ4RewWG7EW5UK+JFswssdcZ
# rEj5XpHZzev/wx7hM4bWsoQ+VIvrH7j3uYGyWkcgYRbdDEkWDv2rsT23lwGYNhht
# oBsrdEBELRw6v4C8doq/+sCmHmuxUMqTGwbArCQVnB1XnLxOEkuqlnfq5MORkzNF
# fxbIRx+LRluOllC0HVaDQd8qxRq1+UC5WIpAcDcrouy4HGgi1onWKrXpgjIAbVyH
# M6cENkK7rnRk96gpeXdmrf0h9HqRciAOY8oUsFsvLyKBOCPBWDrLyOQEY5UoSdtD
# m4QpEVgywCy2z1uU/UObeT/UxJy/9EL/Zb+DHoEK06iEhwONoUJjEBYMJD38RMkk
# mwPTB4UAk9g=
# =s69t
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 12 Oct 2023 09:49:39 EDT
# gpg: using RSA key 1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A
# gpg: issuer "david@redhat.com"
# gpg: Good signature from "David Hildenbrand <david@redhat.com>" [unknown]
# gpg: aka "David Hildenbrand <davidhildenbrand@gmail.com>" [full]
# gpg: aka "David Hildenbrand <hildenbr@in.tum.de>" [unknown]
# gpg: WARNING: The key's User ID is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1BD9 CAAD 735C 4C3A 460D FCCA 4DDE 10F7 00FF 835A

* tag 'mem-2023-10-12' of https://github.com/davidhildenbrand/qemu:
virtio-mem: Mark memslot alias memory regions unmergeable
memory,vhost: Allow for marking memory device memory regions unmergeable
virtio-mem: Expose device memory dynamically via multiple memslots if enabled
virtio-mem: Update state to match bitmap as soon as it's been migrated
virtio-mem: Pass non-const VirtIOMEM via virtio_mem_range_cb
memory: Clarify mapping requirements for RamDiscardManager
memory-device,vhost: Support automatic decision on the number of memslots
vhost: Add vhost_get_max_memslots()
kvm: Add stub for kvm_get_max_memslots()
memory-device,vhost: Support memory devices that dynamically consume memslots
memory-device: Track required and actually used memslots in DeviceMemoryState
stubs: Rename qmp_memory_device.c to memory_device.c
memory-device: Support memory devices with multiple memslots
vhost: Return number of free memslots
kvm: Return number of free memslots
softmmu/physmem: Fixup qemu_ram_block_from_host() documentation
vhost: Remove vhost_backend_can_merge() callback
vhost: Rework memslot filtering and fix "used_memslot" tracking

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

show more ...

virtio-mem: Expose device memory dynamically via multiple memslots if enabled

Having large virtio-mem devices that only expose little memory to a VM
is currently a problem: we map the whole sparse m

virtio-mem: Expose device memory dynamically via multiple memslots if enabled

Having large virtio-mem devices that only expose little memory to a VM
is currently a problem: we map the whole sparse memory region into the
guest using a single memslot, resulting in one gigantic memslot in KVM.
KVM allocates metadata for the whole memslot, which can result in quite
some memory waste.

Assuming we have a 1 TiB virtio-mem device and only expose little (e.g.,
1 GiB) memory, we would create a single 1 TiB memslot and KVM has to
allocate metadata for that 1 TiB memslot: on x86, this implies allocating
a significant amount of memory for metadata:

(1) RMAP: 8 bytes per 4 KiB, 8 bytes per 2 MiB, 8 bytes per 1 GiB
-> For 1 TiB: 2147483648 + 4194304 + 8192 = ~ 2 GiB (0.2 %)

With the TDP MMU (cat /sys/module/kvm/parameters/tdp_mmu) this gets
allocated lazily when required for nested VMs
(2) gfn_track: 2 bytes per 4 KiB
-> For 1 TiB: 536870912 = ~512 MiB (0.05 %)
(3) lpage_info: 4 bytes per 2 MiB, 4 bytes per 1 GiB
-> For 1 TiB: 2097152 + 4096 = ~2 MiB (0.0002 %)
(4) 2x dirty bitmaps for tracking: 2x 1 bit per 4 KiB page
-> For 1 TiB: 536870912 = 64 MiB (0.006 %)

So we primarily care about (1) and (2). The bad thing is, that the
memory consumption *doubles* once SMM is enabled, because we create the
memslot once for !SMM and once for SMM.

Having a 1 TiB memslot without the TDP MMU consumes around:
* With SMM: 5 GiB
* Without SMM: 2.5 GiB
Having a 1 TiB memslot with the TDP MMU consumes around:
* With SMM: 1 GiB
* Without SMM: 512 MiB

... and that's really something we want to optimize, to be able to just
start a VM with small boot memory (e.g., 4 GiB) and a virtio-mem device
that can grow very large (e.g., 1 TiB).

Consequently, using multiple memslots and only mapping the memslots we
really need can significantly reduce memory waste and speed up
memslot-related operations. Let's expose the sparse RAM memory region using
multiple memslots, mapping only the memslots we currently need into our
device memory region container.

The feature can be enabled using "dynamic-memslots=on" and requires
"unplugged-inaccessible=on", which is nowadays the default.

Once enabled, we'll auto-detect the number of memslots to use based on the
memslot limit provided by the core. We'll use at most 1 memslot per
gigabyte. Note that our global limit of memslots accross all memory devices
is currently set to 256: even with multiple large virtio-mem devices,
we'd still have a sane limit on the number of memslots used.

The default is to not dynamically map memslot for now
("dynamic-memslots=off"). The optimization must be enabled manually,
because some vhost setups (e.g., hotplug of vhost-user devices) might be
problematic until we support more memslots especially in vhost-user backends.

Note that "dynamic-memslots=on" is just a hint that multiple memslots
*may* be used for internal optimizations, not that multiple memslots
*must* be used. The actual number of memslots that are used is an
internal detail: for example, once memslot metadata is no longer an
issue, we could simply stop optimizing for that. Migration source and
destination can differ on the setting of "dynamic-memslots".

Message-ID: <20230926185738.277351-17-david@redhat.com>
Reviewed-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>

show more ...

Merge tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Memory device cleanups (especially around machin

Merge tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu into staging

Hi,

"Host Memory Backends" and "Memory devices" queue ("mem"):
- Memory device cleanups (especially around machine initialization)
- "x-ignore-shared" migration support for virtio-mem
- Add an abstract virtio-md-pci device as a common parent for
virtio-mem-pci and virtio-pmem-pci (virtio based memory devices)
- Device unplug support for virtio-mem-pci

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAmSuYAQRHGRhdmlkQHJl
# ZGhhdC5jb20ACgkQTd4Q9wD/g1od9A/9HXT8IqKGup9is7P/mpobPWXczRGZ5sEg
# /q21PzX6crr9aFa+fYRF/Dlm3G/cSMOVXFRKGz3royLjsvaEj/veEewfKF8KWbBf
# eIS9udQTOwoD2kAhcv3pm0SwSJoVizpw2z7IodGVKE6iZxTXsmDksqQuFbrvVLSh
# 2wtP4lizEXco/YsiCoAnStj2QtXBcHw7Ua7W2cDzxFmL+1pM5w3rjQ1ydCNz3bSG
# l4CXXs1i8OmOZbFN78F/E9SEkzQnAuHSO0Sc1aeAJkwVzOt2lj/YMgt0jHjAY0at
# pheWZ5pEE6hnQP740YXpt4Y6IIgO22pH23dLhq9A2reyRnwjt830uObHi3qAE8kB
# KR+ZQ+Z5bI6ZNB/EFiUsC1dFsr2fF20zQlO02MctyJ+lUG6p3gpvwsGScQxt+zdF
# QlkiSecGErYwC+nZ529SQB4gSEJTCjd/STDoidVYnZazdStaOaSyft02xRNzBPW/
# OnOY+6ZxZK6R11KfwGjnsftrovQIP3Pqi9TXGzW2xVlkWJHqlicy6G3ZfceTTlj9
# Gg2Ue694Wr1r4PDV2XlYcZ1IPLjSy5Msp5V2wERRrp3OItxnvegvTevQN7USEHC+
# BPGNMu11jriSY2pE5BSFN0hfGOvuvsk3GreLJiHFUXoje6gzAynuLjCN/CHdIVyK
# 5i0AwdZ+xcA=
# =ch6m
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 12 Jul 2023 09:10:44 AM BST
# gpg: using RSA key 1BD9CAAD735C4C3A460DFCCA4DDE10F700FF835A
# gpg: issuer "david@redhat.com"
# gpg: Good signature from "David Hildenbrand <david@redhat.com>" [unknown]
# gpg: aka "David Hildenbrand <davidhildenbrand@gmail.com>" [undefined]
# gpg: aka "David Hildenbrand <hildenbr@in.tum.de>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1BD9 CAAD 735C 4C3A 460D FCCA 4DDE 10F7 00FF 835A

* tag 'mem-2023-07-12' of https://github.com/davidhildenbrand/qemu: (21 commits)
virtio-mem-pci: Device unplug support
virtio-mem: Prepare for device unplug support
virtio-md-pci: Support unplug requests for compatible devices
virtio-md-pci: Handle unplug of virtio based memory devices
arm/virt: Use virtio-md-pci (un)plug functions
pc: Factor out (un)plug handling of virtio-md-pci devices
virtio-md-pci: New parent type for virtio-mem-pci and virtio-pmem-pci
virtio-mem: Support "x-ignore-shared" migration
migration/ram: Expose ramblock_is_ignored() as migrate_ram_is_ignored()
virtio-mem: Skip most of virtio_mem_unplug_all() without plugged memory
softmmu/physmem: Warn with ram_block_discard_range() on MAP_PRIVATE file mapping
memory-device: Track used region size in DeviceMemoryState
memory-device: Refactor memory_device_pre_plug()
hw/i386/pc: Remove PC_MACHINE_DEVMEM_REGION_SIZE
hw/i386/acpi-build: Rely on machine->device_memory when building SRAT
hw/i386/pc: Use machine_memory_devices_init()
hw/loongarch/virt: Use machine_memory_devices_init()
hw/ppc/spapr: Use machine_memory_devices_init()
hw/arm/virt: Use machine_memory_devices_init()
memory-device: Introduce machine_memory_devices_init()
...

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

virtio-mem: Prepare for device unplug support

In many cases, blindly unplugging a virtio-mem device is problematic. We
can only safely remove a device once:
* The guest is not expecting to be able t

virtio-mem: Prepare for device unplug support

In many cases, blindly unplugging a virtio-mem device is problematic. We
can only safely remove a device once:
* The guest is not expecting to be able to read unplugged memory
(unplugged-inaccessible == on)
* The virtio-mem device does not have memory plugged (size == 0)
* The virtio-mem device does not have outstanding requests to the VM to
plug memory (requested-size == 0)

So let's add a callback to the virtio-mem device class to check for that.
We'll wire-up virtio-mem-pci next.

Message-ID: <20230711153445.514112-7-david@redhat.com>
Tested-by: Mario Casquero <mcasquer@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>

show more ...

Merge tag 'migration-20230206-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request

In this try
- rebase to latest upstream
- same than previous patch
- fix com

Merge tag 'migration-20230206-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request

In this try
- rebase to latest upstream
- same than previous patch
- fix compilation on non linux (userfaultfd.h) (me)
- query-migrationthreads (jiang)
- fix race on reading MultiFDPages_t.block (zhenzhong)
- fix flush of zero copy page send reuest (zhenzhong)

Please apply.

Previous try:
It includes:
- David Hildenbrand fixes for virtio-men
- David Gilbert canary to detect problems
- Fix for rdma return values (Fiona)
- Peter Xu uffd_open fixes
- Peter Xu show right downtime for postcopy
- manish.mishra msg fix fixes
- my vfio changes.

Please apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmPhobYACgkQ9IfvGFhy
# 1yMNaA/9EHDPqrI1HL/VkJG4nNOOsQR7RbburXEberZOzvLjnqpjUD3Ls9qV6rx+
# ieHa5T4imYJFk72Wa5vx4r1/dCjtJD2W6jg5+/0nTvYAHrs1U1VRqpuTr0HiXdbJ
# ZLLCnW5eDyO3eMaOX0MUkgHgL0FNkc/Lq5ViCTFsMu9O9xMuDLLdAC3cdvslKuOu
# X1gKByr9jT817Y9e36amYmRaJKC6Cr/PIekNVFu12HBW79pPusLX8KWEf4RBw4HR
# sPwTvMCR/BwZ0+2Lppan60G5rt/ZxDu40oU7y+RHlfWqevl4hDM84/nhjMvEgzc5
# a4Ahe2ERGLwwnC8z3l7v9+pEzSGzDoPcnRGvZcpUpk68wTDtxd5Bdq8CwmNUfL07
# VzWcYpH0yvmwjBba9jfn9fAVgnG5rVp558XcYLIII3wEToty3UDtm43wSdj2CGr6
# cu+IPAp+n/I5G9SRYBTU9ozJz45ttnEe0hxUtZ4I3MuhzHi1VEDAqTWM/X0LyS41
# TB3Y5B2KKpJYbPyZEH4nyTeetR2k7alTFzahCgKqVfOgL0nJx54petjS1K+B1P72
# g6lhP9WnQ33W+M8S7J/aGEaDJd1lFyFB2Rdjn2ZZnASH/fR9j0mFmXWvulXtjFNp
# Sfim3887+Iv4Uzw4VWEe3mM5Ypi/Ba2CmuTjy/pM08Ey8X1Qs5o=
# =ZQbR
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 07 Feb 2023 00:56:22 GMT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* tag 'migration-20230206-pull-request' of https://gitlab.com/juan.quintela/qemu: (30 commits)
migration: save/delete migration thread info
migration: Introduce interface query-migrationthreads
multifd: Fix flush of zero copy page send request
multifd: Fix a race on reading MultiFDPages_t.block
migration: check magic value for deciding the mapping of channels
io: Add support for MSG_PEEK for socket channel
migration/dirtyrate: Show sample pages only in page-sampling mode
migration: Perform vmsd structure check during tests
migration: Add canary to VMSTATE_END_OF_LIST
migration/rdma: fix return value for qio_channel_rdma_{readv,writev}
migration: Show downtime during postcopy phase
virtio-mem: Proper support for preallocation with migration
virtio-mem: Migrate immutable properties early
virtio-mem: Fail if a memory backend with "prealloc=on" is specified
migration/ram: Factor out check for advised postcopy
migration/vmstate: Introduce VMSTATE_WITH_TMP_TEST() and VMSTATE_BITMAP_TEST()
migration/savevm: Allow immutable device state to be migrated early (i.e., before RAM)
migration/savevm: Prepare vmdesc json writer in qemu_savevm_state_setup()
migration/savevm: Move more savevm handling into vmstate_save()
migration/ram: Optimize ram_write_tracking_start() for RamDiscardManager
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

virtio-mem: Migrate immutable properties early

The bitmap and the size are immutable while migration is active: see
virtio_mem_is_busy(). We can migrate this information early, before
migrating any

virtio-mem: Migrate immutable properties early

The bitmap and the size are immutable while migration is active: see
virtio_mem_is_busy(). We can migrate this information early, before
migrating any actual RAM content. Further, all information we need for
sanity checks is immutable as well.

Having this information in place early will, for example, allow for
properly preallocating memory before touching these memory locations
during RAM migration: this way, we can make sure that all memory was
actually preallocated and that any user errors (e.g., insufficient
hugetlb pages) can be handled gracefully.

In contrast, usable_region_size and requested_size can theoretically
still be modified on the source while the VM is running. Keep migrating
these properties the usual, late, way.

Use a new device property to keep behavior of compat machines
unmodified.

Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>S
Signed-off-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge tag 'for_upstream' of git://git.kernel.org/pub/scm/virt/kvm/mst/qemu into staging

virtio,pci,pc: features,fixes,cleanups

New virtio mem options.
A vhost-user cleanup.
Control over smbios entr

Merge tag 'for_upstream' of git://git.kernel.org/pub/scm/virt/kvm/mst/qemu into staging

virtio,pci,pc: features,fixes,cleanups

New virtio mem options.
A vhost-user cleanup.
Control over smbios entry point type.
Config interrupt support for vdpa.
Fixes, cleanups all over the place.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

# gpg: Signature made Fri 07 Jan 2022 04:30:41 PM PST
# gpg: using RSA key 5D09FD0871C8F85B94CA8A0D281F0DB8D28D5469
# gpg: issuer "mst@redhat.com"
# gpg: Good signature from "Michael S. Tsirkin <mst@kernel.org>" [undefined]
# gpg: aka "Michael S. Tsirkin <mst@redhat.com>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 0270 606B 6F3C DF3D 0B17 0970 C350 3912 AFBE 8E67
# Subkey fingerprint: 5D09 FD08 71C8 F85B 94CA 8A0D 281F 0DB8 D28D 5469

* tag 'for_upstream' of git://git.kernel.org/pub/scm/virt/kvm/mst/qemu: (55 commits)
tests: acpi: Add updated TPM related tables
acpi: tpm: Add missing device identification objects
tests: acpi: prepare for updated TPM related tables
virtio/vhost-vsock: don't double close vhostfd, remove redundant cleanup
hw/scsi/vhost-scsi: don't double close vhostfd on error
hw/scsi/vhost-scsi: don't leak vqs on error
docs: reSTify virtio-balloon-stats documentation and move to docs/interop
hw/i386/pc: Add missing property descriptions
acpihp: simplify acpi_pcihp_disable_root_bus
tests: acpi: SLIC: update expected blobs
tests: acpi: add SLIC table test
tests: acpi: whitelist expected blobs before changing them
acpi: fix QEMU crash when started with SLIC table
intel-iommu: correctly check passthrough during translation
virtio-mem: Set "unplugged-inaccessible=auto" for the 7.0 machine on x86
virtio-mem: Support VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE
linux-headers: sync VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE
MAINTAINERS: Add a separate entry for acpi/VIOT tables
virtio: signal after wrapping packed used_idx
virtio-mem: Support "prealloc=on" option
...

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

virtio-mem: Support VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE

With VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE, we signal the VM that reading
unplugged memory is not supported. We have to fail feature negotiation

virtio-mem: Support VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE

With VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE, we signal the VM that reading
unplugged memory is not supported. We have to fail feature negotiation
in case the guest does not support VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE.

First, VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE is required to properly handle
memory backends (or architectures) without support for the shared zeropage
in the hypervisor cleanly. Without the shared zeropage, even reading an
unpopulated virtual memory location can populate real memory and
consequently consume memory in the hypervisor. We have a guaranteed shared
zeropage only on MAP_PRIVATE anonymous memory.

Second, we want VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE to be the default
long-term as even populating the shared zeropage can be problematic: for
example, without THP support (possible) or without support for the shared
huge zeropage with THP (unlikely), the PTE page tables to hold the shared
zeropage entries can consume quite some memory that cannot be reclaimed
easily.

Third, there are other optimizations+features (e.g., protection of
unplugged memory, reducing the total memory slot size and bitmap sizes)
that will require VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE.

We really only support x86 targets with virtio-mem for now (and
Linux similarly only support x86), but that might change soon, so prepare
for different targets already.

Add a new "unplugged-inaccessible" tristate property for x86 targets:
- "off" will keep VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE unset and legacy
guests working.
- "on" will set VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE and stop legacy guests
from using the device.
- "auto" selects the default based on support for the shared zeropage.

Warn in case the property is set to "off" and we don't have support for the
shared zeropage.

For existing compat machines, the property will default to "off", to
not change the behavior but eventually warn about a problematic setup.
Short-term, we'll set the property default to "auto" for new QEMU machines.
Mid-term, we'll set the property default to "on" for new QEMU machines.
Long-term, we'll deprecate the parameter and disallow legacy
guests completely.

The property has to match on the migration source and destination. "auto"
will result in the same VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE setting as long
as the qemu command line (esp. memdev) match -- so "auto" is good enough
for migration purposes and the parameter doesn't have to be migrated
explicitly.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20211217134039.29670-3-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...

virtio-mem: Support "prealloc=on" option

For scarce memory resources, such as hugetlb, we want to be able to
prealloc such memory resources in order to not crash later on access. On
simple user erro

virtio-mem: Support "prealloc=on" option

For scarce memory resources, such as hugetlb, we want to be able to
prealloc such memory resources in order to not crash later on access. On
simple user errors we could otherwise easily run out of memory resources
an crash the VM -- pretty much undesired.

For ordinary memory devices, such as DIMMs, we preallocate memory via the
memory backend for such use cases; however, with virtio-mem we're dealing
with sparse memory backends; preallocating the whole memory backend
destroys the whole purpose of virtio-mem.

Instead, we want to preallocate memory when actually exposing memory to the
VM dynamically, and fail plugging memory gracefully + warn the user in case
preallocation fails.

A common use case for hugetlb will be using "reserve=off,prealloc=off" for
the memory backend and "prealloc=on" for the virtio-mem device. This
way, no huge pages will be reserved for the process, but we can recover
if there are no actual huge pages when plugging memory. Libvirt is
already prepared for this.

Note that preallocation cannot protect from the OOM killer -- which
holds true for any kind of preallocation in QEMU. It's primarily useful
only for scarce memory resources such as hugetlb, or shared file-backed
memory. It's of little use for ordinary anonymous memory that can be
swapped, KSM merged, ... but we won't forbid it.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20211217134611.31172-9-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211031-pull-request' into staging

Migration Pull request

Hi

this includes pending bits of migration patches.

- virtio-mem suppo

Merge remote-tracking branch 'remotes/juanquintela/tags/migration-20211031-pull-request' into staging

Migration Pull request

Hi

this includes pending bits of migration patches.

- virtio-mem support by David Hildenbrand
- dirtyrate improvements by Hyman Huang
- fix rdma wrid by Li Zhijian
- dump-guest-memory fixes by Peter Xu

Pleas apply.

Thanks, Juan.

# gpg: Signature made Mon 01 Nov 2021 06:03:44 PM EDT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [full]

* remotes/juanquintela/tags/migration-20211031-pull-request:
migration/dirtyrate: implement dirty-bitmap dirtyrate calculation
memory: introduce total_dirty_pages to stat dirty pages
migration/ram: Handle RAMBlocks with a RamDiscardManager on background snapshots
migration/ram: Factor out populating pages readable in ram_block_populate_pages()
migration: Simplify alignment and alignment checks
migration/postcopy: Handle RAMBlocks with a RamDiscardManager on the destination
virtio-mem: Drop precopy notifier
migration/ram: Handle RAMBlocks with a RamDiscardManager on the migration source
virtio-mem: Implement replay_discarded RamDiscardManager callback
memory: Introduce replay_discarded callback for RamDiscardManager
dump-guest-memory: Block live migration
migration: Add migrate_add_blocker_internal()
migration: Make migration blocker work for snapshots too
migration/dirtyrate: implement dirty-ring dirtyrate calculation
migration/dirtyrate: move init step of calculation to main thread
migration/dirtyrate: adjust order of registering thread
migration/dirtyrate: introduce struct and adjust DirtyRateStat
memory: make global_dirty_tracking a bitmask
KVM: introduce dirty_pages and kvm_dirty_ring_enabled
migration/rdma: Fix out of order wrid

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

virtio-mem: Drop precopy notifier

Migration code now properly handles RAMBlocks which are indirectly managed
by a RamDiscardManager. No need for manual handling via the free page
optimization interf

virtio-mem: Drop precopy notifier

Migration code now properly handles RAMBlocks which are indirectly managed
by a RamDiscardManager. No need for manual handling via the free page
optimization interface, let's get rid of it.

Acked-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ehabkost-gl/tags/machine-next-pull-request' into staging

Machine queue, 2021-07-07

Deprecation:
* Deprecate pmem=on with non-DAX capable backend file
(Igor M

Merge remote-tracking branch 'remotes/ehabkost-gl/tags/machine-next-pull-request' into staging

Machine queue, 2021-07-07

Deprecation:
* Deprecate pmem=on with non-DAX capable backend file
(Igor Mammedov)

Feature:
* virtio-mem: vfio support (David Hildenbrand)

Cleanup:
* vmbus: Don't make QOM property registration conditional
(Eduardo Habkost)

# gpg: Signature made Thu 08 Jul 2021 20:55:04 BST
# gpg: using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6
# gpg: issuer "ehabkost@redhat.com"
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full]
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6

* remotes/ehabkost-gl/tags/machine-next-pull-request:
vfio: Disable only uncoordinated discards for VFIO_TYPE1 iommus
virtio-mem: Require only coordinated discards
softmmu/physmem: Extend ram_block_discard_(require|disable) by two discard types
softmmu/physmem: Don't use atomic operations in ram_block_discard_(disable|require)
vfio: Support for RamDiscardManager in the vIOMMU case
vfio: Sanity check maximum number of DMA mappings with RamDiscardManager
vfio: Query and store the maximum number of possible DMA mappings
vfio: Support for RamDiscardManager in the !vIOMMU case
virtio-mem: Implement RamDiscardManager interface
virtio-mem: Don't report errors when ram_block_discard_range() fails
virtio-mem: Factor out traversing unplugged ranges
memory: Helpers to copy/free a MemoryRegionSection
memory: Introduce RamDiscardManager for RAM memory regions
Deprecate pmem=on with non-DAX capable backend file
vmbus: Don't make QOM property registration conditional

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

virtio-mem: Implement RamDiscardManager interface

Let's properly notify when (un)plugging blocks, after discarding memory
and before allowing the guest to consume memory. Handle errors from
notifier

virtio-mem: Implement RamDiscardManager interface

Let's properly notify when (un)plugging blocks, after discarding memory
and before allowing the guest to consume memory. Handle errors from
notifiers gracefully (e.g., no remaining VFIO mappings) when plugging,
rolling back the change and telling the guest that the VM is busy.

One special case to take care of is replaying all notifications after
restoring the vmstate. The device starts out with all memory discarded,
so after loading the vmstate, we have to notify about all plugged
blocks.

Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Auger Eric <eric.auger@redhat.com>
Cc: Wei Yang <richard.weiyang@linux.alibaba.com>
Cc: teawater <teawaterz@linux.alibaba.com>
Cc: Marek Kedzierski <mkedzier@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20210413095531.25603-6-david@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ehabkost/tags/machine-next-pull-request' into staging

QOM queue, 2020-09-18

Fixes:
* Error value corrections (Markus Armbruster)
* Correct object_class_dynamic

Merge remote-tracking branch 'remotes/ehabkost/tags/machine-next-pull-request' into staging

QOM queue, 2020-09-18

Fixes:
* Error value corrections (Markus Armbruster)
* Correct object_class_dynamic_cast_assert() documentation (Eduardo Habkost)
* Ensure objects using QEMU_ALIGNED are properly aligned (Richard Henderson)

QOM cleanups (Eduardo Habkost):
* Rename some constants
* Simplify parameters of OBJECT_DECLARE* macros
* Additional DECLARE_*CHECKER* usage
* Additional OBJECT_DECLARE_TYPE usage
* Additional OBJECT_DECLARE_SIMPLE_TYPE usage

# gpg: Signature made Fri 18 Sep 2020 21:45:29 BST
# gpg: using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6
# gpg: issuer "ehabkost@redhat.com"
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full]
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6

* remotes/ehabkost/tags/machine-next-pull-request:
Use OBJECT_DECLARE_SIMPLE_TYPE when possible
Use OBJECT_DECLARE_TYPE when possible
qom: Remove module_obj_name parameter from OBJECT_DECLARE* macros
qom: Remove ParentClassType argument from OBJECT_DECLARE_SIMPLE_TYPE
scripts/codeconverter: Update to latest version
target/s390x: Set instance_align on S390CPU TypeInfo
target/riscv: Set instance_align on RISCVCPU TypeInfo
target/ppc: Set instance_align on PowerPCCPU TypeInfo
target/arm: Set instance_align on CPUARM TypeInfo
qom: Allow objects to be allocated with increased alignment
qom: Correct error values in two contracts
qom: Clean up object_property_get_enum()'s error value
qom: Correct object_class_dynamic_cast_assert() documentation
sifive: Use DECLARE_*CHECKER* macros
sifive: Move QOM typedefs and add missing includes
sifive_u: Rename memmap enum constants
sifive_e: Rename memmap enum constants

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qom: Remove module_obj_name parameter from OBJECT_DECLARE* macros

One of the goals of having less boilerplate on QOM declarations
is to avoid human error. Requiring an extra argument that is
never

qom: Remove module_obj_name parameter from OBJECT_DECLARE* macros

One of the goals of having less boilerplate on QOM declarations
is to avoid human error. Requiring an extra argument that is
never used is an opportunity for mistakes.

Remove the unused argument from OBJECT_DECLARE_TYPE and
OBJECT_DECLARE_SIMPLE_TYPE.

Coccinelle patch used to convert all users of the macros:

@@
declarer name OBJECT_DECLARE_TYPE;
identifier InstanceType, ClassType, lowercase, UPPERCASE;
@@
OBJECT_DECLARE_TYPE(InstanceType, ClassType,
- lowercase,
UPPERCASE);

@@
declarer name OBJECT_DECLARE_SIMPLE_TYPE;
identifier InstanceType, lowercase, UPPERCASE;
@@
OBJECT_DECLARE_SIMPLE_TYPE(InstanceType,
- lowercase,
UPPERCASE);

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Acked-by: Igor Mammedov <imammedo@redhat.com>
Acked-by: Paul Durrant <paul@xen.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200916182519.415636-4-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/ehabkost/tags/machine-next-pull-request' into staging

QOM boilerplate cleanup

Documentation build fix:
* memory: Remove kernel-doc comment marker (Eduardo Habk

Merge remote-tracking branch 'remotes/ehabkost/tags/machine-next-pull-request' into staging

QOM boilerplate cleanup

Documentation build fix:
* memory: Remove kernel-doc comment marker (Eduardo Habkost)

QOM cleanups:
* Rename QOM macros for consistency between
TYPE_* and type checking constants (Eduardo Habkost)

QOM new macros:
* OBJECT_DECLARE_* and OBJECT_DEFINE_* macros (Daniel P. Berrangé)
* DECLARE_*_CHECKER macros (Eduardo Habkost)

Automated QOM boilerplate changes:
* Automated changes to use DECLARE_*_CHECKER (Eduardo Habkost
* Automated changes to use OBJECT_DECLARE* (Eduardo Habkost)

# gpg: Signature made Thu 10 Sep 2020 19:17:49 BST
# gpg: using RSA key 5A322FD5ABC4D3DBACCFD1AA2807936F984DC5A6
# gpg: issuer "ehabkost@redhat.com"
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>" [full]
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF D1AA 2807 936F 984D C5A6

* remotes/ehabkost/tags/machine-next-pull-request: (33 commits)
virtio-vga: Use typedef name for instance_size
vhost-user-vga: Use typedef name for instance_size
xilinx_axienet: Use typedef name for instance_size
lpc_ich9: Use typedef name for instance_size
omap_intc: Use typedef name for instance_size
xilinx_axidma: Use typedef name for instance_size
tusb6010: Rename TUSB to TUSB6010
pc87312: Rename TYPE_PC87312_SUPERIO to TYPE_PC87312
vfio: Rename PCI_VFIO to VFIO_PCI
usb: Rename USB_SERIAL_DEV to USB_SERIAL
sabre: Rename SABRE_DEVICE to SABRE
rs6000_mc: Rename RS6000MC_DEVICE to RS6000MC
filter-rewriter: Rename FILTER_COLO_REWRITER to FILTER_REWRITER
esp: Rename ESP_STATE to ESP
ahci: Rename ICH_AHCI to ICH9_AHCI
vmgenid: Rename VMGENID_DEVICE to TYPE_VMGENID
vfio: Rename VFIO_AP_DEVICE_TYPE to TYPE_VFIO_AP_DEVICE
dev-smartcard-reader: Rename CCID_DEV_NAME to TYPE_USB_CCID_DEV
ap-device: Rename AP_DEVICE_TYPE to TYPE_AP_DEVICE
gpex: Fix type checking function name
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Use OBJECT_DECLARE_TYPE where possible

Replace DECLARE_OBJ_CHECKERS with OBJECT_DECLARE_TYPE where the
typedefs can be safely removed.

Generated running:

$ ./scripts/codeconverter/converter.py -i

Use OBJECT_DECLARE_TYPE where possible

Replace DECLARE_OBJ_CHECKERS with OBJECT_DECLARE_TYPE where the
typedefs can be safely removed.

Generated running:

$ ./scripts/codeconverter/converter.py -i \
--pattern=DeclareObjCheckers $(git grep -l '' -- '*.[ch]')

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200831210740.126168-16-ehabkost@redhat.com>
Message-Id: <20200831210740.126168-17-ehabkost@redhat.com>
Message-Id: <20200831210740.126168-18-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Use DECLARE_*CHECKER* macros

Generated using:

$ ./scripts/codeconverter/converter.py -i \
--pattern=TypeCheckMacro $(git grep -l '' -- '*.[ch]')

Reviewed-by: Daniel P. Berrangé <berrange@redha

Use DECLARE_*CHECKER* macros

Generated using:

$ ./scripts/codeconverter/converter.py -i \
--pattern=TypeCheckMacro $(git grep -l '' -- '*.[ch]')

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20200831210740.126168-12-ehabkost@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20200831210740.126168-13-ehabkost@redhat.com>
Message-Id: <20200831210740.126168-14-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Move QOM typedefs and add missing includes

Some typedefs and macros are defined after the type check macros.
This makes it difficult to automatically replace their
definitions with OBJECT_DECLARE_TY

Move QOM typedefs and add missing includes

Some typedefs and macros are defined after the type check macros.
This makes it difficult to automatically replace their
definitions with OBJECT_DECLARE_TYPE.

Patch generated using:

$ ./scripts/codeconverter/converter.py -i \
--pattern=QOMStructTypedefSplit $(git grep -l '' -- '*.[ch]')

which will split "typdef struct { ... } TypedefName"
declarations.

Followed by:

$ ./scripts/codeconverter/converter.py -i --pattern=MoveSymbols \
$(git grep -l '' -- '*.[ch]')

which will:
- move the typedefs and #defines above the type check macros
- add missing #include "qom/object.h" lines if necessary

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20200831210740.126168-9-ehabkost@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20200831210740.126168-10-ehabkost@redhat.com>
Message-Id: <20200831210740.126168-11-ehabkost@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging

virtio,acpi: features, fixes, cleanups.

vdpa support
virtio-mem support
a handy script for disassembling acpi tables
misc f

Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging

virtio,acpi: features, fixes, cleanups.

vdpa support
virtio-mem support
a handy script for disassembling acpi tables
misc fixes and cleanups

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

# gpg: Signature made Tue 07 Jul 2020 13:00:35 BST
# gpg: using RSA key 5D09FD0871C8F85B94CA8A0D281F0DB8D28D5469
# gpg: issuer "mst@redhat.com"
# gpg: Good signature from "Michael S. Tsirkin <mst@kernel.org>" [full]
# gpg: aka "Michael S. Tsirkin <mst@redhat.com>" [full]
# Primary key fingerprint: 0270 606B 6F3C DF3D 0B17 0970 C350 3912 AFBE 8E67
# Subkey fingerprint: 5D09 FD08 71C8 F85B 94CA 8A0D 281F 0DB8 D28D 5469

* remotes/mst/tags/for_upstream: (41 commits)
vhost-vdpa: introduce vhost-vdpa net client
vhost-vdpa: introduce vhost-vdpa backend
vhost_net: introduce set_config & get_config
vhost: implement vhost_force_iommu method
vhost: introduce new VhostOps vhost_force_iommu
vhost: implement vhost_vq_get_addr method
vhost: introduce new VhostOps vhost_vq_get_addr
vhost: implement vhost_dev_start method
vhost: introduce new VhostOps vhost_dev_start
vhost: check the existence of vhost_set_iotlb_callback
virtio-pci: implement queue_enabled method
virtio-bus: introduce queue_enabled method
vhost_net: use the function qemu_get_peer
net: introduce qemu_get_peer
MAINTAINERS: add VT-d entry
docs: vhost-user: add Virtio status protocol feature
tests/acpi: remove stale allowed tables
numa: Auto-enable NUMA when any memory devices are possible
virtio-mem: Exclude unplugged memory during migration
virtio-mem: Add trace events
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

# Conflicts:
# hw/arm/virt.c
# hw/virtio/trace-events

show more ...

virtio-mem: Exclude unplugged memory during migration

The content of unplugged memory is undefined and should not be migrated,
ever. Exclude all unplugged memory during precopy using the precopy not

virtio-mem: Exclude unplugged memory during migration

The content of unplugged memory is undefined and should not be migrated,
ever. Exclude all unplugged memory during precopy using the precopy notifier
infrastructure introduced for free page hinting in virtio-balloon.

Unplugged memory is marked as "not dirty", meaning it won't be
considered for migration.

Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20200626072248.78761-21-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...

virtio-mem: Allow notifiers for size changes

We want to send qapi events in case the size of a virtio-mem device
changes. This allows upper layers to always know how much memory is
actually currentl

virtio-mem: Allow notifiers for size changes

We want to send qapi events in case the size of a virtio-mem device
changes. This allows upper layers to always know how much memory is
actually currently consumed via a virtio-mem device.

Unfortuantely, we have to report the id of our proxy device. Let's provide
an easy way for our proxy device to register, so it can send the qapi
events. Piggy-backing on the notifier infrastructure (although we'll
only ever have one notifier registered) seems to be an easy way.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20200626072248.78761-17-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...

virtio-mem: Paravirtualized memory hot(un)plug

This is the very basic/initial version of virtio-mem. An introduction to
virtio-mem can be found in the Linux kernel driver [1]. While it can be
used i

virtio-mem: Paravirtualized memory hot(un)plug

This is the very basic/initial version of virtio-mem. An introduction to
virtio-mem can be found in the Linux kernel driver [1]. While it can be
used in the current state for hotplug of a smaller amount of memory, it
will heavily benefit from resizeable memory regions in the future.

Each virtio-mem device manages a memory region (provided via a memory
backend). After requested by the hypervisor ("requested-size"), the
guest can try to plug/unplug blocks of memory within that region, in order
to reach the requested size. Initially, and after a reboot, all memory is
unplugged (except in special cases - reboot during postcopy).

The guest may only try to plug/unplug blocks of memory within the usable
region size. The usable region size is a little bigger than the
requested size, to give the device driver some flexibility. The usable
region size will only grow, except on reboots or when all memory is
requested to get unplugged. The guest can never plug more memory than
requested. Unplugged memory will get zapped/discarded, similar to in a
balloon device.

The block size is variable, however, it is always chosen in a way such that
THP splits are avoided (e.g., 2MB). The state of each block
(plugged/unplugged) is tracked in a bitmap.

As virtio-mem devices (e.g., virtio-mem-pci) will be memory devices, we now
expose "VirtioMEMDeviceInfo" via "query-memory-devices".

--------------------------------------------------------------------------

There are two important follow-up items that are in the works:
1. Resizeable memory regions: Use resizeable allocations/RAM blocks to
grow/shrink along with the usable region size. This avoids creating
initially very big VMAs, RAM blocks, and KVM slots.
2. Protection of unplugged memory: Make sure the gust cannot actually
make use of unplugged memory.

Other follow-up items that are in the works:
1. Exclude unplugged memory during migration (via precopy notifier).
2. Handle remapping of memory.
3. Support for other architectures.

--------------------------------------------------------------------------

Example usage (virtio-mem-pci is introduced in follow-up patches):

Start QEMU with two virtio-mem devices (one per NUMA node):
$ qemu-system-x86_64 -m 4G,maxmem=20G \
-smp sockets=2,cores=2 \
-numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \
[...]
-object memory-backend-ram,id=mem0,size=8G \
-device virtio-mem-pci,id=vm0,memdev=mem0,node=0,requested-size=0M \
-object memory-backend-ram,id=mem1,size=8G \
-device virtio-mem-pci,id=vm1,memdev=mem1,node=1,requested-size=1G

Query the configuration:
(qemu) info memory-devices
Memory device [virtio-mem]: "vm0"
memaddr: 0x140000000
node: 0
requested-size: 0
size: 0
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem0
Memory device [virtio-mem]: "vm1"
memaddr: 0x340000000
node: 1
requested-size: 1073741824
size: 1073741824
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem1

Add some memory to node 0:
(qemu) qom-set vm0 requested-size 500M

Remove some memory from node 1:
(qemu) qom-set vm1 requested-size 200M

Query the configuration again:
(qemu) info memory-devices
Memory device [virtio-mem]: "vm0"
memaddr: 0x140000000
node: 0
requested-size: 524288000
size: 524288000
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem0
Memory device [virtio-mem]: "vm1"
memaddr: 0x340000000
node: 1
requested-size: 209715200
size: 209715200
max-size: 8589934592
block-size: 2097152
memdev: /objects/mem1

[1] https://lkml.kernel.org/r/20200311171422.10484-1-david@redhat.com

Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Message-Id: <20200626072248.78761-11-david@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

show more ...