Merge tag 'migration-20240621-pull-request' of https://gitlab.com/farosas/qemu into staging

Migration pull request

- Fabiano's fix for fdset + file migration truncating the migration
file

- Fabi

Merge tag 'migration-20240621-pull-request' of https://gitlab.com/farosas/qemu into staging

Migration pull request

- Fabiano's fix for fdset + file migration truncating the migration
file

- Fabiano's fdset + direct-io support for mapped-ram

- Peter's various cleanups (multifd sync, thread names, migration
states, tests)

- Peter's new migration state postcopy-recover-setup

- Philippe's unused vmstate macro cleanup

# -----BEGIN PGP SIGNATURE-----
#
# iQJEBAABCAAuFiEEqhtIsKIjJqWkw2TPx5jcdBvsMZ0FAmZ1vIsQHGZhcm9zYXNA
# c3VzZS5kZQAKCRDHmNx0G+wxnVZTEACdFIsQ/PJw2C9eeLNor5B5MNSEqUjxX0KN
# 6s/uTkJ/dcv+2PI92SzRCZ1dpR5e9AyjTFYbLc9tPRBIROEhlUaoc84iyEy0jCFU
# eJ65/RQbH5QHRpOZwbN5RmGwnapfOWHGTn3bpdrmSQTOAy8R2TPGY4SVYR+gamTn
# bAv1cAsrOOBUfCi8aqvSlmvuliOW0lzJdF4XHa3mAaigLoF14JdwUZdyIMP1mLDp
# /fllbHCKCvJ1vprE9hQmptBR9PzveJZOZamIVt96djJr5+C869+9PMCn3a5vxqNW
# b+/LhOZjac37Ecg5kgbq+cO1E4EXKC3zWOmDTw8kHUwp9oYNi1upwLdpHbAAZaQD
# /JmHKsExx9QuV8mrVyGBXMI92E6RrT54b1Bjcuo63gAP8p9JRRxGT22U3LghNbTm
# 1XcGPR3rswjT1yTgE6qAqAIMR+7X5MrJVWop9ub/lF5DQ1VYIwmlKSNdwDHFDhRq
# 0F1k2+EksNpcZ0BH2+3iFml7qKHLVupLQKTWcLdrlnQnTfSG3+yW7eyA5Mte79Qp
# nJPcHt8qBqUVQ9Uf/4490TM4Lrp+T+m16exIi0tISLaDXSVkFJnlowipSm+tQ7U3
# Sm68JWdWWEsXZVaMqJeBE8nA/hCoQDpo4hVdwftStI+NayXbRX/EgvPqrNAvwh+c
# i4AdHdn6hQ==
# =ZX0p
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 21 Jun 2024 10:46:51 AM PDT
# gpg: using RSA key AA1B48B0A22326A5A4C364CFC798DC741BEC319D
# gpg: issuer "farosas@suse.de"
# gpg: Good signature from "Fabiano Rosas <farosas@suse.de>" [unknown]
# gpg: aka "Fabiano Almeida Rosas <fabiano.rosas@suse.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: AA1B 48B0 A223 26A5 A4C3 64CF C798 DC74 1BEC 319D

* tag 'migration-20240621-pull-request' of https://gitlab.com/farosas/qemu: (28 commits)
migration: Remove unused VMSTATE_ARRAY_TEST() macro
tests/migration-tests: Cover postcopy failure on reconnect
tests/migration-tests: Verify postcopy-recover-setup status
tests/migration-tests: migration_event_wait()
tests/migration-tests: Always enable migration events
tests/migration-tests: Drop most WIN32 ifdefs for postcopy failure tests
migration/docs: Update postcopy recover session for SETUP phase
migration/postcopy: Add postcopy-recover-setup phase
migration: Cleanup incoming migration setup state change
migration: Use MigrationStatus instead of int
migration: Rename thread debug names
migration/multifd: Avoid the final FLUSH in complete()
tests/qtest/migration: Add a test for mapped-ram with passing of fds
migration: Add documentation for fdset with multifd + file
monitor: fdset: Match against O_DIRECT
tests/qtest/migration: Add tests for file migration with direct-io
migration/multifd: Add direct-io support
migration: Add direct-io parameter
io: Stop using qemu_open_old in channel-file
monitor: Report errors from monitor_fdset_dup_fd_add
...

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>

show more ...

monitor: fdset: Match against O_DIRECT

We're about to enable the use of O_DIRECT in the migration code and
due to the alignment restrictions imposed by filesystems we need to
make sure the flag is o

monitor: fdset: Match against O_DIRECT

We're about to enable the use of O_DIRECT in the migration code and
due to the alignment restrictions imposed by filesystems we need to
make sure the flag is only used when doing aligned IO.

The migration will do parallel IO to different regions of a file, so
we need to use more than one file descriptor. Those cannot be obtained
by duplicating (dup()) since duplicated file descriptors share the
file status flags, including O_DIRECT. If one migration channel does
unaligned IO while another sets O_DIRECT to do aligned IO, the
filesystem would fail the unaligned operation.

The add-fd QMP command along with the fdset code are specifically
designed to allow the user to pass a set of file descriptors with
different access flags into QEMU to be later fetched by code that
needs to alternate between those flags when doing IO.

Extend the fdset matching to behave the same with the O_DIRECT flag.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

monitor: Report errors from monitor_fdset_dup_fd_add

I'm keeping the EACCES because callers expect to be able to look at
errno.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosa

monitor: Report errors from monitor_fdset_dup_fd_add

I'm keeping the EACCES because callers expect to be able to look at
errno.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

monitor: Simplify fdset and fd removal

Remove fds right away instead of setting the ->removed flag. We don't
need the extra complexity of having a cleanup function reap the
removed entries at a late

monitor: Simplify fdset and fd removal

Remove fds right away instead of setting the ->removed flag. We don't
need the extra complexity of having a cleanup function reap the
removed entries at a later time.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

monitor: Stop removing non-duplicated fds

monitor_fdsets_cleanup() currently has three responsibilities:

1- Remove the fds that have been marked for removal(->removed=true) by
qmp_remove_fd(). T

monitor: Stop removing non-duplicated fds

monitor_fdsets_cleanup() currently has three responsibilities:

1- Remove the fds that have been marked for removal(->removed=true) by
qmp_remove_fd(). This is overly complicated, but ok.

2- Remove any file descriptors that have been passed into QEMU and
never duplicated[1,2]. A file descriptor without duplicates
indicates that no part of QEMU has made use of it. This is
problematic because the current implementation does it only if the
guest is not running and the monitor is closed.

3- Remove/free fdsets that have become empty due to the above
removals. This is ok.

The scenario described in (2) is starting to show some cracks now that
we're trying to consume fds from the migration code:

- Doing cleanup every time the last monitor connection closes works to
reap unused fds, but also has the side effect of forcing the
management layer to pass the file descriptors again in case of a
disconnect/re-connect, if that happened to be the only monitor
connection.

Another side effect is that removing an fd with qmp_remove_fd() is
effectively delayed until the last monitor connection closes.

The usage of mon_refcount is also problematic because it's racy.

- Checking runstate_is_running() skips the cleanup unless the VM is
running and avoids premature cleanup of the fds, but also has the
side effect of blocking the legitimate removal of an fd via
qmp_remove_fd() if the VM happens to be in another state.

This affects qmp_remove_fd() and qmp_query_fdsets() in particular
because requesting a removal at a bad time (guest stopped) might
cause an fd to never be removed, or to be removed at a much later
point in time, causing the query command to continue showing the
supposedly removed fd/fdset.

Note that file descriptors that *have* been duplicated are owned by
the code that uses them and will be removed after qemu_close() is
called. Therefore we've decided that the best course of action to
avoid the undesired side-effects is to stop managing non-duplicated
file descriptors.

1- efb87c1697 ("monitor: Clean up fd sets on monitor disconnect")
2- ebe52b592d ("monitor: Prevent removing fd from set during init")

Reviewed-by: Peter Xu <peterx@redhat.com>
[fix logic mistake: s/fdset_free/fdset_free_if_empty]
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

monitor: Introduce monitor_fdset_*free

Introduce new functions to remove and free no longer used fds and
fdsets.

We need those to decouple the remove/free routines from
monitor_fdset_cleanup() whic

monitor: Introduce monitor_fdset_*free

Introduce new functions to remove and free no longer used fds and
fdsets.

We need those to decouple the remove/free routines from
monitor_fdset_cleanup() which will go away in the next patches.

The new functions:

- monitor_fdset_free/_if_empty() will be used when a monitor
connection closes and when an fd is removed to cleanup any fdset
that is now empty.

- monitor_fdset_fd_free() will be used to remove one or more fds that
have been explicitly targeted by qmp_remove_fd().

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

monitor: Drop monitor_fdset_dup_fd_find/_remove()

Those functions are not needed, one remove function should already
work. Clean it up.

Here the code doesn't really care about whether we need to k

monitor: Drop monitor_fdset_dup_fd_find/_remove()

Those functions are not needed, one remove function should already
work. Clean it up.

Here the code doesn't really care about whether we need to keep that dupfd
around if close() failed: when that happens something got very wrong,
keeping the dup_fd around the fdsets may not help that situation so far.

Cc: Dr. David Alan Gilbert <dave@treblig.org>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[add missing return statement, removal during traversal is not safe]
Signed-off-by: Fabiano Rosas <farosas@suse.de>

show more ...

Merge tag 'win-socket-pull-request' of https://gitlab.com/marcandre.lureau/qemu into staging

QMP command to import win32 sockets

# -----BEGIN PGP SIGNATURE-----
#
# iQJQBAABCAA6FiEEh6m9kz+HxgbSdvYt

Merge tag 'win-socket-pull-request' of https://gitlab.com/marcandre.lureau/qemu into staging

QMP command to import win32 sockets

# -----BEGIN PGP SIGNATURE-----
#
# iQJQBAABCAA6FiEEh6m9kz+HxgbSdvYt2ujhCXWWnOUFAmQPDQ0cHG1hcmNhbmRy
# ZS5sdXJlYXVAcmVkaGF0LmNvbQAKCRDa6OEJdZac5eTUD/41+bodkctP9wtNQT5g
# 4P2XQysa9dhxIaQuPT48J5bN0velOAv+p+e9jNMojVPHogACRGkjJUGrh2AhaWJO
# bpqB5teNz3pbTLAHNrqiJdUrJDI0WSqBN2q7WgfbzvHMugBqul13n6UG/cVWH8D3
# pDX3miBl9Cv3zUDFzPjHH3eR/MHz+6wXmuzUGQdWqyGBoLwqgWA3Bqh39BDVOeJf
# 03Kq3TbJSP096EjGGrq1pTYDIIv9AKzUWgn8tT8S73sD3J0BN28Gl5HirXDx8e/4
# 2WtW/XLYKjqoUl7RmXOjfOarCV+kxzdoYCAUYfyH6DLWrkXc41L5ugFdyxxQ66Sh
# +on7hKCBzPEOPEXmlm6HlMj3bK4C/GI6mIoaZgCrsvj9xlehhQNtwpndAwAR8esH
# perQ6q+jPdoQnBvOBgC3amckS1kYdbQivTILkoopumw/q4waG5reyA4rshbhm/bs
# U33ZRzob0XyRWqvWAcq9hnWB5gvQCcppeJlu60gocnX5wdZOjbnsBXw3l+r2osIh
# izJbxwM6xmz9oHh50nhDCn42JrNdSnZJdJ/XA/lrOkTHQ6kbZO7v86Y/mKQz+Vyx
# Uyhb8/y8gKrUkZlGCEMvQVcyvOA8vneX8WhfZUm6w5MWWCNKfCrEl6UGmFkPAL0m
# +sUYIhN5PSxWSQU5MWogXtQEPA==
# =/Y/7
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 13 Mar 2023 11:46:21 GMT
# gpg: using RSA key 87A9BD933F87C606D276F62DDAE8E10975969CE5
# gpg: issuer "marcandre.lureau@redhat.com"
# gpg: Good signature from "Marc-André Lureau <marcandre.lureau@redhat.com>" [full]
# gpg: aka "Marc-André Lureau <marcandre.lureau@gmail.com>" [full]
# Primary key fingerprint: 87A9 BD93 3F87 C606 D276 F62D DAE8 E109 7596 9CE5

* tag 'win-socket-pull-request' of https://gitlab.com/marcandre.lureau/qemu: (25 commits)
monitor: restrict command getfd to POSIX hosts
qtest: enable vnc-display test on win32
libqtest: make qtest_qmp_add_client work on win32
qmp: add 'get-win32-socket'
monitor: release the lock before calling close()
qmp: 'add_client' actually expects sockets
osdep: implement qemu_socketpair() for win32
tests/docker: fix a win32 error due to portability
char: do not double-close fd when failing to add client
tests: fix path separator, use g_build_filename()
win32: replace closesocket() with close() wrapper
os-posix: remove useless ioctlsocket() define
win32: avoid mixing SOCKET and file descriptor space
slirp: open-code qemu_socket_(un)select()
slirp: unregister the win32 SOCKET
main-loop: remove qemu_fd_register(), win32/slirp/socket specific
aio/win32: aio_set_fd_handler() only supports SOCKET
aio: make aio_set_fd_poll() static to aio-posix.c
win32/socket: introduce qemu_socket_unselect() helper
win32/socket: introduce qemu_socket_select() helper
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

monitor: restrict command getfd to POSIX hosts

Currently, the function will simply fail if ancillary fds are not
provided, for ex on unsupported platforms.

This changes the failure from:

{"err

monitor: restrict command getfd to POSIX hosts

Currently, the function will simply fail if ancillary fds are not
provided, for ex on unsupported platforms.

This changes the failure from:

{"error": {"class": "GenericError", "desc": "No file descriptor
supplied via SCM_RIGHTS"}}

to:

{"error": {"class": "CommandNotFound", "desc": "The command getfd
has not been found"}}

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>

show more ...

qmp: add 'get-win32-socket'

A process with enough capabilities can duplicate a socket to QEMU. Add a
QMP command to import it and add it to the monitor fd list, so it can be
later used by other comm

qmp: add 'get-win32-socket'

A process with enough capabilities can duplicate a socket to QEMU. Add a
QMP command to import it and add it to the monitor fd list, so it can be
later used by other commands.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230306122751.2355515-9-marcandre.lureau@redhat.com>

show more ...

monitor: release the lock before calling close()

As per comment, presumably to avoid syscall in critical section.

Fixes: 0210c3b39bef08 ("monitor: Use LOCK_GUARD macros")
Signed-off-by: Marc-André

monitor: release the lock before calling close()

As per comment, presumably to avoid syscall in critical section.

Fixes: 0210c3b39bef08 ("monitor: Use LOCK_GUARD macros")
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230306122751.2355515-7-marcandre.lureau@redhat.com>

show more ...

Merge tag 'pull-monitor-2023-02-03-v2' of https://repo.or.cz/qemu/armbru into staging

Monitor patches for 2023-02-03

# -----BEGIN PGP SIGNATURE-----
#
# iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMF

Merge tag 'pull-monitor-2023-02-03-v2' of https://repo.or.cz/qemu/armbru into staging

Monitor patches for 2023-02-03

# -----BEGIN PGP SIGNATURE-----
#
# iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMFAmPeAkgSHGFybWJydUBy
# ZWRoYXQuY29tAAoJEDhwtADrkYZTUagP/iZ24jXaWoFOKaO70wdQ/tdoQObWZnUV
# 8xJNJYmYYbWoiq9wQXHebi/yEgBudso1lLzAnp8lsF12ybnNV1zsjyV/yumEKSNW
# 3nL1NZIcuY9IDmCe97clY9nm9H2lUhjjyCG3gnjg+uC3JjlSjO/T8lbkdT+fYnkR
# AInVTCPYFjSO9MIOhN0WNIY73HlAjr4zx5TEgS/D4pFj6iGq2qEniSDGMRf+/fVr
# uSbIXbQlum+VAdxbGMSVf8yQPlNcFUXUpSJrbgJE272H6saQuvn5mkwD0RcYXyaI
# OlfXpATDRNTsP3yYImxgr7y29Exo1HnCuC6T1n/+fwkirtMR3a7X6XjaQwFsWcrx
# xxGiHQOve3r/I3DAO6A64T2ceD/XuI43LygqkkljfuoXifnJz7Lo39P9HrY0dhpC
# KSld2n/Vv4xYyykvqAzpvzijwq679ILIbTplhm9gOrfrDRZjWad3uLAcYxsTXXR8
# BQbHGovcAzTOEx/0Quo3NThpAeNYPGyrPz3xBIV+XtPJGWvFsrA/s/po4qWDTmF6
# UTzPoEmznsD+DRboNOKfinCsOnpTAru4gbXevi7sfmMHQbLYN5xgsrF7WdlaxWa6
# 4QbJyNUq0O+aL0gyfVLuiZBCQ32Jaz1WvowK856Yl4jwczP5HM0ujyyM75+Kx072
# PdnMgxYYLSij
# =d+wL
# -----END PGP SIGNATURE-----
# gpg: Signature made Sat 04 Feb 2023 06:59:20 GMT
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* tag 'pull-monitor-2023-02-03-v2' of https://repo.or.cz/qemu/armbru: (35 commits)
monitor: Rename misc.c to hmp-target.c
monitor: Loosen coupling between misc.c and monitor.c slightly
monitor: Move remaining QMP stuff from misc.c to qmp-cmds.c
monitor: Move remaining HMP commands from misc.c to hmp-cmds.c
monitor: Move target-dependent HMP commands to hmp-cmds-target.c
monitor: Move monitor_putc() next to monitor_puts & external linkage
monitor: Split file descriptor passing stuff off misc.c
qdev: Move HMP command completion from monitor to softmmu/
acpi: Move the QMP command from monitor/ to hw/acpi/
stats: Move HMP commands from monitor/ to stats/
stats: Move QMP commands from monitor/ to stats/
runstate: Move HMP commands from monitor/ to softmmu/
tpm: Move HMP commands from monitor/ to softmmu/
virtio: Move HMP commands from monitor/ to hw/virtio/
migration: Move the QMP command from monitor/ to migration/
migration: Move HMP commands from monitor/ to migration/
net: Move hmp_info_network() to net-hmp-cmds.c
net: Move HMP commands from monitor to net/
hmp: Rewrite strlist_from_comma_list() as hmp_split_at_comma()
rocker: Move HMP commands from monitor to hw/net/rocker/
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

monitor: Split file descriptor passing stuff off misc.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230124121946.1139465-27-armbru@redhat.com>