Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* ASAN fixes

# gpg: Signature made Tue 12 Mar 2019 14:35:59 GMT
# gpg: using RSA key F13338574B662389866

Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* ASAN fixes

# gpg: Signature made Tue 12 Mar 2019 14:35:59 GMT
# gpg: using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg: issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg: aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4 E2F7 7E15 100C CD36 69B1
# Subkey fingerprint: F133 3857 4B66 2389 866C 7682 BFFB D25F 78C7 AE83

* remotes/bonzini/tags/for-upstream:
test-migration: fix memory leak
migration: fix memory leak
test-bdrv-graph-mod: fix Error leak
test-char: fix undefined behavior

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: fix memory leak

Reported by ASAN.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20190306a' into staging

Migation pull 2019-03-06

(This replaces the pull sent yesterday)

a) 4 small fixes including the cancel

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20190306a' into staging

Migation pull 2019-03-06

(This replaces the pull sent yesterday)

a) 4 small fixes including the cancel problem
that caused the ahci migration test to fail
intermittently
b) Yury's ignore-shared feature
c) Juan's extra tests
d) Wei Wang's free page hinting
e) Some Colo fixes from Zhang Chen

Diff from yesterdays pull:
1) A missing fix of mine (cleanup during exit)
2) Changes from Eric/Markus on 'Create socket-address parameter'

# gpg: Signature made Wed 06 Mar 2019 11:39:53 GMT
# gpg: using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20190306a: (22 commits)
qapi/migration.json: Remove a variable that doesn't exist in example
Migration/colo.c: Make COLO node running after failover
Migration/colo.c: Fix double close bug when occur COLO failover
virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT
migration/ram.c: add the free page optimization enable flag
migration/ram.c: add a notifier chain for precopy
migration: API to clear bits of guest free pages from the dirty bitmap
migration: use bitmap_mutex in migration_bitmap_clear_dirty
bitmap: bitmap_count_one_with_offset
bitmap: fix bitmap_count_one
tests: Add basic migration precopy tcp test
migration: Create socket-address parameter
tests: Add migration xbzrle test
migration: Add capabilities validation
tests/migration-test: Add a test for ignore-shared capability
migration: Add an ability to ignore shared RAM blocks
migration: Introduce ignore-shared capability
exec: Change RAMBlockIterFunc definition
migration/rdma: clang compilation fix
migration: Cleanup during exit
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Create socket-address parameter

It will be used to store the uri parameters. We want this only for
tcp, so we don't set it for other uris. We need it to know what port
is migration runni

migration: Create socket-address parameter

It will be used to store the uri parameters. We want this only for
tcp, so we don't set it for other uris. We need it to know what port
is migration running.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
dgilbert: Removed DummyStruct as suggested by Eric & Markus

--

show more ...

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180710a' into staging

Migration pull 2018-07-10 (for 3.0)

Migration fixes and migration test fixes, mostly
around postcopy and p

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180710a' into staging

Migration pull 2018-07-10 (for 3.0)

Migration fixes and migration test fixes, mostly
around postcopy and postcopy recovery

# gpg: Signature made Tue 10 Jul 2018 16:27:19 BST
# gpg: using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20180710a:
migration: reorder MIG_CMD_POSTCOPY_RESUME
tests: hide stderr for postcopy recovery test
tests: add postcopy recovery test
tests: introduce wait_for_migration_status()
tests: introduce migrate_query*() helpers
tests: allow migrate() to take extra flags
tests: introduce migrate_postcopy_* helpers
migration: show pause/recover state on dst host
migration: fix incorrect bitmap size calculation
migration: loosen recovery check when load vm
migration: simplify check to use qemu file buffer
migration: unify incoming processing
migration: unbreak postcopy recovery
migration: move income process out of multifd
migration: delay postcopy paused state

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: unify incoming processing

This is the 2nd patch to unbreak postcopy recovery.

Let's unify the migration_incoming_process() call at a single place
rather than calling it in connection set

migration: unify incoming processing

This is the 2nd patch to unbreak postcopy recovery.

Let's unify the migration_incoming_process() call at a single place
rather than calling it in connection setup codes. This fixes a problem
that we will go into incoming migration procedure even if we are trying
to recovery from a paused postcopy migration.

Fixes: 36c2f8be2c ("migration: Delay start of migration main routines")
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180627132246.5576-5-peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20180515' into staging

migration/next for 20180515

# gpg: Signature made Tue 15 May 2018 22:54:38 BST
# gpg: using R

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20180515' into staging

migration/next for 20180515

# gpg: Signature made Tue 15 May 2018 22:54:38 BST
# gpg: using RSA key F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20180515: (40 commits)
Migration+TLS: Fix crash due to double cleanup
migration: Textual fixups for blocktime
migration: update index field when delete or qsort RDMALocalBlock
migration: update docs
migration/hmp: add migrate_pause command
migration/qmp: add command migrate-pause
migration: introduce lock for to_dst_file
hmp/migration: add migrate_recover command
qmp/migration: new command migrate-recover
migration: init dst in migration_object_init too
migration: final handshake for the resume
migration: setup ramstate for resume
migration: synchronize dirty bitmap for resume
migration: introduce SaveVMHandlers.resume_prepare
migration: new message MIG_RP_MSG_RESUME_ACK
migration: new cmd MIG_CMD_POSTCOPY_RESUME
migration: new message MIG_RP_MSG_RECV_BITMAP
migration: new cmd MIG_CMD_RECV_BITMAP
migration: wakeup dst ram-load-thread for recover
migration: new state "postcopy-recover"
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: let incoming side use thread context

The old incoming migration is running in main thread and default
gcontext. With the new qio_channel_add_watch_full() we can now let it
run in the thr

migration: let incoming side use thread context

The old incoming migration is running in main thread and default
gcontext. With the new qio_channel_add_watch_full() we can now let it
run in the thread's own gcontext (if there is one).

Currently this patch does nothing alone. But when any of the incoming
migration is run in another iothread (e.g., the upcoming migrate-recover
command), this patch will bind the incoming logic to the iothread
instead of the main thread (which may already get page faulted and
hanged).

RDMA is not considered for now since it's not even using the QIO watch
framework at all.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Message-Id: <20180502104740.12123-2-peterx@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

migration: Delay start of migration main routines

We need to make sure that we have started all the multifd threads.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berran

migration: Delay start of migration main routines

We need to make sure that we have started all the multifd threads.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

migration: Export functions to create send channels

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180323a' into staging

Migration fixes for 2.12

All small fixes. Dan's is a missing piece
of a cleanup that finally completes so

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180323a' into staging

Migration fixes for 2.12

All small fixes. Dan's is a missing piece
of a cleanup that finally completes something,
and between Paolo, Dan and myself we recon it's
still on the edge of being a bug fix.

# gpg: Signature made Fri 23 Mar 2018 20:17:40 GMT
# gpg: using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20180323a:
migration: Fix block migration flag case
migration/block: compare only read blocks against the rate limiter
migration/block: limit the number of parallel I/O requests
migration: Fix rate limiting issue on RDMA migration
migration: convert socket server to QIONetListener

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: convert socket server to QIONetListener

Instead of creating a QIOChannelSocket directly for the migration
server socket, use a QIONetListener. This provides the ability
to listen on multi

migration: convert socket server to QIONetListener

Instead of creating a QIOChannelSocket directly for the migration
server socket, use a QIONetListener. This provides the ability
to listen on multiple sockets at the same time, so enables
full support for IPv4/IPv6 dual stack.

For example, '$QEMU -incoming tcp::9000' now correctly listens
on both 0.0.0.0 and :: at the same time, instead of only on 0.0.0.0.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <20180312141714.7223-1-berrange@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/berrange/tags/qio-next-pull-request' into staging

# gpg: Signature made Wed 07 Mar 2018 11:24:41 GMT
# gpg: using RSA key BE86EBB415104FDF
# gpg:

Merge remote-tracking branch 'remotes/berrange/tags/qio-next-pull-request' into staging

# gpg: Signature made Wed 07 Mar 2018 11:24:41 GMT
# gpg: using RSA key BE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>"
# gpg: aka "Daniel P. Berrange <berrange@redhat.com>"
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF

* remotes/berrange/tags/qio-next-pull-request:
qio: non-default context for TLS handshake
qio: non-default context for async conn
qio: non-default context for threaded qtask
qio: store gsources for net listeners
qio: introduce qio_channel_add_watch_{full|source}
qio: rename qio_task_thread_result

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qio: non-default context for async conn

We have worked on qio_task_run_in_thread() already. Further, let
all the qio channel APIs use that context.

Signed-off-by: Peter Xu <peterx@redhat.com>
Sign

qio: non-default context for async conn

We have worked on qio_task_run_in_thread() already. Further, let
all the qio channel APIs use that context.

Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180206a' into staging

Migration pull 2018-02-06

This is based off Juan's last pull with a few extras, but
also removing:
Add

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20180206a' into staging

Migration pull 2018-02-06

This is based off Juan's last pull with a few extras, but
also removing:
Add migration xbzrle test
Add migration precopy test

As well as my normal test boxes, I also gave it a test
on a 32 bit ARM box and it seems happy (a Calxeda highbank)
and a big-endian power box.

Dave

# gpg: Signature made Tue 06 Feb 2018 15:33:31 GMT
# gpg: using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20180206a:
migration: incoming postcopy advise sanity checks
migration: Don't leak IO channels
migration: Recover block devices if failure in device state
tests: Adjust sleeps for migration test
tests: Create migrate-start-postcopy command
tests: Add deprecated commands migration test
tests: Use consistent names for migration
tests: Consolidate accelerators declaration
tests: Remove deprecated migration tests commands
migration: Drop current address parameter from save_zero_page()
migration: use s->threshold_size inside migration_update_counters
migration/savevm.c: set MAX_VM_CMD_PACKAGED_SIZE to 1ul << 32
migration: Route errors down through migration_channel_connect
migration: Allow migrate_fd_connect to take an Error *

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Route errors down through migration_channel_connect

Route async errors (especially from sockets) down through
migration_channel_connect and on to migrate_fd_connect where they
can be clea

migration: Route errors down through migration_channel_connect

Route async errors (especially from sockets) down through
migration_channel_connect and on to migrate_fd_connect where they
can be cleaned up.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20180115' into staging

migration/next for 20180115

# gpg: Signature made Mon 15 Jan 2018 11:51:00 GMT
# gpg: using R

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20180115' into staging

migration/next for 20180115

# gpg: Signature made Mon 15 Jan 2018 11:51:00 GMT
# gpg: using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20180115: (27 commits)
migration: remove notify in fd_error
migration: remove some block_cleanup_parameters()
migration: put the finish part into a new function
migration: major cleanup for migrate iterations
migration: cleanup stats update into function
migration: use switch at the end of migration
migration: introduce migrate_calculate_complete
migration: introduce downtime_start
migration: move vm_old_running into global state
migration: split use of MigrationState.total_time
migration: remove "enable_colo" var
migration: qemu_savevm_state_cleanup() in cleanup
migration: assert colo instead of check
migration: finalize current_migration object
migration: Guard ram_bytes_remaining against early call
migration: add postcopy total blocktime into query-migrate
migration: add blocktime calculation into migration-test
migration: postcopy_blocktime documentation
migration: calculate vCPU blocktime on dst side
migration: add postcopy blocktime ctx into MigrationIncomingState
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: free addr in the same function that we created it

Otherwise, we can't use it after calling socket_start_incoming_migration

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by:

migration: free addr in the same function that we created it

Otherwise, we can't use it after calling socket_start_incoming_migration

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170922-1' into staging

migration/next for 20170922

# gpg: Signature made Fri 22 Sep 2017 13:15:06 BST
# gpg: using

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170922-1' into staging

migration/next for 20170922

# gpg: Signature made Fri 22 Sep 2017 13:15:06 BST
# gpg: using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20170922-1:
migration: split ufd_version_check onto receive/request features part
migration: fix hardcoded function name in error report
migration: pass MigrationIncomingState* into migration check functions
migration: split common postcopy out of ram postcopy
migration: fix ram_save_pending
migration: add has_postcopy savevm handler
bitmap: provide to_le/from_le helpers
bitmap: introduce bitmap_count_one()
bitmap: remove BITOP_WORD()
migration: Split migration_fd_process_incoming
migration: Create multifd migration threads
migration: Create x-multifd-page-count parameter
migration: Create x-multifd-channels parameter
migration: Add multifd capability
migration: Create migration_has_all_channels
migration: Add comments to channel functions
migration: Teach it about G_SOURCE_REMOVE
migration: Create migration_ioc_process_incoming()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Create migration_has_all_channels

This function allows us to decide when to close the listener socket.
For now, we only need one connection.

Signed-off-by: Juan Quintela <quintela@redhat

migration: Create migration_has_all_channels

This function allows us to decide when to close the listener socket.
For now, we only need one connection.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>

show more ...

migration: Teach it about G_SOURCE_REMOVE

As this is defined on glib 2.32, add compatibility macros for older glibs.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berran

migration: Teach it about G_SOURCE_REMOVE

As this is defined on glib 2.32, add compatibility macros for older glibs.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170613' into staging

migration/next for 20170613

# gpg: Signature made Tue 13 Jun 2017 10:01:45 BST
# gpg: using R

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170613' into staging

migration/next for 20170613

# gpg: Signature made Tue 13 Jun 2017 10:01:45 BST
# gpg: using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20170613:
migration: Move migration.h to migration/
migration: Move remaining exported functions to migration/misc.h
migration: create global_state.c
migration: ram_control_* are implemented in qemu_file
migration: Commands are only used inside migration.c
migration: Move constants to savevm.h
migration: Move dump_vmsate_json_to_file() to misc.h
migration: Split registration functions from vmstate.h
migration: Move self_announce_delay() to misc.h
migration: Remove MigrationState from migration_channel_incomming()
ram: Now POSTCOPY_ACTIVE is the same that STATUS_ACTIVE
ram: Print block stats also in the complete case
migration: Don't try to set *errp directly
migration: isolate return path on src

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Move migration.h to migration/

Nothing uses it outside of migration.h

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by

migration: Move migration.h to migration/

Nothing uses it outside of migration.h

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>

show more ...

migration: Remove MigrationState from migration_channel_incomming()

All callers were calling migrate_get_current(), so do it inside the function.

Signed-off-by: Juan Quintela <quintela@redhat.com>

migration: Remove MigrationState from migration_channel_incomming()

All callers were calling migrate_get_current(), so do it inside the function.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170601' into staging

migration/next for 20170601

# gpg: Signature made Thu 01 Jun 2017 17:51:04 BST
# gpg: using R

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20170601' into staging

migration/next for 20170601

# gpg: Signature made Thu 01 Jun 2017 17:51:04 BST
# gpg: using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20170601:
migration: Move include/migration/block.h into migration/
migration: Export ram.c functions in its own file
migration: Create include for migration snapshots
migration: Export rdma.c functions in its own file
migration: Export tls.c functions in its own file
migration: Export socket.c functions in its own file
migration: Export fd.c functions in its own file
migration: Export exec.c functions in its own file
migration: Split qemu-file.h
migration: Remove unneeded includes of migration/vmstate.h
migration: shut src return path unconditionally
migration: fix leak of src file on dst
migration: Remove section_id parameter from vmstate_load
migration: loadvm handlers are not used
migration: Use savevm_handlers instead of loadvm copy

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...