Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20170124b' into staging

Migration

1 My maintainer change
2 Jianjun's qtailq
3 Ashijeet's only-migratable
4 Zhanghailiang's re-acti

Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20170124b' into staging

Migration

1 My maintainer change
2 Jianjun's qtailq
3 Ashijeet's only-migratable
4 Zhanghailiang's re-active images
5 Pankaj's change name of migration thread
6 My PCI migration merge
7 Juan's debug to tracing
8 My tracing on save

# gpg: Signature made Tue 24 Jan 2017 18:39:35 GMT
# gpg: using RSA key 0x0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20170124b:
migration/tracing: Add tracing on save
migration: transform remaining DPRINTF into trace_
PCI/migration merge vmstate_pci_device and vmstate_pcie_device
migration: Change name of live migration thread
migration: re-active images while migration been canceled after inactive them
migration: Fail migration blocker for --only-migratable
migration: disallow migrate_add_blocker during migration
migration: Allow "device add" options to only add migratable devices
migration: Add a new option to enable only-migratable
block/vvfat: Remove the undesirable comment
migration: add error_report
tests/migration: Add test for QTAILQ migration
migration: migrate QTAILQ
migration: extend VMStateInfo
MAINTAINERS: Add myself as a migration submaintainer

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration/tracing: Add tracing on save

Add some tracing to vmstate_subsection_save and vmstate_save_state
to help in debugging when you're not sure if a conditional piece
of data is being saved.

In

migration/tracing: Add tracing on save

Add some tracing to vmstate_subsection_save and vmstate_save_state
to help in debugging when you're not sure if a conditional piece
of data is being saved.

In vmstate_subsection_save I renamed the inner vmsd to avoid the aliasing
and be able to print both names.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20161212125838.14425-1-dgilbert@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: add error_report

Added error_report where version_ids do not match in vmstate_load_state.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: Jianjun Duan <duanj@li

migration: add error_report

Added error_report where version_ids do not match in vmstate_load_state.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
Message-Id: <1484852453-12728-5-git-send-email-duanj@linux.vnet.ibm.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: migrate QTAILQ

Currently we cannot directly transfer a QTAILQ instance because of the
limitation in the migration code. Here we introduce an approach to
transfer such structures. We creat

migration: migrate QTAILQ

Currently we cannot directly transfer a QTAILQ instance because of the
limitation in the migration code. Here we introduce an approach to
transfer such structures. We created VMStateInfo vmstate_info_qtailq
for QTAILQ. Similar VMStateInfo can be created for other data structures
such as list.

When a QTAILQ is migrated from source to target, it is appended to the
corresponding QTAILQ structure, which is assumed to have been properly
initialized.

This approach will be used to transfer pending_events and ccs_list in spapr
state.

We also create some macros in qemu/queue.h to access a QTAILQ using pointer
arithmetic. This ensures that we do not depend on the implementation
details about QTAILQ in the migration code.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
Message-Id: <1484852453-12728-3-git-send-email-duanj@linux.vnet.ibm.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

migration: extend VMStateInfo

Current migration code cannot handle some data structures such as
QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
in VMStateInfo so that customized han

migration: extend VMStateInfo

Current migration code cannot handle some data structures such as
QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
in VMStateInfo so that customized handling is supported. put now
will return int type.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
Message-Id: <1484852453-12728-2-git-send-email-duanj@linux.vnet.ibm.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20161014' into staging

migration/next for 20161014

# gpg: Signature made Fri 14 Oct 2016 16:24:13 BST
# gpg: using R

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20161014' into staging

migration/next for 20161014

# gpg: Signature made Fri 14 Oct 2016 16:24:13 BST
# gpg: using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20161014:
docs/xbzrle: correction
migrate: move max-bandwidth and downtime-limit to migrate_set_parameter
migration: Fix seg with missing port
migration/postcopy: Explicitly disallow huge pages
RAMBlocks: Store page size
Postcopy vs xbzrle: Don't send xbzrle pages once in postcopy [for 2.8]
migrate: Fix bounds check for migration parameters in migration.c
migrate: Use boxed qapi for migrate-set-parameters
migrate: Share common MigrationParameters struct
migrate: Fix cpu-throttle-increment regression in HMP
migration/rdma: Don't flag an error when we've been told about one
migration: Make failed migration load set file error
migration/rdma: Pass qemu_file errors across link
migration: Report values for comparisons
migration: report an error giving the failed field

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Report values for comparisons

Report the values when a comparison fails; together with
the previous patch that prints the device and field names
this should give a good idea of why loadin

migration: Report values for comparisons

Report the values when a comparison fails; together with
the previous patch that prints the device and field names
this should give a good idea of why loading the migration failed.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

migration: report an error giving the failed field

When a field fails to load (typically due to a limit
check, or a call to a get/put) report the device and field
to give an indication of the cause.

migration: report an error giving the failed field

When a field fails to load (typically due to a limit
check, or a call to a get/put) report the device and field
to give an indication of the cause.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/amit-migration/tags/migration-for-2.7-5' into staging

Migration:

- many compression/decompression fixes
- trace improvements
- static checker fix for detect

Merge remote-tracking branch 'remotes/amit-migration/tags/migration-for-2.7-5' into staging

Migration:

- many compression/decompression fixes
- trace improvements
- static checker fix for detecting size mismatch in unused fields
- fix VM save after snapshot

# gpg: Signature made Fri 17 Jun 2016 13:59:44 BST
# gpg: using RSA key 0xEB0B4DFC657EF670
# gpg: Good signature from "Amit Shah <amit@amitshah.net>"
# gpg: aka "Amit Shah <amit@kernel.org>"
# gpg: aka "Amit Shah <amitshah@gmx.net>"
# Primary key fingerprint: 48CA 3722 5FE7 F4A8 B337 2735 1E9A 3B5F 8540 83B6
# Subkey fingerprint: CC63 D332 AB8F 4617 4529 6534 EB0B 4DFC 657E F670

* remotes/amit-migration/tags/migration-for-2.7-5:
vmstate-static-checker: fix size mismatch detection in unused fields
migration: code clean up
migration: refine the decompression code
migration: refine the compression code
migration: protect the quit flag by lock
migration: refine ram_save_compressed_page
qemu-file: Fix qemu_put_compression_data flaw
migration: remove useless code
migration: Fix a potential issue
migration: Fix multi-thread compression bug
migration: fix inability to save VM after snapshot
migration: Trace improvements
migration: Don't use *_to_cpup() and cpu_to_*w()

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Trace improvements

A couple of improvements to tracing that have come out of helping
people with migration problems:
* vmstate_n_elems trace the count/name - for when you have problems

migration: Trace improvements

A couple of improvements to tracing that have come out of helping
people with migration problems:
* vmstate_n_elems trace the count/name - for when you have problems
getting array counts right
* vmstate_subsection_load_bad - add the idstr, for when you receive a
subsection you weren't expecting.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <1465896986-16132-1-git-send-email-dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/amit-migration/tags/migration-2.7-1' into staging

migration fixes:

- ensure src block devices continue fine after a failed migration
- fail on migration blocke

Merge remote-tracking branch 'remotes/amit-migration/tags/migration-2.7-1' into staging

migration fixes:

- ensure src block devices continue fine after a failed migration
- fail on migration blockers; helps 9p savevm/loadvm
- move autoconverge commands out of experimental state
- move the migration-specific qjson in migration/

# gpg: Signature made Mon 23 May 2016 18:15:09 BST using RSA key ID 657EF670
# gpg: Good signature from "Amit Shah <amit@amitshah.net>"
# gpg: aka "Amit Shah <amit@kernel.org>"
# gpg: aka "Amit Shah <amitshah@gmx.net>"

* remotes/amit-migration/tags/migration-2.7-1:
migration: regain control of images when migration fails to complete
savevm: fail if migration blockers are present
migration: Promote improved autoconverge commands out of experimental state
migration/qjson: Drop gratuitous use of QOM
migration: Move qjson.[ch] to migration/

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Move qjson.[ch] to migration/

Type QJSON lets you build JSON text. Its interface mirrors (a subset
of) abstract JSON syntax.

QAPI output visitors also produce JSON text. They assert th

migration: Move qjson.[ch] to migration/

Type QJSON lets you build JSON text. Its interface mirrors (a subset
of) abstract JSON syntax.

QAPI output visitors also produce JSON text. They assert their
preconditions and invariants, and therefore abort on incorrect use.

Contrastingly, QJSON does *not* detect incorrect use. It happily
produces invalid JSON then. This is what migration wants.

QJSON was designed for migration, and migration is its only user.
Move it to migration/ for proper coverage by MAINTAINERS, and to deter
accidental use outside migration.

[Pointed out by Eric: QJSON was added in commits 0457d07..b174257
-- Amit]

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <1462380558-2030-2-git-send-email-armbru@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>

show more ...

migration: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

Signed

migration: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 1453832250-766-2-git-send-email-peter.maydell@linaro.org

show more ...

Merge remote-tracking branch 'remotes/mcayland/tags/qemu-sparc-signed' into staging

qemu-sparc update

# gpg: Signature made Sat 16 Jan 2016 12:32:06 GMT using RSA key ID AE0F321F
# gpg: Good signat

Merge remote-tracking branch 'remotes/mcayland/tags/qemu-sparc-signed' into staging

qemu-sparc update

# gpg: Signature made Sat 16 Jan 2016 12:32:06 GMT using RSA key ID AE0F321F
# gpg: Good signature from "Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>"

* remotes/mcayland/tags/qemu-sparc-signed:
target-sparc: Migrate CWP and PIL for SPARC64
target-sparc: Use VMState arrays for SPARC64 TLB/MMU state
target-sparc: Convert to VMStateDescription
target-sparc: Don't flush TLB in cpu_load function
target-sparc: Split cpu_put_psr into side-effect and no-side-effect parts
vmstate: define vmstate_info_uinttl
vmstate: Introduce VMSTATE_VARRAY_MULTPLY
vmstate: introduce CPU_DoubleU arrays

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

vmstate: Introduce VMSTATE_VARRAY_MULTPLY

This allows to send a partial array where the size is another
structure field multiplied by a constant.

Signed-off-by: Juan Quintela <quintela@redhat.com>

vmstate: Introduce VMSTATE_VARRAY_MULTPLY

This allows to send a partial array where the size is another
structure field multiplied by a constant.

Signed-off-by: Juan Quintela <quintela@redhat.com>
[PMM: updated to current master]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>

show more ...

vmstate: introduce CPU_DoubleU arrays

Add vmstate support for migrating arrays of CPU_DoubleU via
VMSTATE_CPUDOUBLE_ARRAY.

Signed-off-by: Juan Quintela <quintela@redhat.com>
[PMM: rebased, since fi

vmstate: introduce CPU_DoubleU arrays

Add vmstate support for migrating arrays of CPU_DoubleU via
VMSTATE_CPUDOUBLE_ARRAY.

Signed-off-by: Juan Quintela <quintela@redhat.com>
[PMM: rebased, since files have all moved since 2012;
added VMSTATE_CPUDOUBLE_ARRAY_V for consistency with FLOAT64]
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150707' into staging

migration/next for 20150707

# gpg: Signature made Tue Jul 7 13:56:30 2015 BST using RSA key ID 5872D723
# g

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150707' into staging

migration/next for 20150707

# gpg: Signature made Tue Jul 7 13:56:30 2015 BST using RSA key ID 5872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"

* remotes/juanquintela/tags/migration/20150707: (28 commits)
migration: extend migration_bitmap
migration: protect migration_bitmap
check_section_footers: Check the correct section_id
migration: Add migration events on target side
migration: Make events a capability
migration: create migration event
migration: No need to call trace_migrate_set_state()
migration: Use always helper to set state
migration: ensure we start in NONE state
migration: Use cmpxchg correctly
migration: Add configuration section
vmstate: Create optional sections
global_state: Make section optional
migration: create new section to store global state
runstate: migration allows more transitions now
runstate: Add runstate store
Fix older machine type compatibility on power with section footers
Fail more cleanly in mismatched RAM cases
Sanity check RDMA remote data
Sort destination RAMBlocks to be the same as the source
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

vmstate: Create optional sections

To make sections optional, we need to do it at the beggining of the code.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <d

vmstate: Create optional sections

To make sections optional, we need to do it at the beggining of the code.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150612' into staging

migration/next for 20150612

# gpg: Signature made Fri Jun 12 05:56:21 2015 BST using RSA key ID 5872D723
# g

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150612' into staging

migration/next for 20150612

# gpg: Signature made Fri Jun 12 05:56:21 2015 BST using RSA key ID 5872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg: aka "Juan Quintela <quintela@trasno.org>"

* remotes/juanquintela/tags/migration/20150612: (21 commits)
Remove unneeded memset
Rename RDMA structures to make destination clear
Teach analyze-migration.py about section footers
Add a protective section footer
Disable section footers on older machine types
Merge section header writing
Move loadvm_handlers into MigrationIncomingState
Move copy out of qemu_peek_buffer
Create MigrationIncomingState
qemu_ram_foreach_block: pass up error value, and down the ramblock name
Split header writing out of qemu_savevm_state_begin
Add qemu_get_counted_string to read a string prefixed by a count byte
migration: Use normal VMStateDescriptions for Subsections
migration: create savevm_state
migration: Remove duplicated assignment of SETUP status
rdma: Fix qemu crash when IPv6 address is used for migration
arch_init: Clean up the duplicate variable 'len' defining in ram_load()
migration: reduce include files
migration: Add myself to the copyright list of both files
migration: move savevm.c inside migration/
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Move copy out of qemu_peek_buffer

qemu_peek_buffer currently copies the data it reads into a buffer,
however a future patch wants access to the buffer without the copy,
hence rework to remove the co

Move copy out of qemu_peek_buffer

qemu_peek_buffer currently copies the data it reads into a buffer,
however a future patch wants access to the buffer without the copy,
hence rework to remove the copy to the layer above.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Amit Shah <amit.shah@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

migration: Use normal VMStateDescriptions for Subsections

We create optional sections with this patch. But we already have
optional subsections. Instead of having two mechanism that do the
same, w

migration: Use normal VMStateDescriptions for Subsections

We create optional sections with this patch. But we already have
optional subsections. Instead of having two mechanism that do the
same, we can just generalize it.

For subsections we just change:

- Add a needed function to VMStateDescription
- Remove VMStateSubsection (after removal of the needed function
it is just a VMStateDescription)
- Adjust the whole tree, moving the needed function to the corresponding
VMStateDescription

Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150205' into staging

migration/next for 20150205

# gpg: Signature made Thu 05 Feb 2015 16:17:08 GMT using RSA key ID 5872D723
# g

Merge remote-tracking branch 'remotes/juanquintela/tags/migration/20150205' into staging

migration/next for 20150205

# gpg: Signature made Thu 05 Feb 2015 16:17:08 GMT using RSA key ID 5872D723
# gpg: Can't check signature: public key not found

* remotes/juanquintela/tags/migration/20150205:
fix mc146818rtc wrong subsection name to avoid vmstate_subsection_load() fail
Tracify migration/rdma.c
Add migration stream analyzation script
migration: Append JSON description of migration stream
qemu-file: Add fast ftell code path
QJSON: Add JSON writer
Print errors in some of the early migration failure cases.
Migration: Add lots of trace events
savevm: Convert fprintf to error_report
vmstate-static-checker: update whitelist

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

migration: Append JSON description of migration stream

One of the annoyances of the current migration format is the fact that
it's not self-describing. In fact, it's not properly describing at all.

migration: Append JSON description of migration stream

One of the annoyances of the current migration format is the fact that
it's not self-describing. In fact, it's not properly describing at all.
Some code randomly scattered throughout QEMU elaborates roughly how to
read and write a stream of bytes.

We discussed an idea during KVM Forum 2013 to add a JSON description of
the migration protocol itself to the migration stream. This patch
adds a section after the VM_END migration end marker that contains
description data on what the device sections of the stream are composed of.

This approach is backwards compatible with any QEMU version reading the
stream, because QEMU just stops reading after the VM_END marker and ignores
any data following it.

With an additional external program this allows us to decipher the
contents of any migration stream and hopefully make migration bugs easier
to track down.

Signed-off-by: Alexander Graf <agraf@suse.de>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

Migration: Add lots of trace events

Mostly on the load side, so that when we get a complaint about
a migration failure we can figure out what it didn't like.

Signed-off-by: Dr. David Alan Gilbert <

Migration: Add lots of trace events

Mostly on the load side, so that when we get a complaint about
a migration failure we can figure out what it didn't like.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...

savevm: Convert fprintf to error_report

Convert a bunch of fprintfs to error_reports

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Sign

savevm: Convert fprintf to error_report

Convert a bunch of fprintfs to error_reports

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

show more ...