Merge remote-tracking branch 'remotes/armbru/tags/pull-qobject-2018-08-24' into staging

QObject patches for 2018-08-24

# gpg: Signature made Fri 24 Aug 2018 20:28:53 BST
# gpg: using

Merge remote-tracking branch 'remotes/armbru/tags/pull-qobject-2018-08-24' into staging

QObject patches for 2018-08-24

# gpg: Signature made Fri 24 Aug 2018 20:28:53 BST
# gpg: using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qobject-2018-08-24: (58 commits)
json: Update references to RFC 7159 to RFC 8259
json: Support %% in JSON strings when interpolating
json: Improve safety of qobject_from_jsonf_nofail() & friends
json: Keep interpolation state in JSONParserContext
tests/drive_del-test: Fix harmless JSON interpolation bug
json: Clean up headers
qobject: Drop superfluous includes of qemu-common.h
json: Make JSONToken opaque outside json-parser.c
json: Unbox tokens queue in JSONMessageParser
json: Streamline json_message_process_token()
json: Enforce token count and size limits more tightly
qjson: Have qobject_from_json() & friends reject empty and blank
json: Assert json_parser_parse() consumes all tokens on success
json: Fix streamer not to ignore trailing unterminated structures
json: Fix latent parser aborts at end of input
qjson: Fix qobject_from_json() & friends for multiple values
json: Improve names of lexer states related to numbers
json: Replace %I64d, %I64u by %PRId64, %PRIu64
json: Leave rejecting invalid interpolation to parser
json: Pass lexical errors and limit violations to callback
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

json: Clean up headers

The JSON parser has three public headers, json-lexer.h, json-parser.h,
json-streamer.h. They all contain stuff that is of no interest
outside qobject/json-*.c.

Collect the p

json: Clean up headers

The JSON parser has three public headers, json-lexer.h, json-parser.h,
json-streamer.h. They all contain stuff that is of no interest
outside qobject/json-*.c.

Collect the public interface in include/qapi/qmp/json-parser.h, and
everything else in qobject/json-parser-int.h.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-54-armbru@redhat.com>

show more ...

qobject: Drop superfluous includes of qemu-common.h

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-53-armbru@redha

qobject: Drop superfluous includes of qemu-common.h

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-53-armbru@redhat.com>

show more ...

json: Make JSONToken opaque outside json-parser.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-52-armbru@redhat.

json: Make JSONToken opaque outside json-parser.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-52-armbru@redhat.com>

show more ...

json: Redesign the callback to consume JSON values

The classical way to structure parser and lexer is to have the client
call the parser to get an abstract syntax tree, the parser call the
lexer to

json: Redesign the callback to consume JSON values

The classical way to structure parser and lexer is to have the client
call the parser to get an abstract syntax tree, the parser call the
lexer to get the next token, and the lexer call some function to get
input characters.

Another way to structure them would be to have the client feed
characters to the lexer, the lexer feed tokens to the parser, and the
parser feed abstract syntax trees to some callback provided by the
client. This way is more easily integrated into an event loop that
dispatches input characters as they arrive.

Our JSON parser is kind of between the two. The lexer feeds tokens to
a "streamer" instead of a real parser. The streamer accumulates
tokens until it got the sequence of tokens that comprise a single JSON
value (it counts curly braces and square brackets to decide). It
feeds those token sequences to a callback provided by the client. The
callback passes each token sequence to the parser, and gets back an
abstract syntax tree.

I figure it was done that way to make a straightforward recursive
descent parser possible. "Get next token" becomes "pop the first
token off the token sequence". Drawback: we need to store a complete
token sequence. Each token eats 13 + input characters + malloc
overhead bytes.

Observations:

1. This is not the only way to use recursive descent. If we replaced
"get next token" by a coroutine yield, we could do without a
streamer.

2. The lexer reports errors by passing a JSON_ERROR token to the
streamer. This communicates the offending input characters and
their location, but no more.

3. The streamer reports errors by passing a null token sequence to the
callback. The (already poor) lexical error information is thrown
away.

4. Having the callback receive a token sequence duplicates the code to
convert token sequence to abstract syntax tree in every callback.

5. Known bug: the streamer silently drops incomplete token sequences.

This commit rectifies 4. by lifting the call of the parser from the
callbacks into the streamer. Later commits will address 3. and 5.

The lifting removes a bug from qjson.c's parse_json(): it passed a
pointer to a non-null Error * in certain cases, as demonstrated by
check-qjson.c.

json_parser_parse() is now unused. It's a stupid wrapper around
json_parser_parse_err(). Drop it, and rename json_parser_parse_err()
to json_parser_parse().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-35-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-misc-2018-02-07-v4' into staging

Miscellaneous patches for 2018-02-07

# gpg: Signature made Fri 09 Feb 2018 12:52:51 GMT
# gpg:

Merge remote-tracking branch 'remotes/armbru/tags/pull-misc-2018-02-07-v4' into staging

Miscellaneous patches for 2018-02-07

# gpg: Signature made Fri 09 Feb 2018 12:52:51 GMT
# gpg: using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-misc-2018-02-07-v4:
Move include qemu/option.h from qemu-common.h to actual users
Drop superfluous includes of qapi/qmp/qjson.h
Drop superfluous includes of qapi/qmp/dispatch.h
Include qapi/qmp/qnull.h exactly where needed
Include qapi/qmp/qnum.h exactly where needed
Include qapi/qmp/qbool.h exactly where needed
Include qapi/qmp/qstring.h exactly where needed
Include qapi/qmp/qdict.h exactly where needed
Include qapi/qmp/qlist.h exactly where needed
Include qapi/qmp/qobject.h exactly where needed
qdict qlist: Make most helper macros functions
Eliminate qapi/qmp/types.h
Typedef the subtypes of QObject in qemu/typedefs.h, too
Include qmp-commands.h exactly where needed
Drop superfluous includes of qapi/qmp/qerror.h
Include qapi/error.h exactly where needed
Drop superfluous includes of qapi-types.h and test-qapi-types.h
Clean up includes
Use #include "..." for our own headers, <...> for others
vnc: use stubs for CONFIG_VNC=n dummy functions

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

Include qapi/qmp/qlist.h exactly where needed

This cleanup makes the number of objects depending on qapi/qmp/qlist.h
drop from 4551 (out of 4743) to 16 in my "build everything" tree.

While there, s

Include qapi/qmp/qlist.h exactly where needed

This cleanup makes the number of objects depending on qapi/qmp/qlist.h
drop from 4551 (out of 4743) to 16 in my "build everything" tree.

While there, separate #include from file comment with a blank line.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180201111846.21846-12-armbru@redhat.com>

show more ...

include: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

NB: If t

include: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

NB: If this commit breaks compilation for your out-of-tree
patchseries or fork, then you need to make sure you add
#include "qemu/osdep.h" to any new .c files that you have.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2015-11-26' into staging

QMP and QObject patches

# gpg: Signature made Thu 26 Nov 2015 09:07:18 GMT using RSA key ID EB918653
# gpg: G

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2015-11-26' into staging

QMP and QObject patches

# gpg: Signature made Thu 26 Nov 2015 09:07:18 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-monitor-2015-11-26:
qjson: Limit number of tokens in addition to total size
qjson: surprise, allocating 6 QObjects per token is expensive
qjson: store tokens in a GQueue
qjson: Convert to parser to recursive descent
qjson: replace QString in JSONLexer with GString
qjson: Inline token_is_escape() and simplify
qjson: Inline token_is_keyword() and simplify
qjson: Give each of the six structural chars its own token type
qjson: Spell out some silent assumptions
check-qjson: Add test for JSON nesting depth limit
qjson: Don't crash when input exceeds nesting limit
qjson: Apply nesting limit more sanely
monitor: Plug memory leak on QMP error

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qjson: store tokens in a GQueue

Even though we still have the "streamer" concept, the tokens can now
be deleted as they are read. While doing so convert from QList to
GQueue, since the next step wi

qjson: store tokens in a GQueue

Even though we still have the "streamer" concept, the tokens can now
be deleted as they are read. While doing so convert from QList to
GQueue, since the next step will make tokens not a QObject and we
will have to do the conversion anyway.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1448300659-23559-4-git-send-email-pbonzini@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

Merge branch 'master' of git://git.qemu.org/qemu into prep-up

Conflicts:
hw/Makefile.objs
hw/ppc_prep.c

Signed-off-by: Andreas Färber <andreas.faerber@web.de>

Merge branch 'master' of git://git.qemu.org/qemu into qom-cpu

Adapt header include paths.

Signed-off-by: Andreas Färber <afaerber@suse.de>

Merge remote-tracking branch 'quintela/thread-20121220.next' into staging

* quintela/thread-20121220.next: (79 commits)
migration: merge QEMUFileBuffered into MigrationState
migration: fix qemu_

Merge remote-tracking branch 'quintela/thread-20121220.next' into staging

* quintela/thread-20121220.next: (79 commits)
migration: merge QEMUFileBuffered into MigrationState
migration: fix qemu_get_fd for BufferedFile
ram: refactor ram_save_block() return value
ram: account the amount of transferred ram better
ram: optimize migration bitmap walking
ram: Use memory_region_test_and_clear_dirty
memory: introduce memory_region_test_and_clear_dirty
ram: Add last_sent_block
ram: rename last_block to last_seen_block
migration: move migration notifier
migration: Inline qemu_fopen_ops_buffered into migrate_fd_connect
migration: move migration_fd_put_ready()
migration: add XFER_LIMIT_RATIO
migration: move buffered_file.c code into migration.c
savevm: New save live migration method: pending
buffered_file: unfold buffered_append in buffered_put_buffer
buffered_file: don't flush on put buffer
buffered_file: Unfold the trick to restart generating migration data
migration: just lock migrate_fd_put_ready
migration: remove unfreeze logic
...

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

show more ...

Merge remote-tracking branch 'bonzini/header-dirs' into staging

* bonzini/header-dirs: (45 commits)
janitor: move remaining public headers to include/
hw: move executable format header files to

Merge remote-tracking branch 'bonzini/header-dirs' into staging

* bonzini/header-dirs: (45 commits)
janitor: move remaining public headers to include/
hw: move executable format header files to hw/
fpu: move public header file to include/fpu
softmmu: move remaining include files to include/ subdirectories
softmmu: move include files to include/sysemu/
misc: move include files to include/qemu/
qom: move include files to include/qom/
migration: move include files to include/migration/
monitor: move include files to include/monitor/
exec: move include files to include/exec/
block: move include files to include/block/
qapi: move include files to include/qobject/
janitor: add guards to headers
qapi: make struct Visitor opaque
qapi: remove qapi/qapi-types-core.h
qapi: move inclusions of qemu-common.h from headers to .c files
ui: move files to ui/ and include/ui/
qemu-ga: move qemu-ga files to qga/
net: reorganize headers
net: move net.c to net/
...

Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

show more ...

qapi: move include files to include/qobject/

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>