Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2020-04-07' into staging

QAPI patches for 2020-04-07

# gpg: Signature made Tue 07 Apr 2020 12:43:38 BST
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2020-04-07' into staging

QAPI patches for 2020-04-07

# gpg: Signature made Tue 07 Apr 2020 12:43:38 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2020-04-07:
json: Fix check for unbalanced right curly brace

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

json: Fix check for unbalanced right curly brace

We immediately diagnose unbalanced right curly brace:

$ qemu-kvm --nodefaults --nographic --qmp stdio
{"QMP": {"version": {"qemu": {"micro":

json: Fix check for unbalanced right curly brace

We immediately diagnose unbalanced right curly brace:

$ qemu-kvm --nodefaults --nographic --qmp stdio
{"QMP": {"version": {"qemu": {"micro": 91, "minor": 2, "major": 4},
"package": "v5.0.0-rc1-1-gf6ce4a439a08"}, "capabilities": ["oob"]}}
}
{"error": {"class": "GenericError", "desc": "JSON parse error,
expecting value"}}

except within square bracket:

[}

The check for unbalanced braces has a typo. Fix it.

Fixes: 8d3265b3d00db1071d1d3bf8433b4818088fdeb5
Signed-off-by: Simran Singhal <singhalsimran0@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200402182848.GA3023@simran-Inspiron-5558>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten to explain what's broken]
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qobject-2018-08-24' into staging

QObject patches for 2018-08-24

# gpg: Signature made Fri 24 Aug 2018 20:28:53 BST
# gpg: using

Merge remote-tracking branch 'remotes/armbru/tags/pull-qobject-2018-08-24' into staging

QObject patches for 2018-08-24

# gpg: Signature made Fri 24 Aug 2018 20:28:53 BST
# gpg: using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qobject-2018-08-24: (58 commits)
json: Update references to RFC 7159 to RFC 8259
json: Support %% in JSON strings when interpolating
json: Improve safety of qobject_from_jsonf_nofail() & friends
json: Keep interpolation state in JSONParserContext
tests/drive_del-test: Fix harmless JSON interpolation bug
json: Clean up headers
qobject: Drop superfluous includes of qemu-common.h
json: Make JSONToken opaque outside json-parser.c
json: Unbox tokens queue in JSONMessageParser
json: Streamline json_message_process_token()
json: Enforce token count and size limits more tightly
qjson: Have qobject_from_json() & friends reject empty and blank
json: Assert json_parser_parse() consumes all tokens on success
json: Fix streamer not to ignore trailing unterminated structures
json: Fix latent parser aborts at end of input
qjson: Fix qobject_from_json() & friends for multiple values
json: Improve names of lexer states related to numbers
json: Replace %I64d, %I64u by %PRId64, %PRIu64
json: Leave rejecting invalid interpolation to parser
json: Pass lexical errors and limit violations to callback
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

json: Clean up headers

The JSON parser has three public headers, json-lexer.h, json-parser.h,
json-streamer.h. They all contain stuff that is of no interest
outside qobject/json-*.c.

Collect the p

json: Clean up headers

The JSON parser has three public headers, json-lexer.h, json-parser.h,
json-streamer.h. They all contain stuff that is of no interest
outside qobject/json-*.c.

Collect the public interface in include/qapi/qmp/json-parser.h, and
everything else in qobject/json-parser-int.h.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-54-armbru@redhat.com>

show more ...

qobject: Drop superfluous includes of qemu-common.h

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-53-armbru@redha

qobject: Drop superfluous includes of qemu-common.h

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-53-armbru@redhat.com>

show more ...

json: Make JSONToken opaque outside json-parser.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-52-armbru@redhat.

json: Make JSONToken opaque outside json-parser.c

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-52-armbru@redhat.com>

show more ...

json: Unbox tokens queue in JSONMessageParser

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-51-armbru@redhat.com>

json: Streamline json_message_process_token()

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-50-armbru@redhat.com>

json: Enforce token count and size limits more tightly

Token count and size limits exist to guard against excessive heap
usage. We check them only after we created the token on the heap.
That's ass

json: Enforce token count and size limits more tightly

Token count and size limits exist to guard against excessive heap
usage. We check them only after we created the token on the heap.
That's assigning a cowboy to the barn to lasso the horse after it has
bolted. Close the barn door instead: check before we create the
token.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-49-armbru@redhat.com>

show more ...

json: Fix streamer not to ignore trailing unterminated structures

json_message_process_token() accumulates tokens until it got the
sequence of tokens that comprise a single JSON value (it counts cur

json: Fix streamer not to ignore trailing unterminated structures

json_message_process_token() accumulates tokens until it got the
sequence of tokens that comprise a single JSON value (it counts curly
braces and square brackets to decide). It feeds those token sequences
to json_parser_parse(). If a non-empty sequence of tokens remains at
the end of the parse, it's silently ignored. check-qjson.c cases
unterminated_array(), unterminated_array_comma(), unterminated_dict(),
unterminated_dict_comma() demonstrate this bug.

Fix as follows. Introduce a JSON_END_OF_INPUT token. When the
streamer receives it, it feeds the accumulated tokens to
json_parser_parse().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-46-armbru@redhat.com>

show more ...

json: Pass lexical errors and limit violations to callback

The callback to consume JSON values takes QObject *json, Error *err.
If both are null, the callback is supposed to make up an error by
itse

json: Pass lexical errors and limit violations to callback

The callback to consume JSON values takes QObject *json, Error *err.
If both are null, the callback is supposed to make up an error by
itself. This sucks.

qjson.c's consume_json() neglects to do so, which makes
qobject_from_json() null instead of failing. I consider that a bug.

The culprit is json_message_process_token(): it passes two null
pointers when it runs into a lexical error or a limit violation. Fix
it to pass a proper Error object then. Update the callbacks:

* monitor.c's handle_qmp_command(): the code to make up an error is
now dead, drop it.

* qga/main.c's process_event(): lumps the "both null" case together
with the "not a JSON object" case. The former is now gone. The
error message "Invalid JSON syntax" is misleading for the latter.
Improve it to "Input must be a JSON object".

* qobject/qjson.c's consume_json(): no update; check-qjson
demonstrates qobject_from_json() now sets an error on lexical
errors, but still doesn't on some other errors.

* tests/libqtest.c's qmp_response(): the Error object is now reliable,
so use it to improve the error message.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-40-armbru@redhat.com>

show more ...

json: Treat unwanted interpolation as lexical error

The JSON parser optionally supports interpolation. The lexer
recognizes interpolation tokens unconditionally. The parser rejects
them when inter

json: Treat unwanted interpolation as lexical error

The JSON parser optionally supports interpolation. The lexer
recognizes interpolation tokens unconditionally. The parser rejects
them when interpolation is disabled, in parse_interpolation().
However, it neglects to set an error then, which can make
json_parser_parse() fail without setting an error.

Move the check for unwanted interpolation from the parser's
parse_interpolation() into the lexer's finite state machine. When
interpolation is disabled, '%' is now handled like any other
unexpected character.

The next commit will improve how such lexical errors are handled.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-39-armbru@redhat.com>

show more ...

json: Don't create JSON_ERROR tokens that won't be used

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-37-armbru@r

json: Don't create JSON_ERROR tokens that won't be used

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-37-armbru@redhat.com>

show more ...

json: Don't pass null @tokens to json_parser_parse()

json_parser_parse() normally returns the QObject on success. Except
it returns null when its @tokens argument is null.

Its only caller json_mes

json: Don't pass null @tokens to json_parser_parse()

json_parser_parse() normally returns the QObject on success. Except
it returns null when its @tokens argument is null.

Its only caller json_message_process_token() passes null @tokens when
emitting a lexical error. The call is a rather opaque way to say json
= NULL then.

Simplify matters by lifting the assignment to json out of the emit
path: initialize json to null, set it to the value of
json_parser_parse() when there's no lexical error. Drop the special
case from json_parser_parse().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-36-armbru@redhat.com>

show more ...

json: Redesign the callback to consume JSON values

The classical way to structure parser and lexer is to have the client
call the parser to get an abstract syntax tree, the parser call the
lexer to

json: Redesign the callback to consume JSON values

The classical way to structure parser and lexer is to have the client
call the parser to get an abstract syntax tree, the parser call the
lexer to get the next token, and the lexer call some function to get
input characters.

Another way to structure them would be to have the client feed
characters to the lexer, the lexer feed tokens to the parser, and the
parser feed abstract syntax trees to some callback provided by the
client. This way is more easily integrated into an event loop that
dispatches input characters as they arrive.

Our JSON parser is kind of between the two. The lexer feeds tokens to
a "streamer" instead of a real parser. The streamer accumulates
tokens until it got the sequence of tokens that comprise a single JSON
value (it counts curly braces and square brackets to decide). It
feeds those token sequences to a callback provided by the client. The
callback passes each token sequence to the parser, and gets back an
abstract syntax tree.

I figure it was done that way to make a straightforward recursive
descent parser possible. "Get next token" becomes "pop the first
token off the token sequence". Drawback: we need to store a complete
token sequence. Each token eats 13 + input characters + malloc
overhead bytes.

Observations:

1. This is not the only way to use recursive descent. If we replaced
"get next token" by a coroutine yield, we could do without a
streamer.

2. The lexer reports errors by passing a JSON_ERROR token to the
streamer. This communicates the offending input characters and
their location, but no more.

3. The streamer reports errors by passing a null token sequence to the
callback. The (already poor) lexical error information is thrown
away.

4. Having the callback receive a token sequence duplicates the code to
convert token sequence to abstract syntax tree in every callback.

5. Known bug: the streamer silently drops incomplete token sequences.

This commit rectifies 4. by lifting the call of the parser from the
callbacks into the streamer. Later commits will address 3. and 5.

The lifting removes a bug from qjson.c's parse_json(): it passed a
pointer to a non-null Error * in certain cases, as demonstrated by
check-qjson.c.

json_parser_parse() is now unused. It's a stupid wrapper around
json_parser_parse_err(). Drop it, and rename json_parser_parse_err()
to json_parser_parse().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-35-armbru@redhat.com>

show more ...

json: Have lexer call streamer directly

json_lexer_init() takes the function to process a token as an
argument. It's always json_message_process_token(). Makes the code
harder to understand for no

json: Have lexer call streamer directly

json_lexer_init() takes the function to process a token as an
argument. It's always json_message_process_token(). Makes the code
harder to understand for no actual gain. Drop the indirection.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20180823164025.12553-34-armbru@redhat.com>

show more ...

json: remove useless return value from lexer/parser

The lexer always returns 0 when char feeding. Furthermore, none of the
caller care about the return value.

Signed-off-by: Marc-André Lureau <marc

json: remove useless return value from lexer/parser

The lexer always returns 0 when char feeding. Furthermore, none of the
caller care about the return value.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180326150916.9602-10-marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20180823164025.12553-32-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* SCSI scanner support
* fixes to qemu-char and net exit
* FreeBSD fixes
* Other small bugfixes

# gpg: Signature made W

Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging

* SCSI scanner support
* fixes to qemu-char and net exit
* FreeBSD fixes
* Other small bugfixes

# gpg: Signature made Wed 13 Jul 2016 12:30:11 BST
# gpg: using RSA key 0xBFFBD25F78C7AE83
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>"
# gpg: aka "Paolo Bonzini <pbonzini@redhat.com>"
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4 E2F7 7E15 100C CD36 69B1
# Subkey fingerprint: F133 3857 4B66 2389 866C 7682 BFFB D25F 78C7 AE83

* remotes/bonzini/tags/for-upstream:
hostmem: detect host backend memory is being used properly
hostmem: fix QEMU crash by 'info memdev'
char: do not use atexit cleanup handler
net: do not use atexit for cleanup
slirp: use exit notifier for slirp_smb_cleanup
tap: use an exit notifier to call down_script
util: Fix MIN_NON_ZERO
qemu-sockets: use qapi_free_SocketAddress in cleanup
disas: avoid including everything in headers compiled from C++
json-streamer: fix double-free on exiting during a parse
main-loop: check return value before using pointer
Use "-s" instead of "--quiet" to resolve non-fatal build error on FreeBSD.
scsi-bus: Use longer sense buffer with scanners
scsi-bus: Add SCSI scanner support

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

json-streamer: fix double-free on exiting during a parse

Now that json-streamer tries not to leak tokens on incomplete parse,
the tokens can be freed twice if QEMU destroys the json-streamer
object

json-streamer: fix double-free on exiting during a parse

Now that json-streamer tries not to leak tokens on incomplete parse,
the tokens can be freed twice if QEMU destroys the json-streamer
object during the parser->emit call. To fix this, create the new
empty GQueue earlier, so that it is already in place when the old
one is passed to parser->emit.

Reported-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1467636059-12557-1-git-send-email-pbonzini@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-06-30' into staging

QAPI patches 2016-06-30

# gpg: Signature made Thu 30 Jun 2016 14:29:43 BST
# gpg: using RSA key 0

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2016-06-30' into staging

QAPI patches 2016-06-30

# gpg: Signature made Thu 30 Jun 2016 14:29:43 BST
# gpg: using RSA key 0x3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2016-06-30:
qapi: Fix memleak in string visitors on int lists
qapi: Simplify use of range.h
range: Create range.c for code that should not be inline
qapi: Fix crash on missing alternate member of QAPI struct
checkpatch: There is no qemu_strtod()
qobject: Correct JSON lexer grammar comments
json-streamer: Don't leak tokens on incomplete parse

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

json-streamer: Don't leak tokens on incomplete parse

Valgrind complained about a number of leaks in
tests/check-qobject-json:

==12657== definitely lost: 17,247 bytes in 1,234 blocks

All of whic

json-streamer: Don't leak tokens on incomplete parse

Valgrind complained about a number of leaks in
tests/check-qobject-json:

==12657== definitely lost: 17,247 bytes in 1,234 blocks

All of which had the same root cause: on an incomplete parse,
we were abandoning the token queue without cleaning up the
allocated data within each queue element. Introduced in
commit 95385fe, when we switched from QList (which recursively
frees contents) to g_queue (which does not).

We don't yet require glib 2.32 with its g_queue_free_full(),
so open-code it instead.

CC: qemu-stable@nongnu.org
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <1463608012-12760-1-git-send-email-eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qobject: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

Signed-o

qobject: Clean up includes

Clean up includes so that osdep.h is included first and headers
which it implies are not included manually.

This commit was created with scripts/clean-includes.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1454089805-5470-12-git-send-email-peter.maydell@linaro.org

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2015-11-26' into staging

QMP and QObject patches

# gpg: Signature made Thu 26 Nov 2015 09:07:18 GMT using RSA key ID EB918653
# gpg: G

Merge remote-tracking branch 'remotes/armbru/tags/pull-monitor-2015-11-26' into staging

QMP and QObject patches

# gpg: Signature made Thu 26 Nov 2015 09:07:18 GMT using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-monitor-2015-11-26:
qjson: Limit number of tokens in addition to total size
qjson: surprise, allocating 6 QObjects per token is expensive
qjson: store tokens in a GQueue
qjson: Convert to parser to recursive descent
qjson: replace QString in JSONLexer with GString
qjson: Inline token_is_escape() and simplify
qjson: Inline token_is_keyword() and simplify
qjson: Give each of the six structural chars its own token type
qjson: Spell out some silent assumptions
check-qjson: Add test for JSON nesting depth limit
qjson: Don't crash when input exceeds nesting limit
qjson: Apply nesting limit more sanely
monitor: Plug memory leak on QMP error

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qjson: Limit number of tokens in addition to total size

Commit 29c75dd "json-streamer: limit the maximum recursion depth and
maximum token count" attempts to guard against excessive heap usage by
li

qjson: Limit number of tokens in addition to total size

Commit 29c75dd "json-streamer: limit the maximum recursion depth and
maximum token count" attempts to guard against excessive heap usage by
limiting total token size (it says "token count", but that's a lie).

Total token size is a rather imprecise predictor of heap usage: many
small tokens use more space than few large tokens with the same input
size, because there's a constant per-token overhead: 37 bytes on my
system.

Tighten this up: limit the token count to 2Mi. Chosen to roughly
match the 64MiB total token size limit.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <1448486613-17634-13-git-send-email-armbru@redhat.com>

show more ...

qjson: surprise, allocating 6 QObjects per token is expensive

Replace the contents of the tokens GQueue with a simple struct. This cuts
the amount of memory allocated by tests/check-qjson from ~500

qjson: surprise, allocating 6 QObjects per token is expensive

Replace the contents of the tokens GQueue with a simple struct. This cuts
the amount of memory allocated by tests/check-qjson from ~500MB to ~20MB,
and the execution time from 600ms to 80ms on my laptop. Still a lot (some
could be saved by using an intrusive list, such as QSIMPLEQ, instead of
the GQueue), but the savings are already massive and the right thing to
do would probably be to get rid of json-streamer completely.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <1448300659-23559-5-git-send-email-pbonzini@redhat.com>
[Straightforwardly rebased on my patches]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...