Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-10-22-v3' into staging

QAPI patches for 2019-10-22

# gpg: Signature made Tue 22 Oct 2019 15:56:36 BST
# gpg: using RS

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-10-22-v3' into staging

QAPI patches for 2019-10-22

# gpg: Signature made Tue 22 Oct 2019 15:56:36 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2019-10-22-v3:
qapi: Allow introspecting fix for savevm's cooperation with blockdev
tests/qapi-schema: Cover feature documentation comments
tests: qapi: Test 'features' of commands
qapi: Add feature flags to commands
tests/qapi-schema: Tidy up test output indentation
qapi: Clear scripts/qapi/doc.py executable bits again
qapi: Split up scripts/qapi/common.py
qapi: Move gen_enum(), gen_enum_lookup() back to qapi/types.py
qapi: Speed up frontend tests
qapi: Eliminate accidental global frontend state
qapi: Store pragma state in QAPISourceInfo, not global state
qapi: Don't suppress doc generation without pragma doc-required

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time. We have worse tests. It's
still annoying when you work

qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time. We have worse tests. It's
still annoying when you work on the QAPI generator.

Some 1.4s user + system time is consumed by make figuring out what to
do, measured by making a target that does nothing. There's nothing I
can do about that right now. But let's see what we can do about the
other 8s.

Almost 7s are spent running test-qapi.py for every test case, the rest
normalizing and diffing test-qapi.py output. We have 190 test cases.

If I downgrade to python2, it's 4.5s, but python2 is a goner.

Hacking up test-qapi.py to exit(0) without doing anything makes it
only marginally faster. The problem is Python startup overhead.

Our configure puts -B into $(PYTHON). Running without -B is faster:
4.4s.

We could improve the Makefile to run test cases only when the test
case or the generator changed. But I'm after improvement in the case
where the generator changed.

test-qapi.py is designed to be the simplest possible building block
for a shell script to do the complete job (it's actually a Makefile,
not a shell script; no real difference). Python is just not meant for
that. It's for bigger blocks.

Move the post-processing and diffing into test-qapi.py, and make it
capable of testing multiple schema files. Set executable bits while
there.

Running it once per test case now takes slightly longer than 8s. But
running it once for all of them takes under 0.2s.

Messing with the Makefile to run it only on the tests that need
retesting is clearly not worth the bother.

Expected error output changes because the new normalization strips off
$(SRCDIR)/tests/qapi-schema/ instead of just $(SRCDIR)/.

The .exit files go away, because there is no exit status to test
anymore.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-5-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-28' into staging

QAPI patches for 2019-09-28

# gpg: Signature made Sat 28 Sep 2019 16:18:13 BST
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-28' into staging

QAPI patches for 2019-09-28

# gpg: Signature made Sat 28 Sep 2019 16:18:13 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2019-09-28: (27 commits)
qapi: Improve source file read error handling
qapi: Improve reporting of redefinition
qapi: Improve reporting of missing documentation comment
qapi: Eliminate check_keys(), rename check_known_keys()
qapi: Improve reporting of invalid 'if' further
qapi: Avoid redundant definition references in error messages
qapi: Improve reporting of missing / unknown definition keys
qapi: Improve reporting of invalid flags
qapi: Improve reporting of invalid 'if' errors
qapi: Move context-free checking to the proper place
qapi: Move context-sensitive checking to the proper place
qapi: Inline check_name() into check_union()
qapi: Plumb info to the QAPISchemaMember
qapi: Make check_type()'s array case a bit more obvious
qapi: Move check for reserved names out of add_name()
qapi: Report invalid '*' prefix like any other invalid name
qapi: Use check_name_str() where it suffices
qapi: Improve reporting of invalid name errors
qapi: Reorder check_FOO() parameters for consistency
qapi: Improve reporting of member name clashes
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Change frontend error messages to start with lower case

Starting error messages with a capital letter complicates things when
text can get interpolated both at the beginning and in the middle

qapi: Change frontend error messages to start with lower case

Starting error messages with a capital letter complicates things when
text can get interpolated both at the beginning and in the middle of
an error message. The next patch will do that. Switch to lower case
to keep it simpler.

For what it's worth, the GNU Coding Standards advise the message
"should not begin with a capital letter when it follows a program name
and/or file name, because that isn’t the beginning of a sentence. (The
sentence conceptually starts at the beginning of the line.)"

While there, avoid breaking lines containing multiple arguments in the
middle of an argument.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-7-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-24' into staging

QAPI patches for 2019-09-24

# gpg: Signature made Tue 24 Sep 2019 13:10:36 BST
# gpg: using RSA k

Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2019-09-24' into staging

QAPI patches for 2019-09-24

# gpg: Signature made Tue 24 Sep 2019 13:10:36 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2019-09-24: (37 commits)
qapi: Assert .visit() and .check_clash() run only after .check()
qapi: Fix excessive QAPISchemaEntity.check() recursion
qapi: Fix to .check() empty structs just once
qapi: Delete useless check_exprs() code for simple union kind
qapi: Clean up around check_known_keys()
qapi: Simplify check_keys()
qapi: Normalize 'if' in check_exprs(), like other sugar
qapi: Fix missing 'if' checks in struct, union, alternate 'data'
qapi: Reject blank 'if' conditions in addition to empty ones
qapi: Fix broken discriminator error messages
qapi: Remove null from schema language
qapi: Improve reporting of lexical errors
qapi: Use quotes more consistently in frontend error messages
tests/qapi-schema: Demonstrate suboptimal lexical errors
tests/qapi-schema: Demonstrate insufficient 'if' checking
tests/qapi-schema: Demonstrate broken discriminator errors
tests/qapi-schema: Demonstrate misleading optional tag error
tests/qapi-schema: Delete two redundant tests
tests/qapi-schema: Cover unknown pragma
qapi: Tweak code to match docs/devel/qapi-code-gen.txt
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Drop support for escape sequences other than \\

Since the previous commit restricted strings to printable ASCII,
\uXXXX's only use is obfuscation. Drop it.

This leaves \\, \/, \', and \". S

qapi: Drop support for escape sequences other than \\

Since the previous commit restricted strings to printable ASCII,
\uXXXX's only use is obfuscation. Drop it.

This leaves \\, \/, \', and \". Since QAPI schema strings are all
names, and names are restricted to ASCII letters, digits, hyphen, and
underscore, none of them is useful.

The latter three have no test coverage. Drop them.

Keep \\ to avoid (more) gratuitous incompatibility with JSON.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190913201349.24332-8-armbru@redhat.com>

show more ...

Merge remote-tracking branch 'remotes/armbru/tags/pull-qmp-2015-05-05' into staging

drop qapi nested structs

# gpg: Signature made Tue May 5 17:40:40 2015 BST using RSA key ID EB918653
# gpg: Good

Merge remote-tracking branch 'remotes/armbru/tags/pull-qmp-2015-05-05' into staging

drop qapi nested structs

# gpg: Signature made Tue May 5 17:40:40 2015 BST using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qmp-2015-05-05: (40 commits)
qapi: Check for member name conflicts with a base class
qapi: Support (subset of) \u escapes in strings
qapi: Tweak doc references to QMP when QGA is also meant
qapi: Drop dead visitor code related to nested structs
qapi: Drop support for inline nested types
qapi: Drop inline nested structs in query-pci
qapi: Drop inline nested struct in query-version
qapi: Drop tests for inline nested structs
qapi: Merge UserDefTwo and UserDefNested in tests
qapi: Forbid 'type' in schema
qapi: Use 'struct' instead of 'type' in schema
qapi: Document 'struct' metatype
qapi: Prefer 'struct' over 'type' in generator
qapi: More rigorous checking for type safety bypass
qapi: Whitelist commands that don't return dictionary
qapi: Require valid names
qapi: More rigourous checking of types
qapi: Add some type check tests
qapi: Unify type bypass and add tests
qapi: Allow true, false and null in schema json
...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

show more ...

qapi: Support (subset of) \u escapes in strings

The handling of \ inside QAPI strings was less than ideal, and
really only worked JSON's \/, \\, \", and our extension of \'
(an obvious extension, wh

qapi: Support (subset of) \u escapes in strings

The handling of \ inside QAPI strings was less than ideal, and
really only worked JSON's \/, \\, \", and our extension of \'
(an obvious extension, when you realize we use '' instead of ""
for strings). For other things, like '\n', it resulted in a
literal 'n' instead of a newline.

Of course, at the moment, we really have no use for escaped
characters, as QAPI has to map to C identifiers, and we currently
support ASCII only for that. But down the road, we may add
support for default values for string parameters to a command
or struct; if that happens, it would be nice to correctly support
all JSON escape sequences, such as \n or \uXXXX. This gets us
closer, by supporting Unicode escapes in the ASCII range.

Since JSON does not require \OCTAL or \xXX escapes, and our QMP
implementation does not understand them either, I intentionally
reject it here, but it would be an easy addition if we desired it.
Likewise, intentionally refusing the NUL byte means we don't have
to worry about C strings being shorter than the qapi input.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qapi: Better error messages for bad expressions

The previous commit demonstrated that the generator overlooked some
fairly basic broken expressions:
- missing metataype
- metatype key has a non-stri

qapi: Better error messages for bad expressions

The previous commit demonstrated that the generator overlooked some
fairly basic broken expressions:
- missing metataype
- metatype key has a non-string value
- unknown key in relation to the metatype
- conflicting metatype (this patch treats the second metatype as an
unknown key of the first key visited, which is not necessarily the
first key the user typed)

Add check_keys to cover these situations, and update testcases to
match. A couple other tests (enum-missing-data, indented-expr) had
to change since the validation added here occurs so early.
Conversely, changes to ident-with-escape results show that we still
have problems where our handling of escape sequences differs from
true JSON, which will matter down the road if we allow arbitrary
default string values for optional parameters (but for now is not
too bad, as we currently can avoid unicode escaping as we don't
need to represent anything beyond C identifier material).

While valid .json files won't trigger any of these cases, we might
as well be nicer to developers that make a typo while trying to add
new QAPI code.

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...

qapi: Add some expr tests

Demonstrate that the qapi generator doesn't deal well with
expressions that aren't up to par. Later patches will improve
the expected results as the generator is made stric

qapi: Add some expr tests

Demonstrate that the qapi generator doesn't deal well with
expressions that aren't up to par. Later patches will improve
the expected results as the generator is made stricter. Only
a few of the the added tests actually behave sanely at
rejecting obvious problems or demonstrating success.

Note that in some cases, we reject bad QAPI merely because our
pseudo-JSON parser does not yet know how to parse numbers. This
series does not address that, but when a later series adds support
for numeric defaults of integer fields, the testsuite will ensure
that we don't lose the error (and hopefully that the error
message quality is improved).

Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>

show more ...