qapi: Clear scripts/qapi/doc.py executable bits again

Commit fbf09a2fa4 "qapi: add 'ifcond' to visitor methods" brought back
the executable bits. Fix that. Drop the #! line for good measure.

Sign

qapi: Clear scripts/qapi/doc.py executable bits again

Commit fbf09a2fa4 "qapi: add 'ifcond' to visitor methods" brought back
the executable bits. Fix that. Drop the #! line for good measure.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-8-armbru@redhat.com>

show more ...

qapi: Split up scripts/qapi/common.py

The QAPI code generator clocks in at some 3100 SLOC in 8 source files.
Almost 60% of the code is in qapi/common.py. Split it into more
focused modules:

* Move

qapi: Split up scripts/qapi/common.py

The QAPI code generator clocks in at some 3100 SLOC in 8 source files.
Almost 60% of the code is in qapi/common.py. Split it into more
focused modules:

* Move QAPISchemaPragma and QAPISourceInfo to qapi/source.py.

* Move QAPIError and its sub-classes to qapi/error.py.

* Move QAPISchemaParser and QAPIDoc to parser.py. Use the opportunity
to put QAPISchemaParser first.

* Move check_expr() & friends to qapi/expr.py. Use the opportunity to
put the code into a more sensible order.

* Move QAPISchema & friends to qapi/schema.py

* Move QAPIGen and its sub-classes, ifcontext,
QAPISchemaModularCVisitor, and QAPISchemaModularCVisitor to qapi/gen.py

* Delete camel_case(), it's unused since commit e98859a9b9 "qapi:
Clean up after recent conversions to QAPISchemaVisitor"

A number of helper functions remain in qapi/common.py. I considered
moving the code generator helpers to qapi/gen.py, but decided not to.
Perhaps we should rewrite them as methods of QAPIGen some day.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-7-armbru@redhat.com>
[Add "# -*- coding: utf-8 -*-" lines]

show more ...

qapi: Move gen_enum(), gen_enum_lookup() back to qapi/types.py

The next commit will split up qapi/common.py. gen_enum() needs
QAPISchemaEnumMember, and that's in the way. Move it to qapi/types.py

qapi: Move gen_enum(), gen_enum_lookup() back to qapi/types.py

The next commit will split up qapi/common.py. gen_enum() needs
QAPISchemaEnumMember, and that's in the way. Move it to qapi/types.py
along with its buddy gen_enum_lookup().

Permit me a short a digression on history: how did gen_enum() end up
in qapi/common.py? Commit 21cd70dfc1 "qapi script: add event support"
duplicated qapi-types.py's gen_enum() and gen_enum_lookup() in
qapi-event.py. Simply importing them would have been cleaner, but
wasn't possible as qapi-types.py was a program, not a module. Commit
efd2eaa6c2 "qapi: De-duplicate enum code generation" de-duplicated by
moving them to qapi.py, which was a module.

Since then, program qapi-types.py has morphed into module types.py.
It's where gen_enum() and gen_enum_lookup() started, and where they
belong.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-6-armbru@redhat.com>

show more ...

qapi: Eliminate accidental global frontend state

The frontend can't be run more than once due to its global state.
A future commit will want to do that.

The only global frontend state remaining is

qapi: Eliminate accidental global frontend state

The frontend can't be run more than once due to its global state.
A future commit will want to do that.

The only global frontend state remaining is accidental:
QAPISchemaParser.__init__()'s parameter previously_included=[].
Python evaluates the default once, at definition time. Any
modifications to it are visible in subsequent calls. Well-known
Python trap. Change the default to None and replace it by the real
default in the function body. Use the opportunity to convert
previously_included to a set.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-4-armbru@redhat.com>

show more ...

qapi: Store pragma state in QAPISourceInfo, not global state

The frontend can't be run more than once due to its global state.
A future commit will want to do that.

Recent commit "qapi: Move contex

qapi: Store pragma state in QAPISourceInfo, not global state

The frontend can't be run more than once due to its global state.
A future commit will want to do that.

Recent commit "qapi: Move context-sensitive checking to the proper
place" got rid of many global variables already, but pragma state is
still stored in global variables (that's why a pragma directive's
scope is the complete schema).

Move the pragma state to QAPISourceInfo.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-3-armbru@redhat.com>

show more ...

qapi: Don't suppress doc generation without pragma doc-required

Commit bc52d03ff5 "qapi: Make doc comments optional where we don't
need them" made scripts/qapi2texi.py fail[*] unless the schema had

qapi: Don't suppress doc generation without pragma doc-required

Commit bc52d03ff5 "qapi: Make doc comments optional where we don't
need them" made scripts/qapi2texi.py fail[*] unless the schema had
pragma 'doc-required': true. The stated reason was inability to cope
with incomplete documentation.

When commit fb0bc835e5 "qapi-gen: New common driver for code and doc
generators" folded scripts/qapi2texi.py into scripts/qapi-gen.py, it
turned the failure into silent suppression.

The doc generator can cope with incomplete documentation now. I don't
know since when, or what the problem was, or even whether it ever
existed.

Drop the silent suppression.

[*] The fail part was broken, fixed in commit e8ba07ea9a.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-2-armbru@redhat.com>

show more ...

qapi: Improve source file read error handling

qapi-gen.py crashes when it can't open the main schema file, and when
it can't read from any schema file. Lazy.

Change QAPISchema.__init__() to take a

qapi: Improve source file read error handling

qapi-gen.py crashes when it can't open the main schema file, and when
it can't read from any schema file. Lazy.

Change QAPISchema.__init__() to take a file name instead of a file
object. Move the open code from _include() to __init__(), so it's
used for the main schema file, too.

Move the read into the try for good measure, and rephrase the error
message.

Reporting open or read failure for the main schema file needs a
QAPISourceInfo representing "no source". Make QAPISourceInfo cope
with fname=None.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-27-armbru@redhat.com>

show more ...

qapi: Improve reporting of redefinition

Point to the previous definition, unless it's a built-in.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Me

qapi: Improve reporting of redefinition

Point to the previous definition, unless it's a built-in.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-26-armbru@redhat.com>

show more ...

qapi: Improve reporting of missing documentation comment

Have check_exprs() check this later, so the error message gains an "in
definition line". Tweak the error message.

Signed-off-by: Markus Arm

qapi: Improve reporting of missing documentation comment

Have check_exprs() check this later, so the error message gains an "in
definition line". Tweak the error message.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-25-armbru@redhat.com>

show more ...

qapi: Eliminate check_keys(), rename check_known_keys()

check_keys() has become a trivial wrapper for check_known_keys().
Eliminate it.

This makes its name available. Rename check_known_keys().

S

qapi: Eliminate check_keys(), rename check_known_keys()

check_keys() has become a trivial wrapper for check_known_keys().
Eliminate it.

This makes its name available. Rename check_known_keys().

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-24-armbru@redhat.com>

show more ...

qapi: Improve reporting of invalid 'if' further

check_if()'s errors don't point to the offending part of the
expression. For instance:

tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if

qapi: Improve reporting of invalid 'if' further

check_if()'s errors don't point to the offending part of the
expression. For instance:

tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if' condition ' ' makes no sense

Other check_FOO() do, with the help of a @source argument. Make
check_if() do that, too. The example above improves to:

tests/qapi-schema/alternate-branch-if-invalid.json:2: 'if' condition ' ' of 'data' member 'branch' makes no sense

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190927134639.4284-23-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

qapi: Avoid redundant definition references in error messages

Many error messages refer to the offending definition even though
they're preceded by an "in definition" line. Rephrase them.

Signed-o

qapi: Avoid redundant definition references in error messages

Many error messages refer to the offending definition even though
they're preceded by an "in definition" line. Rephrase them.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20190927134639.4284-22-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>

show more ...

qapi: Improve reporting of missing / unknown definition keys

Have check_exprs() call check_keys() later, so its error messages gain
an "in definition" line.

Both check_keys() and check_name_is_str(

qapi: Improve reporting of missing / unknown definition keys

Have check_exprs() call check_keys() later, so its error messages gain
an "in definition" line.

Both check_keys() and check_name_is_str() check the definition's name
is a string. Since check_keys() now runs after check_name_is_str()
rather than before, its check is dead. Bury it. Checking values in
check_keys() is unclean anyway.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-21-armbru@redhat.com>

show more ...

qapi: Improve reporting of invalid flags

Split check_flags() off check_keys() and have check_exprs() call it
later, so its error messages gain an "in definition" line. Tweak the
error messages.

Ch

qapi: Improve reporting of invalid flags

Split check_flags() off check_keys() and have check_exprs() call it
later, so its error messages gain an "in definition" line. Tweak the
error messages.

Checking values in a function named check_keys() is unclean anyway.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-20-armbru@redhat.com>

show more ...

qapi: Improve reporting of invalid 'if' errors

Move check_if() from check_keys() to check_exprs() and call it later,
so its error messages gain an "in definition" line.

Checking values in a functio

qapi: Improve reporting of invalid 'if' errors

Move check_if() from check_keys() to check_exprs() and call it later,
so its error messages gain an "in definition" line.

Checking values in a function named check_keys() is unclean anyway.
The original sin was commit 0545f6b887 "qapi: Better error messages
for bad expressions", which checks the value of key 'name'. More
sinning in commit 2cbf09925a "qapi: More rigorous checking for type
safety bypass", commit c818408e44 "qapi: Implement boxed types for
commands/events", and commit 967c885108 "qapi: add 'if' to top-level
expressions". This commit does penance for the latter. The next
commits will do penance for the others.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-19-armbru@redhat.com>

show more ...

qapi: Move context-free checking to the proper place

QAPISchemaCommand.check() and QAPISchemaEvent().check() check 'data'
is present when 'boxed': true. That's context-free. Move to
check_command(

qapi: Move context-free checking to the proper place

QAPISchemaCommand.check() and QAPISchemaEvent().check() check 'data'
is present when 'boxed': true. That's context-free. Move to
check_command() and check_event().

Tweak the error message while there.

check_exprs() & friends now check exactly what qapi-code-gen.txt calls
the second layer of syntax.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-18-armbru@redhat.com>

show more ...

qapi: Move context-sensitive checking to the proper place

When we introduced the QAPISchema intermediate representation (commit
ac88219a6c7), we took a shortcut: we left check_exprs() & friends
alon

qapi: Move context-sensitive checking to the proper place

When we introduced the QAPISchema intermediate representation (commit
ac88219a6c7), we took a shortcut: we left check_exprs() & friends
alone instead of moving semantic checks into the
QAPISchemaFOO.check(). The .check() assert check_exprs() did its job.

Time to finish the conversion job. Move exactly the context-sensitive
checks to the .check(). They replace assertions there. Context-free
checks stay put.

Fixes the misleading optional tag error demonstrated by test
flat-union-optional-discriminator.

A few other error message improve.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-17-armbru@redhat.com>

show more ...

qapi: Inline check_name() into check_union()

check_name() consists of check_name_is_str() and check_name_str().
check_union() relies on the latter to catch optional discriminators.
The next commit w

qapi: Inline check_name() into check_union()

check_name() consists of check_name_is_str() and check_name_str().
check_union() relies on the latter to catch optional discriminators.
The next commit will replace that by a more straightforward check.
Inlining check_name() into check_union() now should make that easier
to review.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-16-armbru@redhat.com>

show more ...

qapi: Plumb info to the QAPISchemaMember

Future commits will need info in the .check() methods of
QAPISchemaMember and its descendants. Get it there.

Signed-off-by: Markus Armbruster <armbru@redha

qapi: Plumb info to the QAPISchemaMember

Future commits will need info in the .check() methods of
QAPISchemaMember and its descendants. Get it there.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-15-armbru@redhat.com>

show more ...

qapi: Make check_type()'s array case a bit more obvious

check_type() checks the array's contents, then peels off the array and
falls through to the "not array" code without resetting allow_array
and

qapi: Make check_type()'s array case a bit more obvious

check_type() checks the array's contents, then peels off the array and
falls through to the "not array" code without resetting allow_array
and allow_dict to False. Works because the peeled value is a string,
and allow_array and allow_dict aren't used then. Tidy up anyway:
recurse instead, defaulting allow_array and allow_dict to False.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-14-armbru@redhat.com>

show more ...

qapi: Move check for reserved names out of add_name()

The checks for reserved names are spread far and wide. Move one from
add_name() to new check_defn_name_str(). This is a first step towards
col

qapi: Move check for reserved names out of add_name()

The checks for reserved names are spread far and wide. Move one from
add_name() to new check_defn_name_str(). This is a first step towards
collecting them all in dedicated name checking functions next to
check_name().

While there, drop the quotes around the meta-type in
check_name_str()'s error messages: "'command' uses ... name 'NAME'"
becomes "command uses ... name 'NAME'".

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-13-armbru@redhat.com>

show more ...

qapi: Report invalid '*' prefix like any other invalid name

The special "does not allow optional name" error is well meant, but
confusing in practice. Drop it.

Signed-off-by: Markus Armbruster <ar

qapi: Report invalid '*' prefix like any other invalid name

The special "does not allow optional name" error is well meant, but
confusing in practice. Drop it.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-12-armbru@redhat.com>

show more ...

qapi: Use check_name_str() where it suffices

Replace check_name() by check_name_str() where the name is known to be
a string.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric

qapi: Use check_name_str() where it suffices

Replace check_name() by check_name_str() where the name is known to be
a string.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-11-armbru@redhat.com>

show more ...

qapi: Improve reporting of invalid name errors

Split check_name() into check_name_is_str() and check_name_str(), keep
check_name() as a wrapper.

Move add_name()'s call into its caller check_exprs()

qapi: Improve reporting of invalid name errors

Split check_name() into check_name_is_str() and check_name_str(), keep
check_name() as a wrapper.

Move add_name()'s call into its caller check_exprs(), and inline.

This permits delaying check_name_str() there, so its error message
gains an "in definition" line.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-10-armbru@redhat.com>

show more ...

qapi: Reorder check_FOO() parameters for consistency

Most check_FOO() take the thing being checked as first argument.
check_name(), check_type(), check_known_keys() don't. Clean that up.

While the

qapi: Reorder check_FOO() parameters for consistency

Most check_FOO() take the thing being checked as first argument.
check_name(), check_type(), check_known_keys() don't. Clean that up.

While there, drop a "Todo" comment that should have been dropped in
commit 87adbbffd4 "qapi: add a dictionary form for TYPE".

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20190927134639.4284-9-armbru@redhat.com>

show more ...