python: re-enable pylint duplicate-code warnings

With the old library gone, there's nothing duplicated in the tree, so
the warning suppression can be removed.

Signed-off-by: John Snow <jsnow@redhat

python: re-enable pylint duplicate-code warnings

With the old library gone, there's nothing duplicated in the tree, so
the warning suppression can be removed.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220330172812.3427355-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python: remove the old QMP package

Thank you for your service!

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo

python: remove the old QMP package

Thank you for your service!

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220330172812.3427355-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: copy qmp docstrings to qemu.aqmp.legacy

Copy the docstrings out of qemu.qmp, adjusting them as necessary to
more accurately reflect the current state of this class.

(Licensing: This is

python/aqmp: copy qmp docstrings to qemu.aqmp.legacy

Copy the docstrings out of qemu.qmp, adjusting them as necessary to
more accurately reflect the current state of this class.

(Licensing: This is copying and modifying GPLv2-only licensed docstrings
into a GPLv2-only file.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220330172812.3427355-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: fully separate from qmp.QEMUMonitorProtocol

After this patch, qemu.aqmp.legacy.QEMUMonitorProtocol no longer
inherits from qemu.qmp.QEMUMonitorProtocol. To do this, several
inherited me

python/aqmp: fully separate from qmp.QEMUMonitorProtocol

After this patch, qemu.aqmp.legacy.QEMUMonitorProtocol no longer
inherits from qemu.qmp.QEMUMonitorProtocol. To do this, several
inherited methods need to be explicitly re-defined.

(Licensing: This is copying and modifying GPLv2-only code into a
GPLv2-only file.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220330172812.3427355-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: take QMPBadPortError and parse_address from qemu.qmp

Shift these definitions over from the qmp package to the async qmp
package.

(Licensing: this is a lateral move, from GPLv2 (only) t

python/aqmp: take QMPBadPortError and parse_address from qemu.qmp

Shift these definitions over from the qmp package to the async qmp
package.

(Licensing: this is a lateral move, from GPLv2 (only) to GPLv2 (only))

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220330172812.3427355-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python: temporarily silence pylint duplicate-code warnings

The next several commits copy some code from qemu.qmp to qemu.aqmp, then
delete qemu.qmp. In the interim, to prevent test failures, the dup

python: temporarily silence pylint duplicate-code warnings

The next several commits copy some code from qemu.qmp to qemu.aqmp, then
delete qemu.qmp. In the interim, to prevent test failures, the duplicate
code detection needs to be silenced to prevent bisect problems with CI
testing.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-id: 20220330172812.3427355-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp-tui: relicense as LGPLv2+

aqmp-tui, the async QMP text user interface tool, is presently licensed
as GPLv2+. I intend to include this tool as an add-on to an LGPLv2+
library package host

python/aqmp-tui: relicense as LGPLv2+

aqmp-tui, the async QMP text user interface tool, is presently licensed
as GPLv2+. I intend to include this tool as an add-on to an LGPLv2+
library package hosted on PyPI.org. I've selected LGPLv2+ to maximize
compatibility with other licenses while retaining a copyleft license.

To keep licensing matters simple, I'd like to relicense this tool as
LGPLv2+ as well in order to keep the resultant license of the hosted
release files simple -- even if library users won't "link against" this
command line tool.

Therefore, I am asking permission to loosen the license.

Niteesh is effectively the sole author of this code, with scattered
lines from myself.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: G S Niteesh Babu <niteesh.gs@gmail.com>
Message-id: 20220325200438.2556381-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/qmp-shell: relicense as LGPLv2+

qmp-shell is presently licensed as GPLv2 (only). I intend to include
this tool as an add-on to an LGPLv2+ library package hosted on
PyPI.org. I've selected LGP

python/qmp-shell: relicense as LGPLv2+

qmp-shell is presently licensed as GPLv2 (only). I intend to include
this tool as an add-on to an LGPLv2+ library package hosted on
PyPI.org. I've selected LGPLv2+ to maximize compatibility with other
licenses while retaining a copyleft license.

To keep licensing matters simple, I'd like to relicense this tool as
LGPLv2+ as well in order to keep the resultant license of the hosted
release files simple -- even if library users won't "link against" this
command line tool.

Therefore, I am asking permission from the current authors of this
tool to loosen the license. At present, those people are:

- John Snow (me!), 411/609
- Luiz Capitulino, Author, 97/609
- Daniel Berrangé, 81/609
- Eduardo Habkost, 10/609
- Marc-André Lureau, 6/609
- Fam Zheng, 3/609
- Cleber Rosa, 1/609

(All of which appear to have been written under redhat.com addresses.)

Eduardo's fixes are largely automated from 2to3 conversion tools and may
not necessarily constitute authorship, but his signature would put to
rest any questions.

Cleber's changes concern a single import statement change. Also won't
hurt to ask.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Acked-by: Fam Zheng <fam@euphon.net>
Acked-by: Luiz Capitulino <lcapitulino@redhat.com>
Acked-by: Eduardo Habkost <eduardo@habkost.net>
Acked-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20220325200438.2556381-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: relicense as LGPLv2+

I am the sole author of all of the async QMP code (python/qemu/aqmp)
with the following exceptions:

python/qemu/aqmp/qmp_shell.py and python/qemu/aqmp/legacy.py we

python/aqmp: relicense as LGPLv2+

I am the sole author of all of the async QMP code (python/qemu/aqmp)
with the following exceptions:

python/qemu/aqmp/qmp_shell.py and python/qemu/aqmp/legacy.py were
written by Luiz Capitulino (et al) and are already licensed separately
as GPLv2 (only).

aqmp_tui.py was written by Niteesh Babu G S and is licensed as GPLv2+.

I wish to relicense as LGPLv2+ in order to provide as much flexibility
as I reasonably can, while retaining a copyleft license. It is my belief
that LGPLv2+ is a suitable license for the Python ecosystem that aligns
with the goals and philosophy of the QEMU project.

The intent is to eventually drop legacy.py, leaving only library code
that is LGPLv2+.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220325200438.2556381-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: add explicit GPLv2 license to legacy.py

The legacy.py module is heavily based on the QMP module by Luiz
Capitulino (et al) which is licensed as explicit GPLv2-only. The async
QMP packag

python/aqmp: add explicit GPLv2 license to legacy.py

The legacy.py module is heavily based on the QMP module by Luiz
Capitulino (et al) which is licensed as explicit GPLv2-only. The async
QMP package is currently licensed similarly, but I intend to relicense
the async package to the more flexible LGPLv2+.

In preparation for that change, make the license on legacy.py explicit.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220325200438.2556381-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/machine: permanently switch to AQMP

Remove the QEMU_PYTHON_LEGACY_QMP environment variable, making the
switch from sync qmp to async qmp permanent. Update exceptions and
import paths as neces

python/machine: permanently switch to AQMP

Remove the QEMU_PYTHON_LEGACY_QMP environment variable, making the
switch from sync qmp to async qmp permanent. Update exceptions and
import paths as necessary.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Acked-by: Hanna Reitz <hreitz@redhat.com>
Message-id: 20220321203315.909411-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/utils: add VerboseProcessError

This adds an Exception that extends the Python stdlib
subprocess.CalledProcessError.

The difference is that the str() method of this exception also adds the
st

python/utils: add VerboseProcessError

This adds an Exception that extends the Python stdlib
subprocess.CalledProcessError.

The difference is that the str() method of this exception also adds the
stdout/stderr logs. In effect, if this exception goes unhandled, Python
will print the output in a visually distinct wrapper to the terminal so
that it's easy to spot in a sea of traceback information.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220321201618.903471-3-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

python/utils: add add_visual_margin() text decoration utility

>>> print(add_visual_margin(msg, width=72, name="Commit Message"))
┏━ Commit Message ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

python/utils: add add_visual_margin() text decoration utility

>>> print(add_visual_margin(msg, width=72, name="Commit Message"))
┏━ Commit Message ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
┃ add_visual_margin() takes a chunk of text and wraps it in a visual
┃ container that force-wraps to a specified width. An optional title
┃ label may be given, and any of the individual glyphs used to draw the
┃ box may be replaced or specified as well.
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Acked-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220321201618.903471-2-jsnow@redhat.com>
Signed-off-by: Hanna Reitz <hreitz@redhat.com>

show more ...

python/aqmp: drop _bind_hack()

_bind_hack() was a quick fix to allow async QMP to call bind(2) prior to
calling listen(2) and accept(2). This wasn't sufficient to fully address
the race condition pr

python/aqmp: drop _bind_hack()

_bind_hack() was a quick fix to allow async QMP to call bind(2) prior to
calling listen(2) and accept(2). This wasn't sufficient to fully address
the race condition present in synchronous clients.

With the race condition in legacy.py fixed (see the previous commit),
there are no longer any users of _bind_hack(). Drop it.

Fixes: b0b662bb2b3
Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-11-jsnow@redhat.com
[Expanded commit message. --js]
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: fix race condition in legacy.py

legacy.py provides a synchronous model. iotests frequently uses this
paradigm:

- create QMP client object
- start QEMU process
- await connection fro

python/aqmp: fix race condition in legacy.py

legacy.py provides a synchronous model. iotests frequently uses this
paradigm:

- create QMP client object
- start QEMU process
- await connection from QEMU process

In the switch from sync to async QMP, the QMP client object stopped
calling bind() and listen() during the QMP object creation step, which
creates a race condition if the QEMU process dials in too quickly.

With refactoring out of the way, restore the former behavior of calling
bind() and listen() during __init__() to fix this race condition.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-10-jsnow@redhat.com
[Expanded commit message. --js]
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: add start_server() and accept() methods

Add start_server() and accept() methods that can be used instead of
start_server_and_accept() to allow more fine-grained control over the
incomin

python/aqmp: add start_server() and accept() methods

Add start_server() and accept() methods that can be used instead of
start_server_and_accept() to allow more fine-grained control over the
incoming connection process.

(Eagle-eyed reviewers will surely notice that it's a bit weird that
"CONNECTING" is a state that's shared between both the start_server()
and connect() states. That's absolutely true, and it's very true that
checking on the presence of _accepted as an indicator of state is a
hack. That's also very certainly true. But ... this keeps client code an
awful lot simpler, as it doesn't have to care exactly *how* the
connection is being made, just that it *is*. Is it worth disrupting that
simplicity in order to provide a better state guard on `accept()`? Hm.)

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: stop the server during disconnect()

Before we allow the full separation of starting the server and accepting
new connections, make sure that the disconnect cleans up the server and
its

python/aqmp: stop the server during disconnect()

Before we allow the full separation of starting the server and accepting
new connections, make sure that the disconnect cleans up the server and
its new state, too.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: refactor _do_accept() into two distinct steps

Refactor _do_accept() into _do_start_server() and _do_accept(). As of
this commit, the former calls the latter, but in subsequent commits
t

python/aqmp: refactor _do_accept() into two distinct steps

Refactor _do_accept() into _do_start_server() and _do_accept(). As of
this commit, the former calls the latter, but in subsequent commits
they'll be split apart.

(So please forgive the misnomer for _do_start_server(); it will live up
to its name shortly, and the docstring will be updated then too. I'm
just cutting down on some churn.)

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: squelch pylint warning for too many lines

I would really like to keep this under 1000 lines, I promise. Doesn't
look like it's gonna happen.

Signed-off-by: John Snow <jsnow@redhat.com>

python/aqmp: squelch pylint warning for too many lines

I would really like to keep this under 1000 lines, I promise. Doesn't
look like it's gonna happen.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: split _client_connected_cb() out as _incoming()

As part of disentangling the monolithic nature of _do_accept(), split
out the incoming callback to prepare for factoring out the "wait fo

python/aqmp: split _client_connected_cb() out as _incoming()

As part of disentangling the monolithic nature of _do_accept(), split
out the incoming callback to prepare for factoring out the "wait for a
peer" step. Namely, this means using an event signal we can wait on from
outside of this method.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: remove _new_session and _establish_connection

These two methods attempted to entirely envelop the logic of
establishing a connection to a peer start to finish. However, we need to
break

python/aqmp: remove _new_session and _establish_connection

These two methods attempted to entirely envelop the logic of
establishing a connection to a peer start to finish. However, we need to
break apart the incoming connection step into more granular steps. We
will no longer be able to reasonably constrain the logic inside of these
helper functions.

So, remove them - with _session_guard(), they no longer serve a real
purpose.

Although the public API doesn't change, the internal API does. Now that
there are no intermediary methods between e.g. connect() and
_do_connect(), there's no hook where the runstate is set. As a result,
the test suite changes a little to cope with the new semantics of
_do_accept() and _do_connect().

Lastly, take some pieces of the now-deleted docstrings and move
them up to the public interface level. They were a little more detailed,
and it won't hurt to keep them.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: rename 'accept()' to 'start_server_and_accept()'

Previously, I had a method named "accept()" that under-the-hood calls
bind(2), listen(2) *and* accept(2). I meant this as a simplificati

python/aqmp: rename 'accept()' to 'start_server_and_accept()'

Previously, I had a method named "accept()" that under-the-hood calls
bind(2), listen(2) *and* accept(2). I meant this as a simplification and
counterpart to the one-shot "connect()" method.

This is confusing to readers who expect accept() to mean *just*
accept(2). Since I need to split apart the "accept()" method into
multiple methods anyway (one of which strongly resembling accept(2)), it
feels pertinent to rename this method *now*.

Rename this all-in-one method "start_server_and_accept()" instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

python/aqmp: add _session_guard()

In _new_session, there's a fairly complex except clause that's used to
give semantic errors to callers of accept() and connect(). We need to
create a new two-step r

python/aqmp: add _session_guard()

In _new_session, there's a fairly complex except clause that's used to
give semantic errors to callers of accept() and connect(). We need to
create a new two-step replacement for accept(), so factoring out this
piece of logic will be useful.

Bolster the comments and docstring here to try and demystify what's
going on in this fairly delicate piece of Python magic.

(If we were using Python 3.7+, this would be an @asynccontextmanager. We
don't have that very nice piece of magic, however, so this must take an
Awaitable to manage the Exception contexts properly. We pay the price
for platform compatibility.)

Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20220225205948.3693480-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

Revert "python: pin setuptools below v60.0.0"

This reverts commit 1e4d8b31be35e54b6429fea54f5ecaa0083f91e7.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220204221804.2047468-3-jsnow@re

Revert "python: pin setuptools below v60.0.0"

This reverts commit 1e4d8b31be35e54b6429fea54f5ecaa0083f91e7.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220204221804.2047468-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...

Python: add setuptools v60.0 workaround

Setuptools v60 and later include a bundled version of distutils, a
deprecated standard library scheduled for removal in future versions of
Python. Setuptools

Python: add setuptools v60.0 workaround

Setuptools v60 and later include a bundled version of distutils, a
deprecated standard library scheduled for removal in future versions of
Python. Setuptools v60 is only possible to install for Python 3.7 and later.

Python has a distutils.sysconfig.get_python_lib() function that returns
'/usr/lib/pythonX.Y' on posix systems. RPM-based systems actually use
'/usr/lib64/pythonX.Y' instead, so Fedora patches stdlib distutils for
Python 3.7 and Python 3.8 to return the correct value.

Python 3.9 and later introduce a sys.platlibdir property, which returns
the correct value on RPM-based systems.

The change to a distutils package not provided by Fedora on Python 3.7
and 3.8 causes a regression in distutils.sysconfig.get_python_lib() that
ultimately causes false positives to be emitted by pylint, because it
can no longer find the system source libraries.

Many Python tools are fairly aggressive about updating setuptools
packages, and so even though this package is a fair bit newer than
Python 3.7/3.8, it's not entirely unreasonable for a given user to have
such a modern package with a fairly old Python interpreter.

Updates to Python 3.7 and Python 3.8 are being produced for Fedora which
will fix the problem on up-to-date systems. Until then, we can force the
loading of platform-provided distutils when running the pylint
test. This is the least-invasive yet most comprehensive fix.

References:
https://github.com/pypa/setuptools/pull/2896
https://github.com/PyCQA/pylint/issues/5704
https://github.com/pypa/distutils/issues/110

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220204221804.2047468-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>

show more ...