tests/functional: avoid accessing log_filename on earlier failures

If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
mos

tests/functional: avoid accessing log_filename on earlier failures

If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
most notably if the QEMU_TEST_QEMU_BINARY is not set. We can't
initialize 'log_filename' earlier as we use the binary to identify
the architecture which is then used to build the path in which the
logs are stored.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-19-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-19-alex.bennee@linaro.org>

show more ...

tests/functional: add a QMP backdoor for debugging stalled tests

Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most

tests/functional: add a QMP backdoor for debugging stalled tests

Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:

$ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
PYTHONPATH=./python \
./tests/functional/test_arm_tuxrun.py

And then, when the test stalls, in a second shell run:

$ ./scripts/qmp/qmp-shell backdoor.sock

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>

show more ...

tests/functional: enable debug logging for QEMUMachine

Set the 'qemu.machine' logger to 'DEBUG' level, to ensure we see log
messages related to the QEMUMachine class. Most importantly this
ensures w

tests/functional: enable debug logging for QEMUMachine

Set the 'qemu.machine' logger to 'DEBUG' level, to ensure we see log
messages related to the QEMUMachine class. Most importantly this
ensures we capture the full QEMU command line args for instances we
spawn.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-12-berrange@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20241121165806.476008-12-alex.bennee@linaro.org>

show more ...

tests/functional: honour requested test VM name in QEMUMachine

The functional test case class is going to the trouble of passing
around a machine name, but then fails to give this QEMUMachine. As
a

tests/functional: honour requested test VM name in QEMUMachine

The functional test case class is going to the trouble of passing
around a machine name, but then fails to give this QEMUMachine. As
a result, QEMUMachine will create a completely random name. Since
log file names match the machine name, this results in log files
accumulating over time.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-11-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-11-alex.bennee@linaro.org>

show more ...

tests/functional: put QEMUMachine logs in testcase log directory

We are not passing the 'log_dir' parameter to QEMUMachine, so the
QEMU stdout/err logs are being placed in a temp directory and thus

tests/functional: put QEMUMachine logs in testcase log directory

We are not passing the 'log_dir' parameter to QEMUMachine, so the
QEMU stdout/err logs are being placed in a temp directory and thus
deleted after execution. This makes them inaccessible as gitlab
CI artifacts.

Pass the testcase log directory path into QEMUMachine to make the
logs persistent.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-10-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-10-alex.bennee@linaro.org>

show more ...

tests/functional: automatically clean up scratch files after tests

The build/tests/functional subdirectories are consuming huge amounts
of disk space.

Split the location for scratch files into a 's

tests/functional: automatically clean up scratch files after tests

The build/tests/functional subdirectories are consuming huge amounts
of disk space.

Split the location for scratch files into a 'scratch' sub-directory,
separate from log files, and delete it upon completion of each test.
The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve
this scratch dir for debugging access if required.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>

show more ...

tests/functional: Provide the user with hints where to find more log files

Since the base.log and console.log files are not referenced from the
meson test logs yet, they might be hard to find for th

tests/functional: Provide the user with hints where to find more log files

Since the base.log and console.log files are not referenced from the
meson test logs yet, they might be hard to find for the casual users.
Thus let's print some hints in case a test case failed. For this we
have to run unittest.main() with exit=False to get the results of the
testing. Then we can iterate through the failed test cases to print
out the information accordingly.

Message-ID: <20241105123849.359391-1-thuth@redhat.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Tested-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Add QemuUserTest class

Per commit 5334df4822 ("tests/avocado: Introduce
QemuUserTest base class"):

Similarly to the 'System' Test base class with methods
for testing system em

tests/functional: Add QemuUserTest class

Per commit 5334df4822 ("tests/avocado: Introduce
QemuUserTest base class"):

Similarly to the 'System' Test base class with methods
for testing system emulation, the QemuUserTest class
contains methods useful to test user-mode emulation.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240822104238.75045-2-philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240830133841.142644-38-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: enable pre-emptive caching of assets

Many tests need to access assets stored on remote sites. We don't want
to download these during test execution when run by meson, since this
ri

tests/functional: enable pre-emptive caching of assets

Many tests need to access assets stored on remote sites. We don't want
to download these during test execution when run by meson, since this
risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var
QEMU_TEST_PRECACHE to point to a timestamp file. When this is set,
instead of running the test, the assets will be downloaded and saved
to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite
to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class
level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

ninja test --suite functional

will speculatively download any assets that are not already cached,
so it is advisable to set a timeout multiplier.

QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

ninja precache-functional

will download and cache all assets required by the functional
tests

At the make level, precaching is always done by

make check-functional

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
[thuth: Remove the duplicated "path = os.path.basename(...)" line]
Message-ID: <20240830133841.142644-16-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Set up logging

Create log files for each test separately, one file that contains
the basic logging and one that contains the console output.

Reviewed-by: Daniel P. Berrangé <berra

tests/functional: Set up logging

Create log files for each test separately, one file that contains
the basic logging and one that contains the console output.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-10-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Add base classes for the upcoming pytest-based tests

The files are mostly a copy of the tests/avocado/avocado_qemu/__init__.py
file with some adjustments to get rid of the Avocado

tests/functional: Add base classes for the upcoming pytest-based tests

The files are mostly a copy of the tests/avocado/avocado_qemu/__init__.py
file with some adjustments to get rid of the Avocado dependencies (i.e.
we also have to drop the LinuxSSHMixIn and LinuxTest for now).

The emulator binary and build directory are now passed via
environment variables that will be set via meson.build later.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-9-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...