tests/functional: Move debian boot test from avocado

This simply moves the debian boot test from the avocado testsuite to
the new functional testsuite. No changes in the test.

Reviewed-by: Thomas H

tests/functional: Move debian boot test from avocado

This simply moves the debian boot test from the avocado testsuite to
the new functional testsuite. No changes in the test.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-8-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for rainier-bmc machine

This simply moves the rainier-bmc test to a new test file. No changes
in the test. The test_arm_aspeed.py is deleted.

Link: https

tests/functional: Introduce a specific test for rainier-bmc machine

This simply moves the rainier-bmc test to a new test file. No changes
in the test. The test_arm_aspeed.py is deleted.

Link: https://lore.kernel.org/r/20241206131132.520911-7-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for ast2600 SoC

This moves the ast2600-evb tests to a new test file. No changes in the
test. The routines used to run the buildroot and sdk tests are remo

tests/functional: Introduce a specific test for ast2600 SoC

This moves the ast2600-evb tests to a new test file. No changes in the
test. The routines used to run the buildroot and sdk tests are removed
from the test_arm_aspeed.py file because now unused.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-6-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for ast2500 SoC

This moves the ast2500-evb tests to a new test file and extends the
aspeed module with routines used to run the buildroot and sdk
tests. N

tests/functional: Introduce a specific test for ast2500 SoC

This moves the ast2500-evb tests to a new test file and extends the
aspeed module with routines used to run the buildroot and sdk
tests. No changes in the test.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-5-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for romulus-bmc machine

This simply moves the romulus-bmc test to a new test file. No changes
in the test. The do_test_arm_aspeed routine is removed from

tests/functional: Introduce a specific test for romulus-bmc machine

This simply moves the romulus-bmc test to a new test file. No changes
in the test. The do_test_arm_aspeed routine is removed from the
test_arm_aspeed.py file because it is now unused.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-4-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for palmetto-bmc machine

This introduces a new aspeed module for sharing code between tests and
moves the palmetto test to a new test file. No changes in

tests/functional: Introduce a specific test for palmetto-bmc machine

This introduces a new aspeed module for sharing code between tests and
moves the palmetto test to a new test file. No changes in the test.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-3-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional: Introduce a specific test for ast1030 SoC

This simply moves the ast1030 tests to a new test file. No changes.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel

tests/functional: Introduce a specific test for ast1030 SoC

This simply moves the ast1030 tests to a new test file. No changes.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Link: https://lore.kernel.org/r/20241206131132.520911-2-clg@redhat.com
Signed-off-by: Cédric Le Goater <clg@redhat.com>

show more ...

tests/functional/test_rx_gdbsim: Use stable URL for test_linux_sash

Yoshinori said [*] URL references on OSDN were stable, but they
appear not to be. Mirror the artifacts on GitHub to avoid failures

tests/functional/test_rx_gdbsim: Use stable URL for test_linux_sash

Yoshinori said [*] URL references on OSDN were stable, but they
appear not to be. Mirror the artifacts on GitHub to avoid failures
while testing on CI.

[*] https://www.mail-archive.com/qemu-devel@nongnu.org/msg686487.html

Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Reported-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-ID: <20200630202631.7345-1-f4bug@amsat.org>
[huth: Adapt the patch to the new version in the functional framework]
Message-ID: <20241229083419.180423-1-huth@tuxfamily.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
(cherry picked from commit ec2dfb7c389b94d71ee825caa20b709d5df6c166)
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
(Mjt: fixup for missing v9.2.0-421-g65d35a4e27a8 "tests/functional: convert tests to new uncompress helper")

show more ...

tests/functional: increase timeouts for arm sx1 test

When under high load the test VM does not complete running in the
default 30 second timeout. Double it to give more headroom.

Signed-off-by: Dan

tests/functional: increase timeouts for arm sx1 test

When under high load the test VM does not complete running in the
default 30 second timeout. Double it to give more headroom.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241129173120.761728-2-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional/test_virtio_version: Check for the availability of the machine

Use self_set_machine() to set and check for the availability of the
default pc machine (so that the test is not failin

tests/functional/test_virtio_version: Check for the availability of the machine

Use self_set_machine() to set and check for the availability of the
default pc machine (so that the test is not failing if the machine
has not been included in the QEMU binary).

Message-ID: <20241128120142.593408-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional/test_acpi_bits: Turn the test into a QemuSystemTest

By using QemuSystemTest as a base class, we can use the set_machine()
command to check whether the required machine is available

tests/functional/test_acpi_bits: Turn the test into a QemuSystemTest

By using QemuSystemTest as a base class, we can use the set_machine()
command to check whether the required machine is available in the
binary (otherwise this test is failing when QEMU has been compiled
without the default 'pc' machine type).

Message-ID: <20241128115019.591362-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Ani Sinha <anisinha@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Remove sleep workarounds from sh4 test

These were introduced in the avocado tests to workaround read issues
when interacting with console. They are no longer necessary and we can
u

tests/functional: Remove sleep workarounds from sh4 test

These were introduced in the avocado tests to workaround read issues
when interacting with console. They are no longer necessary and we can
use the expected login string instead.

Test always passes now. Remove skipUnless test on QEMU_TEST_FLAKY_TESTS.

Signed-off-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20241122141827.2039984-1-clg@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Fix the running test case causes loongarch64 to hang

There is a bug in the process of resolving the serial port base address
in the fdt of the loongarch VM UEFI. When both serial p

tests/functional: Fix the running test case causes loongarch64 to hang

There is a bug in the process of resolving the serial port base address
in the fdt of the loongarch VM UEFI. When both serial port information
and rng-seed information are chosen in the fdt, there is a probability
that the serial port base address cannot be resolved correctly.
This problem can be fixed by updating UEFI.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2686
Signed-off-by: Xianglai Li <lixianglai@loongson.cn>
Message-ID: <20241127013438.2206426-1-lixianglai@loongson.cn>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>

show more ...

tests/functional: Remove sleep workarounds from Aspeed tests

These were introduced in the avocado tests to workaround read issues
when interacting with console. They are no longer necessary and we c

tests/functional: Remove sleep workarounds from Aspeed tests

These were introduced in the avocado tests to workaround read issues
when interacting with console. They are no longer necessary and we can
use the expected "login:" string or the command prompt now. Drop the
last use of exec_command.

Signed-off-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20241122090322.1934697-4-clg@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

tests/functional: Convert Aspeed arm SDK tests

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

EXTRA_BOOTARGS was introduc

tests/functional: Convert Aspeed arm SDK tests

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

EXTRA_BOOTARGS was introduced to reduce the console output at Linux
boot time. This didn't have the desired effect as we still had issues
when trying to match patterns on the console and we had to use the ssh
connection as a workaround.

While at it, remove the U-Boot EXTRA_BOOTARGS variable which has
become useless.

Signed-off-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20241122090322.1934697-3-clg@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

tests/functional: Convert Aspeed aarch64 SDK tests

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

Signed-off-by: Cédric L

tests/functional: Convert Aspeed aarch64 SDK tests

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

Signed-off-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20241122090322.1934697-2-clg@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

show more ...

tests/functional: update the aarch64 tuxrun tests

Now there are new up to date images available we should update to them.
With the new rootfs the blk I/O errors also go away on arm64be.

Cc: Anders

tests/functional: update the aarch64 tuxrun tests

Now there are new up to date images available we should update to them.
With the new rootfs the blk I/O errors also go away on arm64be.

Cc: Anders Roxell <anders.roxell@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-22-alex.bennee@linaro.org>

show more ...

tests/functional: Convert the Avocado aarch64 tuxrun tests

Move the tests to a new file so that they can be run via
qemu-system-aarch64 in the functional framework.

Since these were the last tests

tests/functional: Convert the Avocado aarch64 tuxrun tests

Move the tests to a new file so that they can be run via
qemu-system-aarch64 in the functional framework.

Since these were the last tests in tests/avocado/tuxrun_baselines.py,
we can now remove that file, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-20-alex.bennee@linaro.org>

show more ...

tests/functional: avoid accessing log_filename on earlier failures

If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
mos

tests/functional: avoid accessing log_filename on earlier failures

If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
most notably if the QEMU_TEST_QEMU_BINARY is not set. We can't
initialize 'log_filename' earlier as we use the binary to identify
the architecture which is then used to build the path in which the
logs are stored.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-19-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-19-alex.bennee@linaro.org>

show more ...

tests/functional: add a QMP backdoor for debugging stalled tests

Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most

tests/functional: add a QMP backdoor for debugging stalled tests

Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:

$ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
PYTHONPATH=./python \
./tests/functional/test_arm_tuxrun.py

And then, when the test stalls, in a second shell run:

$ ./scripts/qmp/qmp-shell backdoor.sock

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>

show more ...

tests/functional: remove time.sleep usage from tuxrun tests

The tuxrun tests send a series of strings to the guest to login
and then run commands. Since we have been unable to match on
console outpu

tests/functional: remove time.sleep usage from tuxrun tests

The tuxrun tests send a series of strings to the guest to login
and then run commands. Since we have been unable to match on
console output that isn't followed by a newline, the test used
many time.sleep() statements to pretend to synchronize with
the guest.

This has proved to be unreliable for the aarch64be instance of
the tuxrun tests, with the test often hanging. The hang is a
very subtle timing problem, and it is suspected that some
(otherwise apparently harmless) I/O error messages could be
resulting in full FIFO buffers, stalling interaction with
the guest.

With the newly rewritten console interaction able to match
strings that don't have a following newline, the tux run
tests can now match directly on the login prompt, and/or
shell PS1 prompt.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2689
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-17-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-17-alex.bennee@linaro.org>

show more ...

tests/functional: rewrite console handling to be bytewise

The console interaction that waits for predicted strings uses
readline(), and thus is only capable of waiting for strings
that are followed

tests/functional: rewrite console handling to be bytewise

The console interaction that waits for predicted strings uses
readline(), and thus is only capable of waiting for strings
that are followed by a newline.

This is inconvenient when needing to match on some things,
particularly login prompts, or shell prompts, causing tests
to use time.sleep(...) instead, which is unreliable.

Switch to reading the console 1 byte at a time, comparing
against the success/failure messages until we see a match,
regardless of whether a newline is encountered.

The success/failure comparisons are done with the python bytes
type, rather than strings, to avoid the problem of needing to
decode partially received multibyte utf8 characters.

Heavily inspired by a patch proposed by Cédric, but written
again to work in bytes, rather than strings.

Co-developed-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-16-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-16-alex.bennee@linaro.org>

show more ...

tests/functional: require non-NULL success_message for console wait

When waiting for expected output, the 'success_message' is a mandatory
parameter, with 'failure_message' defaulting to None.

The

tests/functional: require non-NULL success_message for console wait

When waiting for expected output, the 'success_message' is a mandatory
parameter, with 'failure_message' defaulting to None.

The code has logic which indicates it was trying to cope with
'success_message' being None and 'failure_message' being non-None but
it does not appear able to actually do anything useful. The check for
'success_message is None' will break out of the loop before any check
for 'failure_message' has been performed.

IOW, for practcal purposes 'success_message' must be non-None unless
'send_string' is set. Assert this expectation and simplify the loop
logic.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-15-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-15-alex.bennee@linaro.org>

show more ...

tests/functional: don't try to wait for the empty string

Telling exec_command_wand_wait_for_pattern to wait for the empty
string does not make any conceptual sense, as a check for empty
string will

tests/functional: don't try to wait for the empty string

Telling exec_command_wand_wait_for_pattern to wait for the empty
string does not make any conceptual sense, as a check for empty
string will always succeed. It makes even less sense when followed
by a call to wait_for_console_pattern() with a real match.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-14-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-14-alex.bennee@linaro.org>

show more ...

tests/functional: logs details of console interaction operations

When functional tests go wrong, it will often be related to the console
interaction wait state. By logging the messages that we're lo

tests/functional: logs details of console interaction operations

When functional tests go wrong, it will often be related to the console
interaction wait state. By logging the messages that we're looking for,
and data we're about to be sending, it'll be easier to diagnose where
tests are getting stuck.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-13-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-13-alex.bennee@linaro.org>

show more ...