| 723c8258 | 25-Jul-2023 |
Rae Moar <rmoar@google.com> |
kunit: tool: Add command line interface to filter and report attributes
Add ability to kunit.py to filter attributes and report a list of tests
including attributes without running tests.
Add flag
kunit: tool: Add command line interface to filter and report attributes
Add ability to kunit.py to filter attributes and report a list of tests
including attributes without running tests.
Add flag "--filter" to input filters on test attributes. Tests will be
filtered out if they do not match all inputted filters.
Example: --filter speed=slow (This filter would run only the tests that are
marked as slow)
Filters have operations: <, >, <=, >=, !=, and =. But note that the
characters < and > are often interpreted by the shell, so they may need to
be quoted or escaped.
Example: --filter "speed>slow" or --filter speed\>slow (This filter would
run only the tests that have the speed faster than slow.
Additionally, multiple filters can be used.
Example: --filter "speed=slow, module!=example" (This filter would run
only the tests that have the speed slow and are not in the "example"
module)
Note if the user wants to skip filtered tests instead of not
running/showing them use the "--filter_action=skip" flag instead.
Expose the output of kunit.action=list option with flag "--list_tests" to
output a list of tests. Additionally, add flag "--list_tests_attr" to
output a list of tests and their attributes. These flags are useful to see
tests and test attributes without needing to run tests.
Example of the output of "--list_tests_attr":
example
example.test_1
example.test_2
# example.test_2.speed: slow
This output includes a suite, example, with two test cases, test_1 and
test_2. And in this instance test_2 has been marked as slow.
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| b7dc237e | 17-Jul-2023 |
Mark Brown <broonie@kernel.org> |
ASoC: topology: Add explicit build option
The default KUnit build options are not supposed to enable any
subsystems that were not already enabled but the topology code is a
library which is generall
ASoC: topology: Add explicit build option
The default KUnit build options are not supposed to enable any
subsystems that were not already enabled but the topology code is a
library which is generally selected by drivers that want to use it.
Since KUnit is frequently run in virtual environments with minimal
driver support this makes it difficult to enable the toplogy tests so
provide an explicit Kconfig option which can be directly enabled when
using KUnit, and also include this in the KUnit all_tests.config.
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Takashi Iwai <tiwai@suse.de>
Signed-off-by: Mark Brown <broonie@kernel.org>
Link: https://lore.kernel.org/r/20230718-asoc-topology-kunit-enable-v2-5-0ee11e662b92@kernel.org
Signed-off-by: Mark Brown <broonie@kernel.org>
show more ...
|
| 8110a3ca | 28-Feb-2023 |
Geert Uytterhoeven <geert+renesas@glider.be> |
kunit: tool: Add support for SH under QEMU
Add basic support to run SH under QEMU via kunit_tool using the
virtualized r2d platform.
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Revi
kunit: tool: Add support for SH under QEMU
Add basic support to run SH under QEMU via kunit_tool using the
virtualized r2d platform.
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 5ffb8629 | 28-Feb-2023 |
Geert Uytterhoeven <geert+renesas@glider.be> |
kunit: tool: Add support for overriding the QEMU serial port
On some platforms, the console is not the first serial port. To make
this work, the first serial port in QEMU must be set to "null".
Ad
kunit: tool: Add support for overriding the QEMU serial port
On some platforms, the console is not the first serial port. To make
this work, the first serial port in QEMU must be set to "null".
Add support for this by adding an optional "serial" parameter, which
defaults to "stdio", and can be overridden by platform-specific
configuration.
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 1da2e622 | 16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: fix pre-existing `mypy --strict` errors and update run_checks.py
Basically, get this command to be happy and make run_checks.py happy
$ mypy --strict --exclude '_test.py$' --exclude qe
kunit: tool: fix pre-existing `mypy --strict` errors and update run_checks.py
Basically, get this command to be happy and make run_checks.py happy
$ mypy --strict --exclude '_test.py$' --exclude qemu_configs/ ./tools/testing/kunit/
Primarily the changes are
* add `-> None` return type annotations
* add all the missing argument type annotations
Previously, we had false positives from mypy in `main()`, see commit
09641f7c7d8f ("kunit: tool: surface and address more typing issues").
But after commit 2dc9d6ca52a4 ("kunit: kunit.py extract handlers")
refactored things, the variable name reuse mypy hated is gone.
Note: mypy complains we don't annotate the types the unused args in our
signal handler. That's silly.
But to make it happy, I've copy-pasted an appropriate annotation from
https://github.com/python/typing/discussions/1042#discussioncomment-2013595.
Reported-by: Johannes Berg <johannes.berg@intel.com>
Link: https://lore.kernel.org/linux-kselftest/9a172b50457f4074af41fe1dc8e55dcaf4795d7e.camel@sipsolutions.net/
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 126901ba | 16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: remove unused imports and variables
We don't run a linter regularly over kunit.py code (the default settings
on most don't like kernel style, e.g. tabs) so some of these imports
didn't
kunit: tool: remove unused imports and variables
We don't run a linter regularly over kunit.py code (the default settings
on most don't like kernel style, e.g. tabs) so some of these imports
didn't get removed when they stopped being used.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 695e2603 | 16-Mar-2023 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: add subscripts for type annotations where appropriate
E.g. for subprocess.Popen, it can be opened in `text=True` mode where it
returns strings, or `text=False` where it returns bytes.
T
kunit: tool: add subscripts for type annotations where appropriate
E.g. for subprocess.Popen, it can be opened in `text=True` mode where it
returns strings, or `text=False` where it returns bytes.
To differentiate, you can annotate types as `Popen[str]` or
`Popen[bytes]`.
This patch should add subscripts in all the places we were missing them.
Reported-by: Johannes Berg <johannes.berg@intel.com>
Link: https://lore.kernel.org/linux-kselftest/20230315105055.9b2be0153625.I7a2cb99b95dff216c0feed4604255275e0b156a7@changeid/
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 2dc9d6ca | 21-Jan-2023 |
Alexander Pantyukhin <apantykhin@gmail.com> |
kunit: kunit.py extract handlers
The main function contains a wide if-elif block that handles different
subcommands. It's possible to make code refactoring to extract
subcommands handlers.
Fixed co
kunit: kunit.py extract handlers
The main function contains a wide if-elif block that handles different
subcommands. It's possible to make code refactoring to extract
subcommands handlers.
Fixed commit summary line.
Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Alexander Pantyukhin <apantykhin@gmail.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| c2bb92bc | 30-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: make parser preserve whitespace when printing test log
Currently, kunit_parser.py is stripping all leading whitespace to make
parsing easier. But this means we can't accurately show ker
kunit: tool: make parser preserve whitespace when printing test log
Currently, kunit_parser.py is stripping all leading whitespace to make
parsing easier. But this means we can't accurately show kernel output
for failing tests or when the kernel crashes.
Embarassingly, this affects even KUnit's own output, e.g.
[13:40:46] Expected 2 + 1 == 2, but
[13:40:46] 2 + 1 == 3 (0x3)
[13:40:46] not ok 1 example_simple_test
[13:40:46] [FAILED] example_simple_test
After this change, here's what the output in context would look like
[13:40:46] =================== example (4 subtests) ===================
[13:40:46] # example_simple_test: initializing
[13:40:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[13:40:46] Expected 2 + 1 == 2, but
[13:40:46] 2 + 1 == 3 (0x3)
[13:40:46] [FAILED] example_simple_test
[13:40:46] [SKIPPED] example_skip_test
[13:40:46] [SKIPPED] example_mark_skipped_test
[13:40:46] [PASSED] example_all_expect_macros_test
[13:40:46] # example: initializing suite
[13:40:46] # example: pass:1 fail:1 skip:2 total:4
[13:40:46] # Totals: pass:1 fail:1 skip:2 total:4
[13:40:46] ===================== [FAILED] example =====================
This example shows one minor cosmetic defect this approach has.
The test counts lines prevent us from dedenting the suite-level output.
But at the same time, any form of non-KUnit output would do the same
unless it happened to be indented as well.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 5937e0c0 | 28-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: don't include KTAP headers and the like in the test log
We print the "test log" on failure.
This is meant to be all the kernel output that happened during the test.
But we also include
kunit: tool: don't include KTAP headers and the like in the test log
We print the "test log" on failure.
This is meant to be all the kernel output that happened during the test.
But we also include the special KTAP lines in it, which are often
redundant.
E.g. we include the "not ok" line in the log, right before we print
that the test case failed...
[13:51:48] Expected 2 + 1 == 2, but
[13:51:48] 2 + 1 == 3 (0x3)
[13:51:48] not ok 1 example_simple_test
[13:51:48] [FAILED] example_simple_test
More full example after this patch:
[13:51:48] =================== example (4 subtests) ===================
[13:51:48] # example_simple_test: initializing
[13:51:48] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
[13:51:48] Expected 2 + 1 == 2, but
[13:51:48] 2 + 1 == 3 (0x3)
[13:51:48] [FAILED] example_simple_test
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 309e22ef | 21-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: make --json do nothing if --raw_ouput is set
When --raw_output is set (to any value), we don't actually parse the
test results. So asking to print the test results as json doesn't make
kunit: tool: make --json do nothing if --raw_ouput is set
When --raw_output is set (to any value), we don't actually parse the
test results. So asking to print the test results as json doesn't make
sense.
We internally create a fake test with one passing subtest, so --json
would actually print out something misleading.
This patch:
* Rewords the flag descriptions so hopefully this is more obvious.
* Also updates --raw_output's description to note the default behavior
is to print out only "KUnit" results (actually any KTAP results)
* also renames and refactors some related logic for clarity (e.g.
test_result => test, it's a kunit_parser.Test object).
Notably, this patch does not make it an error to specify --json and
--raw_output together. This is an edge case, but I know of at least one
wrapper around kunit.py that always sets --json. You'd never be able to
use --raw_output with that wrapper.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 0a7d5c30 | 10-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: tweak error message when no KTAP found
We currently tell people we "couldn't find any KTAP output" with no
indication as to what this might mean.
After this patch, we get:
$ ./tools/t
kunit: tool: tweak error message when no KTAP found
We currently tell people we "couldn't find any KTAP output" with no
indication as to what this might mean.
After this patch, we get:
$ ./tools/testing/kunit/kunit.py parse /dev/null
============================================================
[ERROR] Test: <missing>: Could not find any KTAP output. Did any KUnit tests run?
============================================================
Testing complete. Ran 0 tests: errors: 1
Note: we could try and generate a more verbose message like
> Please check .kunit/test.log to see the raw kernel output.
or the like, but we'd need to know what the build dir was to know where
test.log actually lives.
This patch tries to make a more minimal improvement.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 101e32a0 | 03-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: remove redundant file.close() call in unit test
We're using a `with` block above, so the file object is already closed.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by:
kunit: tool: remove redundant file.close() call in unit test
We're using a `with` block above, so the file object is already closed.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| 05d9d2c3 | 03-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: unit tests all check parser errors, standardize formatting a bit
Let's verify that the parser isn't reporting any errors for valid
inputs.
This change also
* does result.status checkin
kunit: tool: unit tests all check parser errors, standardize formatting a bit
Let's verify that the parser isn't reporting any errors for valid
inputs.
This change also
* does result.status checking on one line
* makes sure we consistently do it outside of the `with` block
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| f473dd94 | 03-Nov-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: make TestCounts a dataclass
Since we're using Python 3.7+, we can use dataclasses to tersen the
code.
It also lets us create pre-populated TestCounts() objects and compare
them in our
kunit: tool: make TestCounts a dataclass
Since we're using Python 3.7+, we can use dataclasses to tersen the
code.
It also lets us create pre-populated TestCounts() objects and compare
them in our unit test. (Before, you could only create empty ones).
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|
| f19dd011 | 28-Oct-2022 |
Daniel Latypov <dlatypov@google.com> |
kunit: tool: print summary of failed tests if a few failed out of a lot
E.g. all the hw_breakpoint tests are failing right now.
So if I run `kunit.py run --altests --arch=x86_64`, then I see
> Testi
kunit: tool: print summary of failed tests if a few failed out of a lot
E.g. all the hw_breakpoint tests are failing right now.
So if I run `kunit.py run --altests --arch=x86_64`, then I see
> Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7
Seeing which 9 tests failed out of the hundreds is annoying.
If my terminal doesn't have scrollback support, I have to resort to
looking at `.kunit/test.log` for the `not ok` lines.
Teach kunit.py to print a summarized list of failures if the # of tests
reachs an arbitrary threshold (>=100 tests).
To try and keep the output from being too long/noisy, this new logic
a) just reports "parent_test failed" if every child test failed
b) won't print anything if there are >10 failures (also arbitrary).
With this patch, we get an extra line of output showing:
> Testing complete. Ran 408 tests: passed: 392, failed: 9, skipped: 7
> Failures: hw_breakpoint
This also works with parameterized tests, e.g. if I add a fake failure
> Failures: kcsan.test_atomic_builtins_missing_barrier.threads=6
Note: we didn't have enough tests for this to be a problem before.
But with commit 980ac3ad0512 ("kunit: tool: rename all_test_uml.config,
use it for --alltests"), --alltests works and thus running >100 tests
will probably become more common.
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
show more ...
|