test/py/README.md

1 # U-Boot pytest suite
5 This tool aims to test U-Boot by executing U-Boot shell commands using the
6 console interface. A single top-level script exists to execute or attach to the
7 U-Boot console, run the entire script of tests against it, and summarize the
10 - Testing is performed in the same way a user or script would interact with
11   U-Boot; there can be no disconnect.
12 - There is no need to write or embed test-related code into U-Boot itself.
13   It is asserted that writing test-related code in Python is simpler and more
15 - It is reasonably simple to interact with U-Boot in this way.
19 The test suite is implemented using pytest. Interaction with the U-Boot console
24 On Debian or Debian-like distributions, the following packages are required.
29 | -------------- | ----------------------------- |
30 | python         | 2.7.5-5ubuntu3                |
31 | python-pytest  | 2.5.1-1                       |
32 | python-subunit | -                             |
33 | gdisk          | 0.8.8-1ubuntu0.1              |
34 | dfu-util       | 0.5-1                         |
35 | dtc            | 1.4.0+dfsg-1                  |
36 | openssl        | 1.0.1f-1ubuntu2.22            |
40 - Executing a sandbox port of U-Boot on the local machine as a sub-process,
42 - Executing an external "hook" scripts to flash a U-Boot binary onto a
50 can use the Python `virtualenv` script to locally install more up-to-date
55 $ cd /path/to/u-boot
56 $ sudo apt-get install python python-virtualenv
64 To run the testsuite on the sandbox port (U-Boot built as a native user-space
68 ./test/py/test.py --bd sandbox --build
71 The `--bd` option tells the test suite which board type is being tested. This
75 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
76 omit this option and build U-Boot yourself, in whatever way you choose, before
79 The test script will attach to U-Boot, execute all valid tests for the board,
81 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
87 If you need to run sandbox under a debugger, you may pass the command-line
88 option `--gdbserver COMM`. This causes two things to happens:
90 - Instead of running U-Boot directly, it will be run under gdbserver, with
94 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
95   time to execute commands. This is useful if U-Boot is stopped at a breakpoint
102 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
107 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
110 Alternatively, you could leave off the `-ex` option and type the command
117 reset command, or sandbox's CTRL-C handling. When this happens, you will need
119 relevant to your debugging session, you can skip them using pytest's -k
120 command-line option; see the next section.
122 ## Command-line options
124 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
126 - `--board-identity`, `--id` set the identity of the board to be tested.
130   to the hook scripts described below, and may be used in any site-specific
132 - `--build` indicates that the test script should compile U-Boot itself
136 - `--build-dir` sets the directory containing the compiled U-Boot binaries.
137   If omitted, this is `${source_dir}/build-${board_type}`.
138 - `--result-dir` sets the directory to write results, such as log files,
140 - `--persistent-data-dir` sets the directory used to store persistent test
141   data. This is test data that may be re-used across test runs, such as file-
144 `pytest` also implements a number of its own command-line options. Commonly used
146 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
147 test.py script passes all command-line arguments directly to `pytest` for
150 - `-k` selects which tests to run. The default is to run all known tests. This
153   - `'ums'` runs only tests with "ums" in their name.
154   - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
157   - `'not reset'` runs everything except tests with "reset" in their name.
158   - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
159   - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
161 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
162   U-Boot's console log in real time on pytest's stdout.
172 unrelated to U-Boot features.
180 - `UBOOT_BOARD_TYPE` the board type being tested.
181 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
183 - `UBOOT_SOURCE_DIR` the U-Boot source directory.
184 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
185 - `UBOOT_BUILD_DIR` the U-Boot build directory.
186 - `UBOOT_RESULT_DIR` the test result directory.
187 - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
189 #### `u-boot-test-console`
191 This script provides access to the U-Boot console. The script's stdin/stdout
198 If you are able to run U-Boot under a hardware simulator such as qemu, then
200 `u-boot-test-reset` may be called multiple times per test script run, and must
201 cause U-Boot to start execution from scratch each time. Hopefully your
203 simulator from `u-boot-test-reset` instead, while arranging for this console
206 #### `u-boot-test-flash`
209 so that the board executes the particular U-Boot binary to be tested. Often,
210 this involves writing the U-Boot binary to the board's flash ROM. The test
215 cause the newly flashed U-Boot binary to be executed.
220 - Some other process has already written the desired U-Boot binary into the
222 - The board allows U-Boot to be downloaded directly into RAM, and executed
224   may be preferable if available, and if cold boot testing of U-Boot is not
225   required. If this feature is used, the `u-boot-test-reset` script should
232 This script will typically be implemented by calling out to some SoC- or
233 board-specific vendor flashing utility.
235 #### `u-boot-test-reset`
239 first test function. If any test fails, the test infra-structure will execute
240 this script again to restore U-Boot to an operational state before running the
246 The semantics of this script require that when it is executed, U-Boot will
247 start running from scratch. If the U-Boot binary to be tested has been written
250 example, it may call out to some SoC- or board-specific vendor utility in order
251 to download the U-Boot binary directly into RAM and execute it. This would
252 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
257 https://github.com/swarren/uboot-test-hooks contains some working example hook
259 your platform. These scripts are not considered part of U-Boot itself.
261 ### Board-type-specific configuration
268 to provide board-specific information to the test script. Any global value
270 contained in these scripts must be purely derived from U-Boot source code.
271 Hence, these configuration files are part of the U-Boot source tree too.
276 implemented by a particular board's configuration of U-Boot. For example, a
277 U-Boot configuration may support USB device mode and USB Mass Storage, but this
283 board-specific and board-identity-specific information to the test script. Any
287 U-Boot source tree, and should be installed outside of the source tree. Users
296 - None at present.
298 ### U-Boot `.config` feature usage
300 The test scripts rely on various U-Boot `.config` features, either directly in
302 the running U-Boot instance in order to test other features.
311 - `buildconfig.get(...`
312 - `@pytest.mark.buildconfigspec(...`
320 If U-Boot has already been built:
325     ./test/py/test.py --bd seaboard
328 If you want the test script to compile U-Boot for you too, then you likely
333 CROSS_COMPILE=arm-none-eabi- \
336     ./test/py/test.py --bd seaboard --build
342 Details specific to the U-Boot test suite are described below.
345 provides the means to interact with the U-Boot console, and retrieve board and
349 U-Boot console, and returns all output from that command. This allows
351 that certain strings are not seen on the U-Boot console. These include shell
352 error messages and the U-Boot sign-on message (in order to detect unexpected
358 Board- and board-environment configuration values may be accessed as sub-fields