test/py/README.md

1 # U-Boot pytest suite
5 This tool aims to test U-Boot by executing U-Boot shell commands using the
6 console interface. A single top-level script exists to execute or attach to the
7 U-Boot console, run the entire script of tests against it, and summarize the
10 - Testing is performed in the same way a user or script would interact with
11   U-Boot; there can be no disconnect.
12 - There is no need to write or embed test-related code into U-Boot itself.
13   It is asserted that writing test-related code in Python is simpler and more
15 - It is reasonably simple to interact with U-Boot in this way.
19 The test suite is implemented using pytest. Interaction with the U-Boot console
24 On Debian or Debian-like distributions, the following packages are required.
25 Some packages are required to execute any test, and others only for specific
29 | -------------- | ----------------------------- |
30 | python         | 2.7.5-5ubuntu3                |
31 | python-pytest  | 2.5.1-1                       |
32 | python-subunit | -                             |
33 | gdisk          | 0.8.8-1ubuntu0.1              |
34 | dfu-util       | 0.5-1                         |
35 | dtc            | 1.4.0+dfsg-1                  |
36 | openssl        | 1.0.1f-1ubuntu2.22            |
40 - Executing a sandbox port of U-Boot on the local machine as a sub-process,
42 - Executing an external "hook" scripts to flash a U-Boot binary onto a
43   physical board, attach to the board's console stream, and reset the board.
50 can use the Python `virtualenv` script to locally install more up-to-date
55 $ cd /path/to/u-boot
56 $ sudo apt-get install python python-virtualenv
64 To run the testsuite on the sandbox port (U-Boot built as a native user-space
68 ./test/py/test.py --bd sandbox --build
71 The `--bd` option tells the test suite which board type is being tested. This
72 lets the test suite know which features the board has, and hence exactly what
75 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
76 omit this option and build U-Boot yourself, in whatever way you choose, before
79 The test script will attach to U-Boot, execute all valid tests for the board,
81 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
87 If you need to run sandbox under a debugger, you may pass the command-line
88 option `--gdbserver COMM`. This causes two things to happens:
90 - Instead of running U-Boot directly, it will be run under gdbserver, with
94 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
95   time to execute commands. This is useful if U-Boot is stopped at a breakpoint
102 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
107 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
110 Alternatively, you could leave off the `-ex` option and type the command
117 reset command, or sandbox's CTRL-C handling. When this happens, you will need
119 relevant to your debugging session, you can skip them using pytest's -k
120 command-line option; see the next section.
122 ## Command-line options
124 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
126 - `--board-identity`, `--id` set the identity of the board to be tested.
128   physical board that are attached to the same host machine. This parameter is
130   to the hook scripts described below, and may be used in any site-specific
132 - `--build` indicates that the test script should compile U-Boot itself
136 - `--build-dir` sets the directory containing the compiled U-Boot binaries.
137   If omitted, this is `${source_dir}/build-${board_type}`.
138 - `--result-dir` sets the directory to write results, such as log files,
140 - `--persistent-data-dir` sets the directory used to store persistent test
141   data. This is test data that may be re-used across test runs, such as file-
144 `pytest` also implements a number of its own command-line options. Commonly used
146 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
147 test.py script passes all command-line arguments directly to `pytest` for
150 - `-k` selects which tests to run. The default is to run all known tests. This
153   - `'ums'` runs only tests with "ums" in their name.
154   - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
157   - `'not reset'` runs everything except tests with "reset" in their name.
158   - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
159   - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
161 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
162   U-Boot's console log in real time on pytest's stdout.
172 unrelated to U-Boot features.
180 - `UBOOT_BOARD_TYPE` the board type being tested.
181 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
183 - `UBOOT_SOURCE_DIR` the U-Boot source directory.
184 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
185 - `UBOOT_BUILD_DIR` the U-Boot build directory.
186 - `UBOOT_RESULT_DIR` the test result directory.
187 - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
189 #### `u-boot-test-console`
191 This script provides access to the U-Boot console. The script's stdin/stdout
192 should be connected to the board's console. This process should continue to run
198 If you are able to run U-Boot under a hardware simulator such as qemu, then
200 `u-boot-test-reset` may be called multiple times per test script run, and must
201 cause U-Boot to start execution from scratch each time. Hopefully your
203 simulator from `u-boot-test-reset` instead, while arranging for this console
206 #### `u-boot-test-flash`
208 Prior to running the test suite against a board, some arrangement must be made
209 so that the board executes the particular U-Boot binary to be tested. Often,
210 this involves writing the U-Boot binary to the board's flash ROM. The test
214 script should only exit once flashing is complete, and a board reset will
215 cause the newly flashed U-Boot binary to be executed.
220 - Some other process has already written the desired U-Boot binary into the
221   board's flash prior to running the test suite.
222 - The board allows U-Boot to be downloaded directly into RAM, and executed
223   from there. Use of this feature will reduce wear on the board's flash, so
224   may be preferable if available, and if cold boot testing of U-Boot is not
225   required. If this feature is used, the `u-boot-test-reset` script should
226   perform this download, since the board could conceivably be reset multiple
232 This script will typically be implemented by calling out to some SoC- or
233 board-specific vendor flashing utility.
235 #### `u-boot-test-reset`
237 Whenever the test suite needs to reset the target board, this script is
239 first test function. If any test fails, the test infra-structure will execute
240 this script again to restore U-Boot to an operational state before running the
244 relay or electronic switch attached to the board's reset signal.
246 The semantics of this script require that when it is executed, U-Boot will
247 start running from scratch. If the U-Boot binary to be tested has been written
248 to flash, pulsing the board's reset signal is likely all this script need do.
250 example, it may call out to some SoC- or board-specific vendor utility in order
251 to download the U-Boot binary directly into RAM and execute it. This would
252 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
257 https://github.com/swarren/uboot-test-hooks contains some working example hook
259 your platform. These scripts are not considered part of U-Boot itself.
261 ### Board-type-specific configuration
263 Each board has a different configuration and behaviour. Many of these
267 For each board, an optional Python module `u_boot_board_${board_type}` may exist
268 to provide board-specific information to the test script. Any global value
270 contained in these scripts must be purely derived from U-Boot source code.
271 Hence, these configuration files are part of the U-Boot source tree too.
276 implemented by a particular board's configuration of U-Boot. For example, a
277 U-Boot configuration may support USB device mode and USB Mass Storage, but this
278 can only be tested if a USB cable is connected between the board and the host
281 For each board, optional Python modules `u_boot_boardenv_${board_type}` and
283 board-specific and board-identity-specific information to the test script. Any
285 function. The data contained in these is specific to a particular user's
287 U-Boot source tree, and should be installed outside of the source tree. Users
291 ### Board module parameter usage
293 The test scripts rely on the following variables being defined by the board
296 - None at present.
298 ### U-Boot `.config` feature usage
300 The test scripts rely on various U-Boot `.config` features, either directly in
302 the running U-Boot instance in order to test other features.
311 - `buildconfig.get(...`
312 - `@pytest.mark.buildconfigspec(...`
320 If U-Boot has already been built:
325     ./test/py/test.py --bd seaboard
328 If you want the test script to compile U-Boot for you too, then you likely
333 CROSS_COMPILE=arm-none-eabi- \
336     ./test/py/test.py --bd seaboard --build
342 Details specific to the U-Boot test suite are described below.
345 provides the means to interact with the U-Boot console, and retrieve board and
349 U-Boot console, and returns all output from that command. This allows
351 that certain strings are not seen on the U-Boot console. These include shell
352 error messages and the U-Boot sign-on message (in order to detect unexpected
353 board resets). See the source of `u_boot_console_base.py` for a complete list of
355 `u_boot_console.disable_check()` to temporarily disable checking for specific
358 Board- and board-environment configuration values may be accessed as sub-fields