xref: /openbmc/u-boot/test/py/README.md (revision 7b3dc45e)
1# U-Boot pytest suite
2
3## Introduction
4
5This tool aims to test U-Boot by executing U-Boot shell commands using the
6console interface. A single top-level script exists to execute or attach to the
7U-Boot console, run the entire script of tests against it, and summarize the
8results. Advantages of this approach are:
9
10- Testing is performed in the same way a user or script would interact with
11  U-Boot; there can be no disconnect.
12- There is no need to write or embed test-related code into U-Boot itself.
13  It is asserted that writing test-related code in Python is simpler and more
14  flexible that writing it all in C.
15- It is reasonably simple to interact with U-Boot in this way.
16
17## Requirements
18
19The test suite is implemented using pytest. Interaction with the U-Boot console
20involves executing some binary and interacting with its stdin/stdout. You will
21need to implement various "hook" scripts that are called by the test suite at
22the appropriate time.
23
24On Debian or Debian-like distributions, the following packages are required.
25Similar package names should exist in other distributions.
26
27| Package        | Version tested (Ubuntu 14.04) |
28| -------------- | ----------------------------- |
29| python         | 2.7.5-5ubuntu3                |
30| python-pytest  | 2.5.1-1                       |
31
32The test script supports either:
33
34- Executing a sandbox port of U-Boot on the local machine as a sub-process,
35  and interacting with it over stdin/stdout.
36- Executing an external "hook" scripts to flash a U-Boot binary onto a
37  physical board, attach to the board's console stream, and reset the board.
38  Further details are described later.
39
40### Using `virtualenv` to provide requirements
41
42Older distributions (e.g. Ubuntu 10.04) may not provide all the required
43packages, or may provide versions that are too old to run the test suite. One
44can use the Python `virtualenv` script to locally install more up-to-date
45versions of the required packages without interfering with the OS installation.
46For example:
47
48```bash
49$ cd /path/to/u-boot
50$ sudo apt-get install python python-virtualenv
51$ virtualenv venv
52$ . ./venv/bin/activate
53$ pip install pytest
54```
55
56## Testing sandbox
57
58To run the testsuite on the sandbox port (U-Boot built as a native user-space
59application), simply execute:
60
61```
62./test/py/test.py --bd sandbox --build
63```
64
65The `--bd` option tells the test suite which board type is being tested. This
66lets the test suite know which features the board has, and hence exactly what
67can be tested.
68
69The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
70omit this option and build U-Boot yourself, in whatever way you choose, before
71running the test script.
72
73The test script will attach to U-Boot, execute all valid tests for the board,
74then print a summary of the test process. A complete log of the test session
75will be written to `${build_dir}/test-log.html`. This is best viewed in a web
76browser, but may be read directly as plain text, perhaps with the aid of the
77`html2text` utility.
78
79### Testing under a debugger
80
81If you need to run sandbox under a debugger, you may pass the command-line
82option `--gdbserver COMM`. This causes two things to happens:
83
84- Instead of running U-Boot directly, it will be run under gdbserver, with
85  debug communication via the channel `COMM`. You can attach a debugger to the
86  sandbox process in order to debug it. See `man gdbserver` and the example
87  below for details of valid values for `COMM`.
88- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
89  time to execute commands. This is useful if U-Boot is stopped at a breakpoint
90  during debugging.
91
92A usage example is:
93
94Window 1:
95```shell
96./test/py/test.py --bd sandbox --gdbserver localhost:1234
97```
98
99Window 2:
100```shell
101gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
102```
103
104Alternatively, you could leave off the `-ex` option and type the command
105manually into gdb once it starts.
106
107You can use any debugger you wish, so long as it speaks the gdb remote
108protocol, or any graphical wrapper around gdb.
109
110Some tests deliberately cause the sandbox process to exit, e.g. to test the
111reset command, or sandbox's CTRL-C handling. When this happens, you will need
112to attach the debugger to the new sandbox instance. If these tests are not
113relevant to your debugging session, you can skip them using pytest's -k
114command-line option; see the next section.
115
116## Command-line options
117
118- `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
119  example, `sandbox` or `seaboard`.
120- `--board-identity`, `--id` set the identity of the board to be tested.
121  This allows differentiation between multiple instances of the same type of
122  physical board that are attached to the same host machine. This parameter is
123  not interpreted by the test script in any way, but rather is simply passed
124  to the hook scripts described below, and may be used in any site-specific
125  way deemed necessary.
126- `--build` indicates that the test script should compile U-Boot itself
127  before running the tests. If using this option, make sure that any
128  environment variables required by the build process are already set, such as
129  `$CROSS_COMPILE`.
130- `--build-dir` sets the directory containing the compiled U-Boot binaries.
131  If omitted, this is `${source_dir}/build-${board_type}`.
132- `--result-dir` sets the directory to write results, such as log files,
133  into. If omitted, the build directory is used.
134- `--persistent-data-dir` sets the directory used to store persistent test
135  data. This is test data that may be re-used across test runs, such as file-
136  system images.
137
138`pytest` also implements a number of its own command-line options. Commonly used
139options are mentioned below. Please see `pytest` documentation for complete
140details. Execute `py.test --version` for a brief summary. Note that U-Boot's
141test.py script passes all command-line arguments directly to `pytest` for
142processing.
143
144- `-k` selects which tests to run. The default is to run all known tests. This
145  option takes a single argument which is used to filter test names. Simple
146  logical operators are supported. For example:
147  - `'ums'` runs only tests with "ums" in their name.
148  - ``ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
149    case, "ut_dm" is a parameter to a test rather than the test name. The full
150    test name is e.g. "test_ut[ut_dm_leak]".
151  - `'not reset'` runs everything except tests with "reset" in their name.
152  - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
153  - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
154    their name.
155- `-s` prevents pytest from hiding a test's stdout. This allows you to see
156  U-Boot's console log in real time on pytest's stdout.
157
158## Testing real hardware
159
160The tools and techniques used to interact with real hardware will vary
161radically between different host and target systems, and the whims of the user.
162For this reason, the test suite does not attempt to directly interact with real
163hardware in any way. Rather, it executes a standardized set of "hook" scripts
164via `$PATH`. These scripts implement certain actions on behalf of the test
165suite. This keeps the test suite simple and isolated from system variances
166unrelated to U-Boot features.
167
168### Hook scripts
169
170#### Environment variables
171
172The following environment variables are set when running hook scripts:
173
174- `UBOOT_BOARD_TYPE` the board type being tested.
175- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
176  specified.
177- `UBOOT_SOURCE_DIR` the U-Boot source directory.
178- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
179- `UBOOT_BUILD_DIR` the U-Boot build directory.
180- `UBOOT_RESULT_DIR` the test result directory.
181- `UBOOT_PERSISTENT_DATA_DIR` the test peristent data directory.
182
183#### `u-boot-test-console`
184
185This script provides access to the U-Boot console. The script's stdin/stdout
186should be connected to the board's console. This process should continue to run
187indefinitely, until killed. The test suite will run this script in parallel
188with all other hooks.
189
190This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc.
191
192If you are able to run U-Boot under a hardware simulator such as qemu, then
193you would likely spawn that simulator from this script. However, note that
194`u-boot-test-reset` may be called multiple times per test script run, and must
195cause U-Boot to start execution from scratch each time. Hopefully your
196simulator includes a virtual reset button! If not, you can launch the
197simulator from `u-boot-test-reset` instead, while arranging for this console
198process to always communicate with the current simulator instance.
199
200#### `u-boot-test-flash`
201
202Prior to running the test suite against a board, some arrangement must be made
203so that the board executes the particular U-Boot binary to be tested. Often,
204this involves writing the U-Boot binary to the board's flash ROM. The test
205suite calls this hook script for that purpose.
206
207This script should perform the entire flashing process synchronously; the
208script should only exit once flashing is complete, and a board reset will
209cause the newly flashed U-Boot binary to be executed.
210
211It is conceivable that this script will do nothing. This might be useful in
212the following cases:
213
214- Some other process has already written the desired U-Boot binary into the
215  board's flash prior to running the test suite.
216- The board allows U-Boot to be downloaded directly into RAM, and executed
217  from there. Use of this feature will reduce wear on the board's flash, so
218  may be preferable if available, and if cold boot testing of U-Boot is not
219  required. If this feature is used, the `u-boot-test-reset` script should
220  peform this download, since the board could conceivably be reset multiple
221  times in a single test run.
222
223It is up to the user to determine if those situations exist, and to code this
224hook script appropriately.
225
226This script will typically be implemented by calling out to some SoC- or
227board-specific vendor flashing utility.
228
229#### `u-boot-test-reset`
230
231Whenever the test suite needs to reset the target board, this script is
232executed. This is guaranteed to happen at least once, prior to executing the
233first test function. If any test fails, the test infra-structure will execute
234this script again to restore U-Boot to an operational state before running the
235next test function.
236
237This script will likely be implemented by communicating with some form of
238relay or electronic switch attached to the board's reset signal.
239
240The semantics of this script require that when it is executed, U-Boot will
241start running from scratch. If the U-Boot binary to be tested has been written
242to flash, pulsing the board's reset signal is likely all this script need do.
243However, in some scenarios, this script may perform other actions. For
244example, it may call out to some SoC- or board-specific vendor utility in order
245to download the U-Boot binary directly into RAM and execute it. This would
246avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
247saving wear on the flash chip(s).
248
249#### Examples
250
251https://github.com/swarren/uboot-test-hooks contains some working example hook
252scripts, and may be useful as a reference when implementing hook scripts for
253your platform. These scripts are not considered part of U-Boot itself.
254
255### Board-type-specific configuration
256
257Each board has a different configuration and behaviour. Many of these
258differences can be automatically detected by parsing the `.config` file in the
259build directory. However, some differences can't yet be handled automatically.
260
261For each board, an optional Python module `u_boot_board_${board_type}` may exist
262to provide board-specific information to the test script. Any global value
263defined in these modules is available for use by any test function. The data
264contained in these scripts must be purely derived from U-Boot source code.
265Hence, these configuration files are part of the U-Boot source tree too.
266
267### Execution environment configuration
268
269Each user's hardware setup may enable testing different subsets of the features
270implemented by a particular board's configuration of U-Boot. For example, a
271U-Boot configuration may support USB device mode and USB Mass Storage, but this
272can only be tested if a USB cable is connected between the board and the host
273machine running the test script.
274
275For each board, optional Python modules `u_boot_boardenv_${board_type}` and
276`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
277board-specific and board-identity-specific information to the test script. Any
278global value defined in these modules is available for use by any test
279function. The data contained in these is specific to a particular user's
280hardware configuration. Hence, these configuration files are not part of the
281U-Boot source tree, and should be installed outside of the source tree. Users
282should set `$PYTHONPATH` prior to running the test script to allow these
283modules to be loaded.
284
285### Board module parameter usage
286
287The test scripts rely on the following variables being defined by the board
288module:
289
290- None at present.
291
292### U-Boot `.config` feature usage
293
294The test scripts rely on various U-Boot `.config` features, either directly in
295order to test those features, or indirectly in order to query information from
296the running U-Boot instance in order to test other features.
297
298One example is that testing of the `md` command requires knowledge of a RAM
299address to use for the test. This data is parsed from the output of the
300`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
301
302For a complete list of dependencies, please search the test scripts for
303instances of:
304
305- `buildconfig.get(...`
306- `@pytest.mark.buildconfigspec(...`
307
308### Complete invocation example
309
310Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
311any required environment configuration Python modules into $HOME/ubtest/py,
312then you would likely invoke the test script as follows:
313
314If U-Boot has already been built:
315
316```bash
317PATH=$HOME/ubtest/bin:$PATH \
318    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
319    ./test/py/test.py --bd seaboard
320```
321
322If you want the test script to compile U-Boot for you too, then you likely
323need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
324follow:
325
326```bash
327CROSS_COMPILE=arm-none-eabi- \
328    PATH=$HOME/ubtest/bin:$PATH \
329    PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \
330    ./test/py/test.py --bd seaboard --build
331```
332
333## Writing tests
334
335Please refer to the pytest documentation for details of writing pytest tests.
336Details specific to the U-Boot test suite are described below.
337
338A test fixture named `u_boot_console` should be used by each test function. This
339provides the means to interact with the U-Boot console, and retrieve board and
340environment configuration information.
341
342The function `u_boot_console.run_command()` executes a shell command on the
343U-Boot console, and returns all output from that command. This allows
344validation or interpretation of the command output. This function validates
345that certain strings are not seen on the U-Boot console. These include shell
346error messages and the U-Boot sign-on message (in order to detect unexpected
347board resets). See the source of `u_boot_console_base.py` for a complete list of
348"bad" strings. Some test scenarios are expected to trigger these strings. Use
349`u_boot_console.disable_check()` to temporarily disable checking for specific
350strings. See `test_unknown_cmd.py` for an example.
351
352Board- and board-environment configuration values may be accessed as sub-fields
353of the `u_boot_console.config` object, for example
354`u_boot_console.config.ram_base`.
355
356Build configuration values (from `.config`) may be accessed via the dictionary
357`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable
358names.
359