1 # U-Boot pytest suite 2 3 ## Introduction 4 5 This tool aims to test U-Boot by executing U-Boot shell commands using the 6 console interface. A single top-level script exists to execute or attach to the 7 U-Boot console, run the entire script of tests against it, and summarize the 8 results. Advantages of this approach are: 9 10 - Testing is performed in the same way a user or script would interact with 11 U-Boot; there can be no disconnect. 12 - There is no need to write or embed test-related code into U-Boot itself. 13 It is asserted that writing test-related code in Python is simpler and more 14 flexible than writing it all in C. 15 - It is reasonably simple to interact with U-Boot in this way. 16 17 ## Requirements 18 19 The test suite is implemented using pytest. Interaction with the U-Boot console 20 involves executing some binary and interacting with its stdin/stdout. You will 21 need to implement various "hook" scripts that are called by the test suite at 22 the appropriate time. 23 24 On Debian or Debian-like distributions, the following packages are required. 25 Some packages are required to execute any test, and others only for specific 26 tests. Similar package names should exist in other distributions. 27 28 | Package | Version tested (Ubuntu 14.04) | 29 | -------------- | ----------------------------- | 30 | python | 2.7.5-5ubuntu3 | 31 | python-pytest | 2.5.1-1 | 32 | python-subunit | - | 33 | gdisk | 0.8.8-1ubuntu0.1 | 34 | dfu-util | 0.5-1 | 35 | dtc | 1.4.0+dfsg-1 | 36 | openssl | 1.0.1f-1ubuntu2.22 | 37 38 The test script supports either: 39 40 - Executing a sandbox port of U-Boot on the local machine as a sub-process, 41 and interacting with it over stdin/stdout. 42 - Executing an external "hook" scripts to flash a U-Boot binary onto a 43 physical board, attach to the board's console stream, and reset the board. 44 Further details are described later. 45 46 ### Using `virtualenv` to provide requirements 47 48 Older distributions (e.g. Ubuntu 10.04) may not provide all the required 49 packages, or may provide versions that are too old to run the test suite. One 50 can use the Python `virtualenv` script to locally install more up-to-date 51 versions of the required packages without interfering with the OS installation. 52 For example: 53 54 ```bash 55 $ cd /path/to/u-boot 56 $ sudo apt-get install python python-virtualenv 57 $ virtualenv venv 58 $ . ./venv/bin/activate 59 $ pip install pytest 60 ``` 61 62 ## Testing sandbox 63 64 To run the testsuite on the sandbox port (U-Boot built as a native user-space 65 application), simply execute: 66 67 ``` 68 ./test/py/test.py --bd sandbox --build 69 ``` 70 71 The `--bd` option tells the test suite which board type is being tested. This 72 lets the test suite know which features the board has, and hence exactly what 73 can be tested. 74 75 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may 76 omit this option and build U-Boot yourself, in whatever way you choose, before 77 running the test script. 78 79 The test script will attach to U-Boot, execute all valid tests for the board, 80 then print a summary of the test process. A complete log of the test session 81 will be written to `${build_dir}/test-log.html`. This is best viewed in a web 82 browser, but may be read directly as plain text, perhaps with the aid of the 83 `html2text` utility. 84 85 ### Testing under a debugger 86 87 If you need to run sandbox under a debugger, you may pass the command-line 88 option `--gdbserver COMM`. This causes two things to happens: 89 90 - Instead of running U-Boot directly, it will be run under gdbserver, with 91 debug communication via the channel `COMM`. You can attach a debugger to the 92 sandbox process in order to debug it. See `man gdbserver` and the example 93 below for details of valid values for `COMM`. 94 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of 95 time to execute commands. This is useful if U-Boot is stopped at a breakpoint 96 during debugging. 97 98 A usage example is: 99 100 Window 1: 101 ```shell 102 ./test/py/test.py --bd sandbox --gdbserver localhost:1234 103 ``` 104 105 Window 2: 106 ```shell 107 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' 108 ``` 109 110 Alternatively, you could leave off the `-ex` option and type the command 111 manually into gdb once it starts. 112 113 You can use any debugger you wish, so long as it speaks the gdb remote 114 protocol, or any graphical wrapper around gdb. 115 116 Some tests deliberately cause the sandbox process to exit, e.g. to test the 117 reset command, or sandbox's CTRL-C handling. When this happens, you will need 118 to attach the debugger to the new sandbox instance. If these tests are not 119 relevant to your debugging session, you can skip them using pytest's -k 120 command-line option; see the next section. 121 122 ## Command-line options 123 124 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For 125 example, `sandbox` or `seaboard`. 126 - `--board-identity`, `--id` set the identity of the board to be tested. 127 This allows differentiation between multiple instances of the same type of 128 physical board that are attached to the same host machine. This parameter is 129 not interpreted by the test script in any way, but rather is simply passed 130 to the hook scripts described below, and may be used in any site-specific 131 way deemed necessary. 132 - `--build` indicates that the test script should compile U-Boot itself 133 before running the tests. If using this option, make sure that any 134 environment variables required by the build process are already set, such as 135 `$CROSS_COMPILE`. 136 - `--build-dir` sets the directory containing the compiled U-Boot binaries. 137 If omitted, this is `${source_dir}/build-${board_type}`. 138 - `--result-dir` sets the directory to write results, such as log files, 139 into. If omitted, the build directory is used. 140 - `--persistent-data-dir` sets the directory used to store persistent test 141 data. This is test data that may be re-used across test runs, such as file- 142 system images. 143 144 `pytest` also implements a number of its own command-line options. Commonly used 145 options are mentioned below. Please see `pytest` documentation for complete 146 details. Execute `py.test --version` for a brief summary. Note that U-Boot's 147 test.py script passes all command-line arguments directly to `pytest` for 148 processing. 149 150 - `-k` selects which tests to run. The default is to run all known tests. This 151 option takes a single argument which is used to filter test names. Simple 152 logical operators are supported. For example: 153 - `'ums'` runs only tests with "ums" in their name. 154 - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this 155 case, "ut_dm" is a parameter to a test rather than the test name. The full 156 test name is e.g. "test_ut[ut_dm_leak]". 157 - `'not reset'` runs everything except tests with "reset" in their name. 158 - `'ut or hush'` runs only tests with "ut" or "hush" in their name. 159 - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in 160 their name. 161 - `-s` prevents pytest from hiding a test's stdout. This allows you to see 162 U-Boot's console log in real time on pytest's stdout. 163 164 ## Testing real hardware 165 166 The tools and techniques used to interact with real hardware will vary 167 radically between different host and target systems, and the whims of the user. 168 For this reason, the test suite does not attempt to directly interact with real 169 hardware in any way. Rather, it executes a standardized set of "hook" scripts 170 via `$PATH`. These scripts implement certain actions on behalf of the test 171 suite. This keeps the test suite simple and isolated from system variances 172 unrelated to U-Boot features. 173 174 ### Hook scripts 175 176 #### Environment variables 177 178 The following environment variables are set when running hook scripts: 179 180 - `UBOOT_BOARD_TYPE` the board type being tested. 181 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was 182 specified. 183 - `UBOOT_SOURCE_DIR` the U-Boot source directory. 184 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. 185 - `UBOOT_BUILD_DIR` the U-Boot build directory. 186 - `UBOOT_RESULT_DIR` the test result directory. 187 - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. 188 189 #### `u-boot-test-console` 190 191 This script provides access to the U-Boot console. The script's stdin/stdout 192 should be connected to the board's console. This process should continue to run 193 indefinitely, until killed. The test suite will run this script in parallel 194 with all other hooks. 195 196 This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc. 197 198 If you are able to run U-Boot under a hardware simulator such as qemu, then 199 you would likely spawn that simulator from this script. However, note that 200 `u-boot-test-reset` may be called multiple times per test script run, and must 201 cause U-Boot to start execution from scratch each time. Hopefully your 202 simulator includes a virtual reset button! If not, you can launch the 203 simulator from `u-boot-test-reset` instead, while arranging for this console 204 process to always communicate with the current simulator instance. 205 206 #### `u-boot-test-flash` 207 208 Prior to running the test suite against a board, some arrangement must be made 209 so that the board executes the particular U-Boot binary to be tested. Often, 210 this involves writing the U-Boot binary to the board's flash ROM. The test 211 suite calls this hook script for that purpose. 212 213 This script should perform the entire flashing process synchronously; the 214 script should only exit once flashing is complete, and a board reset will 215 cause the newly flashed U-Boot binary to be executed. 216 217 It is conceivable that this script will do nothing. This might be useful in 218 the following cases: 219 220 - Some other process has already written the desired U-Boot binary into the 221 board's flash prior to running the test suite. 222 - The board allows U-Boot to be downloaded directly into RAM, and executed 223 from there. Use of this feature will reduce wear on the board's flash, so 224 may be preferable if available, and if cold boot testing of U-Boot is not 225 required. If this feature is used, the `u-boot-test-reset` script should 226 perform this download, since the board could conceivably be reset multiple 227 times in a single test run. 228 229 It is up to the user to determine if those situations exist, and to code this 230 hook script appropriately. 231 232 This script will typically be implemented by calling out to some SoC- or 233 board-specific vendor flashing utility. 234 235 #### `u-boot-test-reset` 236 237 Whenever the test suite needs to reset the target board, this script is 238 executed. This is guaranteed to happen at least once, prior to executing the 239 first test function. If any test fails, the test infra-structure will execute 240 this script again to restore U-Boot to an operational state before running the 241 next test function. 242 243 This script will likely be implemented by communicating with some form of 244 relay or electronic switch attached to the board's reset signal. 245 246 The semantics of this script require that when it is executed, U-Boot will 247 start running from scratch. If the U-Boot binary to be tested has been written 248 to flash, pulsing the board's reset signal is likely all this script need do. 249 However, in some scenarios, this script may perform other actions. For 250 example, it may call out to some SoC- or board-specific vendor utility in order 251 to download the U-Boot binary directly into RAM and execute it. This would 252 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus 253 saving wear on the flash chip(s). 254 255 #### Examples 256 257 https://github.com/swarren/uboot-test-hooks contains some working example hook 258 scripts, and may be useful as a reference when implementing hook scripts for 259 your platform. These scripts are not considered part of U-Boot itself. 260 261 ### Board-type-specific configuration 262 263 Each board has a different configuration and behaviour. Many of these 264 differences can be automatically detected by parsing the `.config` file in the 265 build directory. However, some differences can't yet be handled automatically. 266 267 For each board, an optional Python module `u_boot_board_${board_type}` may exist 268 to provide board-specific information to the test script. Any global value 269 defined in these modules is available for use by any test function. The data 270 contained in these scripts must be purely derived from U-Boot source code. 271 Hence, these configuration files are part of the U-Boot source tree too. 272 273 ### Execution environment configuration 274 275 Each user's hardware setup may enable testing different subsets of the features 276 implemented by a particular board's configuration of U-Boot. For example, a 277 U-Boot configuration may support USB device mode and USB Mass Storage, but this 278 can only be tested if a USB cable is connected between the board and the host 279 machine running the test script. 280 281 For each board, optional Python modules `u_boot_boardenv_${board_type}` and 282 `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide 283 board-specific and board-identity-specific information to the test script. Any 284 global value defined in these modules is available for use by any test 285 function. The data contained in these is specific to a particular user's 286 hardware configuration. Hence, these configuration files are not part of the 287 U-Boot source tree, and should be installed outside of the source tree. Users 288 should set `$PYTHONPATH` prior to running the test script to allow these 289 modules to be loaded. 290 291 ### Board module parameter usage 292 293 The test scripts rely on the following variables being defined by the board 294 module: 295 296 - None at present. 297 298 ### U-Boot `.config` feature usage 299 300 The test scripts rely on various U-Boot `.config` features, either directly in 301 order to test those features, or indirectly in order to query information from 302 the running U-Boot instance in order to test other features. 303 304 One example is that testing of the `md` command requires knowledge of a RAM 305 address to use for the test. This data is parsed from the output of the 306 `bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. 307 308 For a complete list of dependencies, please search the test scripts for 309 instances of: 310 311 - `buildconfig.get(...` 312 - `@pytest.mark.buildconfigspec(...` 313 314 ### Complete invocation example 315 316 Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and 317 any required environment configuration Python modules into $HOME/ubtest/py, 318 then you would likely invoke the test script as follows: 319 320 If U-Boot has already been built: 321 322 ```bash 323 PATH=$HOME/ubtest/bin:$PATH \ 324 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ 325 ./test/py/test.py --bd seaboard 326 ``` 327 328 If you want the test script to compile U-Boot for you too, then you likely 329 need to set `$CROSS_COMPILE` to allow this, and invoke the test script as 330 follow: 331 332 ```bash 333 CROSS_COMPILE=arm-none-eabi- \ 334 PATH=$HOME/ubtest/bin:$PATH \ 335 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ 336 ./test/py/test.py --bd seaboard --build 337 ``` 338 339 ## Writing tests 340 341 Please refer to the pytest documentation for details of writing pytest tests. 342 Details specific to the U-Boot test suite are described below. 343 344 A test fixture named `u_boot_console` should be used by each test function. This 345 provides the means to interact with the U-Boot console, and retrieve board and 346 environment configuration information. 347 348 The function `u_boot_console.run_command()` executes a shell command on the 349 U-Boot console, and returns all output from that command. This allows 350 validation or interpretation of the command output. This function validates 351 that certain strings are not seen on the U-Boot console. These include shell 352 error messages and the U-Boot sign-on message (in order to detect unexpected 353 board resets). See the source of `u_boot_console_base.py` for a complete list of 354 "bad" strings. Some test scenarios are expected to trigger these strings. Use 355 `u_boot_console.disable_check()` to temporarily disable checking for specific 356 strings. See `test_unknown_cmd.py` for an example. 357 358 Board- and board-environment configuration values may be accessed as sub-fields 359 of the `u_boot_console.config` object, for example 360 `u_boot_console.config.ram_base`. 361 362 Build configuration values (from `.config`) may be accessed via the dictionary 363 `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable 364 names. 365