1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- DMTF Redfish 6- Out-of-band IPMI 7- SSH to BMC and Host OS 8- [Legacy REST](https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable) 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51Installing requirement dependencies: 52 53``` 54 $ pip install -r requirements.txt 55``` 56 57Optional packages but required for `redfish/dmtf_tools/` 58 59``` 60 $ pip install -r requirements_optional.txt 61``` 62 63you'll find this file once your clone openbmc-test-automation repository. 64 65For Web UI ( GUI ) Testing setup instructions see 66[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 67 68Installing tox: 69 70``` 71 $ pip install -U tox 72``` 73 74Installing expect (Ubuntu example): 75 76``` 77 $ sudo apt-get install expect 78``` 79 80## OpenBMC Test Development 81 82These documents contain details on developing OpenBMC test code and debugging. 83 84- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 85- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 86- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 87 To check common code misspellings, syntax and standard checks. 88- [Redfish Coding Guidelines](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_coding_guidelines.md): 89 Refer for Redfish coding guidelines. 90- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 91 Quick reference for some common curl commands required for legacy REST 92 testing. 93- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 94 Quick reference for some common curl commands required for redfish testing. 95- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 96 UI setup reference. 97- [Redfish Request Via mTLS](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_request_via_mTLS.md): 98 Reference for Redfish Request Via mTLS . 99- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 100 Submitting changes via Gerrit server 101 102## OpenBMC Test Documentation 103 104- [OpenBMC Test Architecture](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_architecture.md): 105 Reference for OpenBMC Test Architecture. 106- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 107 Reference information for helper tools. 108- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 109 Currently supported BMC and PNOR update. 110- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 111 Steps to create and install CA signed certificate. 112- [Boot Test](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/boot_test.md): 113 Boot test for OpenBMC. 114 115## Supported Systems Architecture 116 117OpenBMC test infrastructure is proven capable of running on: 118 119- POWER 120- x86 systems running OpenBMC firmware stack. 121 122## Testing Setup Steps 123 124To verify the installation setup is completed and ready to execute. 125 126- Download the openbmc-test-automation repository: 127 128 ``` 129 $ git clone https://github.com/openbmc/openbmc-test-automation 130 $ cd openbmc-test-automation 131 ``` 132 133- Execute basic setup test run: 134 135 ``` 136 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 137 ``` 138 139 where xx.xx.xx.xx is the BMC hostname or IP. 140 141## Test Layout 142 143There are several sub-directories within the openbmc-test-automation base which 144contain test suites, tools, templates, etc. These sub-directories are classified 145as follows: 146 147`docs/`: Contains the documentation related to OpenBMC. 148 149`redfish/`: Contains the general test cases for OpenBMC stack functional 150verification. 151 152`systest/`: Contains test cases for HTX bootme testing. 153 154`xcat/`: Contains test cases for XCAT automation. 155 156`gui/test/`: Contains test cases for testing web-based interface built on 157AngularJS. 158 159`gui/gui_test/`: Contains test cases for testing web-based user interface built 160on Vue.js. 161 162`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 163platform, OEM). 164 165`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 166configuration testing. 167 168`openpower/`: Contains test cases for an OpenPOWER based system. 169 170`tools/`: Contains various tools. 171 172`templates/`: Contains sample code examples and setup testing. 173 174`test_list/`: Contains the argument files used for skipping test cases (e.g 175"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 176"CT_basic_run", etc.). 177 178## Redfish Test Layout 179 180OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 181standard specification and schema that meets the expectations of end users for 182simple, modern and secure management of scalable platform hardware. 183 184`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 185OpenBMC. 186 187`redfish/extended/`: Contains test cases for combined DMTF Redfish-related 188feature supported on OpenBMC. Some of the test will be deprecated. 189 190Note: Work in progress test development parameter 191`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 192redfish mode only. 193 194## Quickstart 195 196To run openbmc-automation first you need to install the prerequisite Python 197packages which will help to invoke tests through tox (Note that tox version 1982.3.1 or greater is required) or via Robot CLI command. 199 200**Robot Command Line** 201 202- Execute all test suites for `tests/`: 203 204 ``` 205 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 206 ``` 207 208- Execute a test suite: 209 210 ``` 211 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 212 ``` 213 214- Initialize the following test variables which will be used during test 215 execution: 216 217 User can forward declare as environment variables: 218 219 ``` 220 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 221 $ export OPENBMC_USERNAME=<openbmc username> 222 $ export OPENBMC_PASSWORD=<openbmc password> 223 $ export IPMI_COMMAND=<Dbus/External> 224 $ export REST_USERNAME=<REST interface username> 225 $ export REST_PASSWORD=<REST interface password> 226 ``` 227 228 or 229 230 User can input as robot variables as part of the CLI command: 231 232 ``` 233 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 234 -v OPENBMC_USERNAME:<openbmc username> 235 -v OPENBMC_PASSWORD:<openbmc password> 236 ``` 237 238- Testing in qemu: 239 240 Set extra environment variables: 241 242 ``` 243 $ export SSH_PORT=<ssh port number> 244 $ export HTTPS_PORT=<https port number> 245 ``` 246 247 Run the QEMU CI test suite (not all tests will pass in qemu): 248 249 ``` 250 $ OPENBMC_HOST=x.x.x.x SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI tests/ 251 ``` 252 253- Run tests: 254 255 ``` 256 $ tox tests 257 ``` 258 259- How to run an individual test: 260 261 ``` 262 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 263 ``` 264 265- No preset environment variables, default configuration for all supported 266 systems: 267 268 ``` 269 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 270 ``` 271 272- No preset environment variables, one test case from a test suite: 273 274 ``` 275 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 276 ``` 277 278- No preset environment variables, the entire test suite: 279 280 ``` 281 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 282 ``` 283 284- No preset environment variables, the entire test suite excluding test cases 285 using argument file: 286 287 ``` 288 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 289 ``` 290 291- Exclude test list for supported systems: 292 293 ``` 294 Palmetto: test_lists/skip_test_palmetto 295 Witherspoon: test_lists/skip_test_witherspoon 296 ``` 297 298 Using the exclude lists (example for Witherspoon) 299 300 ``` 301 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon tests/ 302 ``` 303 304- Run IPMI tests: 305 306 Running only out-of-band IPMI tests: 307 308 ``` 309 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 310 ``` 311 312 Running only inband IPMI tests: 313 314 ``` 315 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 316 ``` 317 318- Run GUI tests: 319 320 By default, GUI runs with Firefox browser and headless mode. Example with 321 Chrome browser and header mode: 322 323 ``` 324 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 325 ``` 326 327 Run GUI default CI test bucket: 328 329 ``` 330 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 331 ``` 332 333- Run LDAP tests: 334 335 Before using LDAP test functions, be sure appropriate LDAP user(s) and 336 group(s) have been created on your LDAP server. Note: There are multiple ways 337 to create LDAP users / groups and all depend on your LDAP server. One common 338 way for openldap is ldapadd / ldapmodify refer 339 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 340 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 341 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 342 343 Note: Currently, LDAP test automation for Redfish API is in progress. The 344 format to invoke LDAP test is as follows: 345 346 ``` 347 $ cd redfish/account_service/ 348 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 349 ``` 350 351- How to run CI and CT bucket test: 352 353 Default CI test bucket list: 354 355 ``` 356 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 357 ``` 358 359 Default CI smoke test bucket list: 360 361 ``` 362 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 363 ``` 364 365- Run extended tests: 366 367 For-loop test (default iteration is 10): 368 369 ``` 370 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 371 ``` 372 373 Example using tox testing a test suite for 5 iterations "witherspoon": 374 375 ``` 376 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 377 ``` 378 379- Host CPU architecture 380 381 By default openbmc-test-automation framework assumes that host CPU is based on 382 the POWER architecture. If your host CPU is x86 add 383 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 384 environment variable: 385 386 ``` 387 $ export PLATFORM_ARCH_TYPE=x86 388 ``` 389 390**Jenkins jobs tox commands** 391 392- HW CI tox command: 393 394 ``` 395 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 396 ``` 397