1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- DMTF Redfish 6- Out-of-band IPMI 7- SSH to BMC and Host OS 8- [Legacy REST](https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable) 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51Installing requirement dependencies: 52 53``` 54 $ pip install -r requirements.txt 55``` 56 57Optional packages but required for `redfish/dmtf_tools/` 58 59``` 60 $ pip install -r requirements_optional.txt 61``` 62 63you'll find this file once your clone openbmc-test-automation repository. 64 65For Web UI ( GUI ) Testing setup, please the follow instructions mentioned in 66the OpenBMC GUI Test Setup Guide. Note: GUI test cases under `gui/` will not 67work without the above setup completed in your environment. 68[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 69 70Installing tox: 71 72``` 73 $ pip install -U tox 74``` 75 76Installing expect (Ubuntu example): 77 78``` 79 $ sudo apt-get install expect 80``` 81 82## OpenBMC Test Development 83 84These documents contain details on developing OpenBMC test code and debugging. 85 86- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 87- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 88- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 89 To check common code misspellings, syntax and standard checks. 90- [Redfish Coding Guidelines](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_coding_guidelines.md): 91 Refer for Redfish coding guidelines. 92- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 93 Quick reference for some common curl commands required for legacy REST 94 testing. 95- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 96 Quick reference for some common curl commands required for redfish testing. 97- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 98 UI setup reference. 99- [Redfish Request Via mTLS](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_request_via_mTLS.md): 100 Reference for Redfish Request Via mTLS . 101- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 102 Submitting changes via Gerrit server 103 104## OpenBMC Test Documentation 105 106- [OpenBMC Test Architecture](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_architecture.md): 107 Reference for OpenBMC Test Architecture. 108- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 109 Reference information for helper tools. 110- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 111 Currently supported BMC and PNOR update. 112- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 113 Steps to create and install CA signed certificate. 114- [Boot Test](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/boot_test.md): 115 Boot test for OpenBMC. 116 117## Supported Systems Architecture 118 119OpenBMC test infrastructure is proven capable of running on: 120 121- POWER 122- x86 systems running OpenBMC firmware stack. 123 124## Testing Setup Steps 125 126To verify the installation setup is completed and ready to execute. 127 128- Download the openbmc-test-automation repository: 129 130 ``` 131 $ git clone https://github.com/openbmc/openbmc-test-automation 132 $ cd openbmc-test-automation 133 ``` 134 135- Execute basic setup test run: 136 137 ``` 138 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 139 ``` 140 141 where xx.xx.xx.xx is the BMC hostname or IP. 142 143## Test Layout 144 145There are several sub-directories within the openbmc-test-automation base which 146contain test suites, tools, templates, etc. These sub-directories are classified 147as follows: 148 149`docs/`: Contains the documentation related to OpenBMC. 150 151`redfish/`: Contains the general test cases for OpenBMC stack functional 152verification. 153 154`systest/`: Contains test cases for HTX bootme testing. 155 156`xcat/`: Contains test cases for XCAT automation. 157 158`gui/test/`: Contains test cases for testing web-based interface built on 159AngularJS. 160 161`gui/gui_test/`: Contains test cases for testing web-based user interface built 162on Vue.js. 163 164`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 165platform, OEM). 166 167`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 168configuration testing. 169 170`openpower/`: Contains test cases for an OpenPOWER based system. 171 172`tools/`: Contains various tools. 173 174`templates/`: Contains sample code examples and setup testing. 175 176`test_list/`: Contains the argument files used for skipping test cases (e.g 177"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 178"CT_basic_run", etc.). 179 180## Redfish Test Layout 181 182OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 183standard specification and schema that meets the expectations of end users for 184simple, modern and secure management of scalable platform hardware. 185 186`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 187OpenBMC. 188 189`redfish/extended/`: Contains test cases for combined DMTF Redfish-related 190feature supported on OpenBMC. Some of the test will be deprecated. 191 192Note: Work in progress test development parameter 193`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 194redfish mode only. 195 196## Quickstart 197 198To run openbmc-automation first you need to install the prerequisite Python 199packages which will help to invoke tests through tox (Note that tox version 2002.3.1 or greater is required) or via Robot CLI command. 201 202**Robot Command Line** 203 204- Execute all test suites for `tests/`: 205 206 ``` 207 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 208 ``` 209 210- Execute a test suite: 211 212 ``` 213 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 214 ``` 215 216- Initialize the following test variables which will be used during test 217 execution: 218 219 User can forward declare as environment variables: 220 221 ``` 222 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 223 $ export OPENBMC_USERNAME=<openbmc username> 224 $ export OPENBMC_PASSWORD=<openbmc password> 225 $ export IPMI_COMMAND=<Dbus/External> 226 $ export REST_USERNAME=<REST interface username> 227 $ export REST_PASSWORD=<REST interface password> 228 ``` 229 230 or 231 232 User can input as robot variables as part of the CLI command: 233 234 ``` 235 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 236 -v OPENBMC_USERNAME:<openbmc username> 237 -v OPENBMC_PASSWORD:<openbmc password> 238 ``` 239 240- Testing in qemu: 241 242 Set extra environment variables: 243 244 ``` 245 $ export SSH_PORT=<ssh port number> 246 $ export HTTPS_PORT=<https port number> 247 ``` 248 249 Run the QEMU CI test suite (not all tests will pass in qemu): 250 251 ``` 252 $ OPENBMC_HOST=x.x.x.x SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI tests/ 253 ``` 254 255- Run tests: 256 257 ``` 258 $ tox tests 259 ``` 260 261- How to run an individual test: 262 263 ``` 264 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 265 ``` 266 267- No preset environment variables, default configuration for all supported 268 systems: 269 270 ``` 271 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 272 ``` 273 274- No preset environment variables, one test case from a test suite: 275 276 ``` 277 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 278 ``` 279 280- No preset environment variables, the entire test suite: 281 282 ``` 283 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 284 ``` 285 286- No preset environment variables, the entire test suite excluding test cases 287 using argument file: 288 289 ``` 290 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 291 ``` 292 293- Exclude test list for supported systems: 294 295 ``` 296 Palmetto: test_lists/skip_test_palmetto 297 Witherspoon: test_lists/skip_test_witherspoon 298 ``` 299 300 Using the exclude lists (example for Witherspoon) 301 302 ``` 303 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon tests/ 304 ``` 305 306- Run IPMI tests: 307 308 Running only out-of-band IPMI tests: 309 310 ``` 311 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 312 ``` 313 314 Running only inband IPMI tests: 315 316 ``` 317 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 318 ``` 319 320- Run GUI tests: 321 322 By default, GUI runs with Firefox browser and headless mode. Example with 323 Chrome browser and header mode: 324 325 ``` 326 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 327 ``` 328 329 Run GUI default CI test bucket: 330 331 ``` 332 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 333 ``` 334 335- Run LDAP tests: 336 337 Before using LDAP test functions, be sure appropriate LDAP user(s) and 338 group(s) have been created on your LDAP server. Note: There are multiple ways 339 to create LDAP users / groups and all depend on your LDAP server. One common 340 way for openldap is ldapadd / ldapmodify refer 341 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 342 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 343 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 344 345 Note: Currently, LDAP test automation for Redfish API is in progress. The 346 format to invoke LDAP test is as follows: 347 348 ``` 349 $ cd redfish/account_service/ 350 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 351 ``` 352 353- How to run CI and CT bucket test: 354 355 Default CI test bucket list: 356 357 ``` 358 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 359 ``` 360 361 Default CI smoke test bucket list: 362 363 ``` 364 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 365 ``` 366 367- Run extended tests: 368 369 For-loop test (default iteration is 10): 370 371 ``` 372 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 373 ``` 374 375 Example using tox testing a test suite for 5 iterations "witherspoon": 376 377 ``` 378 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 379 ``` 380 381- Host CPU architecture 382 383 By default openbmc-test-automation framework assumes that host CPU is based on 384 the POWER architecture. If your host CPU is x86 add 385 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 386 environment variable: 387 388 ``` 389 $ export PLATFORM_ARCH_TYPE=x86 390 ``` 391 392**Jenkins jobs tox commands** 393 394- HW CI tox command: 395 396 ``` 397 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 398 ``` 399