1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- DMTF Redfish 6- Out-of-band IPMI 7- SSH to BMC and Host OS 8- [Legacy REST](https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable) 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed instructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82For Web UI ( GUI ) Testing setup instructions see 83[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 84 85Installing tox: 86 87``` 88 $ pip install -U tox 89``` 90 91Installing expect (Ubuntu example): 92 93``` 94 $ sudo apt-get install expect 95``` 96 97## OpenBMC Test Development 98 99These documents contain details on developing OpenBMC test code and debugging. 100 101- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 102- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 103- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 104 To check common code misspellings, syntax and standard checks. 105- [Redfish Coding Guidelines](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_coding_guidelines.md): 106 Refer for Redfish coding guidelines. 107- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 108 Quick reference for some common curl commands required for legacy REST 109 testing. 110- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 111 Quick reference for some common curl commands required for redfish testing. 112- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 113 UI setup reference. 114- [Redfish Request Via mTLS](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_request_via_mTLS.md): 115 Reference for Redfish Request Via mTLS . 116- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 117 Submitting changes via Gerrit server 118 119## OpenBMC Test Documentation 120 121- [OpenBMC Test Architecture](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_architecture.md): 122 Reference for OpenBMC Test Architecture. 123- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 124 Reference information for helper tools. 125- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 126 Currently supported BMC and PNOR update. 127- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 128 Steps to create and install CA signed certificate. 129- [Boot Test](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/boot_test.md): 130 Boot test for OpenBMC. 131 132## Supported Systems Architecture 133 134OpenBMC test infrastructure is proven capable of running on: 135 136- POWER 137- x86 systems running OpenBMC firmware stack. 138 139## Testing Setup Steps 140 141To verify the installation setup is completed and ready to execute. 142 143- Download the openbmc-test-automation repository: 144 145 ``` 146 $ git clone https://github.com/openbmc/openbmc-test-automation 147 $ cd openbmc-test-automation 148 ``` 149 150- Execute basic setup test run: 151 152 ``` 153 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 154 ``` 155 156 where xx.xx.xx.xx is the BMC hostname or IP. 157 158## Test Layout 159 160There are several sub-directories within the openbmc-test-automation base which 161contain test suites, tools, templates, etc. These sub-directories are classified 162as follows: 163 164`docs/`: Contains the documentation related to OpenBMC. 165 166`redfish/`: Contains the general test cases for OpenBMC stack functional 167verification. 168 169`systest/`: Contains test cases for HTX bootme testing. 170 171`xcat/`: Contains test cases for XCAT automation. 172 173`gui/test/`: Contains test cases for testing web-based interface built on 174AngularJS. 175 176`gui/gui_test/`: Contains test cases for testing web-based user interface built 177on Vue.js. 178 179`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 180platform, OEM). 181 182`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 183configuration testing. 184 185`openpower/`: Contains test cases for an OpenPOWER based system. 186 187`tools/`: Contains various tools. 188 189`templates/`: Contains sample code examples and setup testing. 190 191`test_list/`: Contains the argument files used for skipping test cases (e.g 192"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 193"CT_basic_run", etc.). 194 195## Redfish Test Layout 196 197OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 198standard specification and schema that meets the expectations of end users for 199simple, modern and secure management of scalable platform hardware. 200 201`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 202OpenBMC. 203 204`redfish/extended/`: Contains test cases for combined DMTF Redfish-related 205feature supported on OpenBMC. Some of the test will be deprecated. 206 207Note: Work in progress test development parameter 208`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 209redfish mode only. 210 211## Quickstart 212 213To run openbmc-automation first you need to install the prerequisite Python 214packages which will help to invoke tests through tox (Note that tox version 2152.3.1 or greater is required) or via Robot CLI command. 216 217**Robot Command Line** 218 219- Execute all test suites for `tests/`: 220 221 ``` 222 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 223 ``` 224 225- Execute a test suite: 226 227 ``` 228 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 229 ``` 230 231- Initialize the following test variables which will be used during test 232 execution: 233 234 User can forward declare as environment variables: 235 236 ``` 237 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 238 $ export OPENBMC_USERNAME=<openbmc username> 239 $ export OPENBMC_PASSWORD=<openbmc password> 240 $ export IPMI_COMMAND=<Dbus/External> 241 $ export REST_USERNAME=<REST interface username> 242 $ export REST_PASSWORD=<REST interface password> 243 ``` 244 245 or 246 247 User can input as robot variables as part of the CLI command: 248 249 ``` 250 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 251 -v OPENBMC_USERNAME:<openbmc username> 252 -v OPENBMC_PASSWORD:<openbmc password> 253 ``` 254 255- Testing in qemu: 256 257 Set extra environment variables: 258 259 ``` 260 $ export SSH_PORT=<ssh port number> 261 $ export HTTPS_PORT=<https port number> 262 ``` 263 264 Run the QEMU CI test suite (not all tests will pass in qemu): 265 266 ``` 267 $ OPENBMC_HOST=x.x.x.x SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI tests/ 268 ``` 269 270- Run tests: 271 272 ``` 273 $ tox tests 274 ``` 275 276- How to run an individual test: 277 278 ``` 279 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 280 ``` 281 282- No preset environment variables, default configuration for all supported 283 systems: 284 285 ``` 286 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 287 ``` 288 289- No preset environment variables, one test case from a test suite: 290 291 ``` 292 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 293 ``` 294 295- No preset environment variables, the entire test suite: 296 297 ``` 298 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 299 ``` 300 301- No preset environment variables, the entire test suite excluding test cases 302 using argument file: 303 304 ``` 305 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 306 ``` 307 308- Exclude test list for supported systems: 309 310 ``` 311 Palmetto: test_lists/skip_test_palmetto 312 Witherspoon: test_lists/skip_test_witherspoon 313 ``` 314 315 Using the exclude lists (example for Witherspoon) 316 317 ``` 318 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon tests/ 319 ``` 320 321- Run IPMI tests: 322 323 Running only out-of-band IPMI tests: 324 325 ``` 326 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 327 ``` 328 329 Running only inband IPMI tests: 330 331 ``` 332 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 333 ``` 334 335- Run GUI tests: 336 337 By default, GUI runs with Firefox browser and headless mode. Example with 338 Chrome browser and header mode: 339 340 ``` 341 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 342 ``` 343 344 Run GUI default CI test bucket: 345 346 ``` 347 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 348 ``` 349 350- Run LDAP tests: 351 352 Before using LDAP test functions, be sure appropriate LDAP user(s) and 353 group(s) have been created on your LDAP server. Note: There are multiple ways 354 to create LDAP users / groups and all depend on your LDAP server. One common 355 way for openldap is ldapadd / ldapmodify refer 356 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 357 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 358 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 359 360 Note: Currently, LDAP test automation for Redfish API is in progress. The 361 format to invoke LDAP test is as follows: 362 363 ``` 364 $ cd redfish/account_service/ 365 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 366 ``` 367 368- How to run CI and CT bucket test: 369 370 Default CI test bucket list: 371 372 ``` 373 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 374 ``` 375 376 Default CI smoke test bucket list: 377 378 ``` 379 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 380 ``` 381 382- Run extended tests: 383 384 For-loop test (default iteration is 10): 385 386 ``` 387 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 388 ``` 389 390 Example using tox testing a test suite for 5 iterations "witherspoon": 391 392 ``` 393 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 394 ``` 395 396- Host CPU architecture 397 398 By default openbmc-test-automation framework assumes that host CPU is based on 399 the POWER architecture. If your host CPU is x86 add 400 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 401 environment variable: 402 403 ``` 404 $ export PLATFORM_ARCH_TYPE=x86 405 ``` 406 407**Jenkins jobs tox commands** 408 409- HW CI tox command: 410 411 ``` 412 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 413 ``` 414