1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- REST 6- DMTF Redfish 7- Out-of-band IPMI 8- SSH to BMC and Host OS 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed intstructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82Installing tox: 83 84``` 85 $ pip install -U tox 86``` 87 88Installing expect: 89 90``` 91 $ sudo apt-get install expect (Ubuntu example) 92``` 93 94## OpenBMC Test Development 95 96These documents contain details on developing OpenBMC test code and debugging. 97 98- [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information. 99- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 100- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 101 To check common code misspellings, syntax and standard checks. 102- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 103 Quick reference for some common curl commands required for legacy REST 104 testing. 105- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 106 Quick reference for some common curl commands required for redfish testing. 107- [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): 108 Web UI setup reference. 109- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 110 Submitting changes via Gerrit server 111 112## OpenBMC Test Documentation 113 114- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 115 Reference information for helper tools. 116- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 117 Currently supported BMC and PNOR update. 118- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 119 Steps to create and install CA signed certificate. 120 121## Supported Systems Architecture 122 123OpenBMC test infrastructure is proven capable of running on: 124 125- POWER 126- x86 systems running OpenBMC firmware stack. 127 128## Testing Setup Steps 129 130To verify the installation setup is completed and ready to execute. 131 132- Download the openbmc-test-automation repository: 133 ``` 134 $ git clone https://github.com/openbmc/openbmc-test-automation 135 $ cd openbmc-test-automation 136 ``` 137- Execute basic setup test run: 138 ``` 139 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 140 ``` 141 where xx.xx.xx.xx is the BMC hostname or IP. 142 143## Test Layout 144 145There are several sub-directories within the openbmc-test-automation base which 146contain test suites, tools, templates, etc. These sub-directories are classified 147as follows: 148 149`tests/`: Contains the general test cases for OpenBMC stack functional 150verification. The "tests" subdirectory uses legacy REST and will be deprecated 151at some point and therefore no longer supported. 152 153`extended/`: Contains test cases for boot testing, code update testing using 154legacy REST, etc. 155 156`systest/`: Contains test cases for HTX bootme testing. 157 158`xcat/`: Contains test cases for XCAT automation. 159 160`gui/test/`: Contains test cases for testing web-based interface built on 161AngularJS. 162 163`gui/gui_test/`: Contains test cases for testing web-based user interface built 164on Vue.js. 165 166`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 167platform, OEM). 168 169`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 170configuration testing. 171 172`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 173Serviceability) for an OpenPOWER system. 174 175`openpower/secureboot/`: Contains test cases for secure boot testing on a secure 176boot feature enabled OpenPOWER system only. 177 178`tools/`: Contains various tools. 179 180`templates/`: Contains sample code examples and setup testing. 181 182`test_list/`: Contains the argument files used for skipping test cases (e.g 183"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 184"CT_basic_run", etc.). 185 186## Redfish Test Layout 187 188OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 189standard specification and schema that meets the expectations of end users for 190simple, modern and secure management of scalable platform hardware. 191 192`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 193OpenBMC. 194 195`redfish/extended/`: Contains test cases for combined legacy REST and DMTF 196Redfish-related feature supported on OpenBMC. 197 198Note: Work in progress test development parameter 199`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 200redfish mode only. 201 202## Quickstart 203 204To run openbmc-automation first you need to install the prerequisite Python 205packages which will help to invoke tests through tox (Note that tox version 2062.3.1 or greater is required) or via Robot CLI command. 207 208**Robot Command Line** 209 210- Execute all test suites for `tests/`: 211 212 ``` 213 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 214 ``` 215 216- Execute a test suite: 217 218 ``` 219 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 220 ``` 221 222- Initialize the following test variables which will be used during test 223 execution: 224 225 User can forward declare as environment variables: 226 227 ``` 228 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 229 $ export OPENBMC_USERNAME=<openbmc username> 230 $ export OPENBMC_PASSWORD=<openbmc password> 231 $ export IPMI_COMMAND=<Dbus/External> 232 ``` 233 234 or 235 236 User can input as robot variables as part of the CLI command: 237 238 ``` 239 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 240 -v OPENBMC_USERNAME:<openbmc username> 241 -v OPENBMC_PASSWORD:<openbmc password> 242 ``` 243 244- For QEMU tests, set the following environment variables as well: 245 246 ``` 247 $ export SSH_PORT=<ssh port number> 248 $ export HTTPS_PORT=<https port number> 249 ``` 250 251- Run tests: 252 253 ``` 254 $ tox tests 255 ``` 256 257- How to run individual test: 258 259 One specific test: 260 261 ``` 262 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 263 ``` 264 265 No preset environment variables, default configuration for all supported 266 systems: 267 268 ``` 269 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 270 ``` 271 272 No preset environment variables, one test case from a test suite: 273 274 ``` 275 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 276 ``` 277 278 No preset environment variables, the entire test suite: 279 280 ``` 281 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 282 ``` 283 284 No preset environment variables, the entire test suite excluding test cases 285 using argument file: 286 287 ``` 288 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 289 ``` 290 291 Exclude test list for supported systems: 292 293 ``` 294 Palmetto: test_lists/skip_test_palmetto 295 Witherspoon: test_lists/skip_test_witherspoon 296 ``` 297 298- Run IPMI tests: 299 300 Running only out-of-band IPMI tests: 301 302 ``` 303 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 304 ``` 305 306 Running only inband IPMI tests: 307 308 ``` 309 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 310 ``` 311 312- Run GUI tests: 313 314 By default, GUI runs with Firefox browser and headless mode. Example with 315 chrome browser and header mode: 316 317 ``` 318 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 319 ``` 320 321 Run GUI default CI test bucket: 322 323 ``` 324 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 325 ``` 326 327- Run LDAP tests: 328 329 Before using LDAP test functions, be sure appropriate LDAP user(s) and 330 group(s) have been created on your LDAP server. Note: There are multiple ways 331 to create LDAP users / groups and all depend on your LDAP server. One common 332 way for openldap is ldapadd / ldapmodify refer 333 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 334 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 335 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 336 337 Note: Currently, LDAP test automation for Redfish API is in progress. The 338 format to invoke LDAP test is as follows: 339 340 ``` 341 $ cd redfish/account_service/ 342 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 343 ``` 344 345- How to run CI and CT bucket test: 346 347 Default CI test bucket list: 348 349 ``` 350 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 351 ``` 352 353 Default CI smoke test bucket list: 354 355 ``` 356 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 357 ``` 358 359- Run extended tests: 360 361 For-loop test (default iteration is 10): 362 363 ``` 364 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 365 ``` 366 367 Example using tox testing a test suite for 5 iterations "witherspoon": 368 369 ``` 370 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 371 ``` 372 373- Host CPU architecture 374 375 By default openbmc-test-automation framework assumes that host CPU is based on 376 the POWER architecture. If your host CPU is x86 add 377 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 378 environment variable: 379 380 ``` 381 $ export PLATFORM_ARCH_TYPE=x86 382 ``` 383 384**Jenkins jobs tox commands** 385 386- HW CI tox command: 387 ``` 388 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 389 ``` 390