1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- REST 6- DMTF Redfish 7- Out-of-band IPMI 8- SSH to BMC and Host OS 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed intstructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82For Web UI ( GUI ) Testing setup instructions see 83[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 84 85Installing tox: 86 87``` 88 $ pip install -U tox 89``` 90 91Installing expect: 92 93``` 94 $ sudo apt-get install expect (Ubuntu example) 95``` 96 97## OpenBMC Test Development 98 99These documents contain details on developing OpenBMC test code and debugging. 100 101- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 102- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 103- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 104 To check common code misspellings, syntax and standard checks. 105- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 106 Quick reference for some common curl commands required for legacy REST 107 testing. 108- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 109 Quick reference for some common curl commands required for redfish testing. 110- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 111 UI setup reference. 112- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 113 Submitting changes via Gerrit server 114 115## OpenBMC Test Documentation 116 117- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 118 Reference information for helper tools. 119- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 120 Currently supported BMC and PNOR update. 121- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 122 Steps to create and install CA signed certificate. 123 124## Supported Systems Architecture 125 126OpenBMC test infrastructure is proven capable of running on: 127 128- POWER 129- x86 systems running OpenBMC firmware stack. 130 131## Testing Setup Steps 132 133To verify the installation setup is completed and ready to execute. 134 135- Download the openbmc-test-automation repository: 136 ``` 137 $ git clone https://github.com/openbmc/openbmc-test-automation 138 $ cd openbmc-test-automation 139 ``` 140- Execute basic setup test run: 141 ``` 142 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 143 ``` 144 where xx.xx.xx.xx is the BMC hostname or IP. 145 146## Test Layout 147 148There are several sub-directories within the openbmc-test-automation base which 149contain test suites, tools, templates, etc. These sub-directories are classified 150as follows: 151 152`tests/`: Contains the general test cases for OpenBMC stack functional 153verification. The "tests" subdirectory uses legacy REST and will be deprecated 154at some point and therefore no longer supported. 155 156`extended/`: Contains test cases for boot testing, code update testing using 157legacy REST, etc. 158 159`systest/`: Contains test cases for HTX bootme testing. 160 161`xcat/`: Contains test cases for XCAT automation. 162 163`gui/test/`: Contains test cases for testing web-based interface built on 164AngularJS. 165 166`gui/gui_test/`: Contains test cases for testing web-based user interface built 167on Vue.js. 168 169`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 170platform, OEM). 171 172`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 173configuration testing. 174 175`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 176Serviceability) for an OpenPOWER system. 177 178`openpower/secureboot/`: Contains test cases for secure boot testing on a secure 179boot feature enabled OpenPOWER system only. 180 181`tools/`: Contains various tools. 182 183`templates/`: Contains sample code examples and setup testing. 184 185`test_list/`: Contains the argument files used for skipping test cases (e.g 186"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 187"CT_basic_run", etc.). 188 189## Redfish Test Layout 190 191OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 192standard specification and schema that meets the expectations of end users for 193simple, modern and secure management of scalable platform hardware. 194 195`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 196OpenBMC. 197 198`redfish/extended/`: Contains test cases for combined legacy REST and DMTF 199Redfish-related feature supported on OpenBMC. 200 201Note: Work in progress test development parameter 202`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 203redfish mode only. 204 205## Quickstart 206 207To run openbmc-automation first you need to install the prerequisite Python 208packages which will help to invoke tests through tox (Note that tox version 2092.3.1 or greater is required) or via Robot CLI command. 210 211**Robot Command Line** 212 213- Execute all test suites for `tests/`: 214 215 ``` 216 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 217 ``` 218 219- Execute a test suite: 220 221 ``` 222 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 223 ``` 224 225- Initialize the following test variables which will be used during test 226 execution: 227 228 User can forward declare as environment variables: 229 230 ``` 231 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 232 $ export OPENBMC_USERNAME=<openbmc username> 233 $ export OPENBMC_PASSWORD=<openbmc password> 234 $ export IPMI_COMMAND=<Dbus/External> 235 ``` 236 237 or 238 239 User can input as robot variables as part of the CLI command: 240 241 ``` 242 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 243 -v OPENBMC_USERNAME:<openbmc username> 244 -v OPENBMC_PASSWORD:<openbmc password> 245 ``` 246 247- For QEMU tests, set the following environment variables as well: 248 249 ``` 250 $ export SSH_PORT=<ssh port number> 251 $ export HTTPS_PORT=<https port number> 252 ``` 253 254- Run tests: 255 256 ``` 257 $ tox tests 258 ``` 259 260- How to run individual test: 261 262 One specific test: 263 264 ``` 265 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 266 ``` 267 268 No preset environment variables, default configuration for all supported 269 systems: 270 271 ``` 272 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 273 ``` 274 275 No preset environment variables, one test case from a test suite: 276 277 ``` 278 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 279 ``` 280 281 No preset environment variables, the entire test suite: 282 283 ``` 284 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 285 ``` 286 287 No preset environment variables, the entire test suite excluding test cases 288 using argument file: 289 290 ``` 291 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 292 ``` 293 294 Exclude test list for supported systems: 295 296 ``` 297 Palmetto: test_lists/skip_test_palmetto 298 Witherspoon: test_lists/skip_test_witherspoon 299 ``` 300 301- Run IPMI tests: 302 303 Running only out-of-band IPMI tests: 304 305 ``` 306 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 307 ``` 308 309 Running only inband IPMI tests: 310 311 ``` 312 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 313 ``` 314 315- Run GUI tests: 316 317 By default, GUI runs with Firefox browser and headless mode. Example with 318 chrome browser and header mode: 319 320 ``` 321 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 322 ``` 323 324 Run GUI default CI test bucket: 325 326 ``` 327 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 328 ``` 329 330- Run LDAP tests: 331 332 Before using LDAP test functions, be sure appropriate LDAP user(s) and 333 group(s) have been created on your LDAP server. Note: There are multiple ways 334 to create LDAP users / groups and all depend on your LDAP server. One common 335 way for openldap is ldapadd / ldapmodify refer 336 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 337 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 338 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 339 340 Note: Currently, LDAP test automation for Redfish API is in progress. The 341 format to invoke LDAP test is as follows: 342 343 ``` 344 $ cd redfish/account_service/ 345 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 346 ``` 347 348- How to run CI and CT bucket test: 349 350 Default CI test bucket list: 351 352 ``` 353 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 354 ``` 355 356 Default CI smoke test bucket list: 357 358 ``` 359 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 360 ``` 361 362- Run extended tests: 363 364 For-loop test (default iteration is 10): 365 366 ``` 367 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 368 ``` 369 370 Example using tox testing a test suite for 5 iterations "witherspoon": 371 372 ``` 373 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 374 ``` 375 376- Host CPU architecture 377 378 By default openbmc-test-automation framework assumes that host CPU is based on 379 the POWER architecture. If your host CPU is x86 add 380 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 381 environment variable: 382 383 ``` 384 $ export PLATFORM_ARCH_TYPE=x86 385 ``` 386 387**Jenkins jobs tox commands** 388 389- HW CI tox command: 390 ``` 391 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 392 ``` 393