1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- REST 6- DMTF Redfish 7- Out-of-band IPMI 8- SSH to BMC and Host OS 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed intstructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82For Web UI ( GUI ) Testing setup instructions see 83[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 84 85Installing tox: 86 87``` 88 $ pip install -U tox 89``` 90 91Installing expect: 92 93``` 94 $ sudo apt-get install expect (Ubuntu example) 95``` 96 97## OpenBMC Test Development 98 99These documents contain details on developing OpenBMC test code and debugging. 100 101- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 102- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 103- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 104 To check common code misspellings, syntax and standard checks. 105- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 106 Quick reference for some common curl commands required for legacy REST 107 testing. 108- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 109 Quick reference for some common curl commands required for redfish testing. 110- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 111 UI setup reference. 112- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 113 Submitting changes via Gerrit server 114 115## OpenBMC Test Documentation 116 117- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 118 Reference information for helper tools. 119- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 120 Currently supported BMC and PNOR update. 121- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 122 Steps to create and install CA signed certificate. 123 124## Supported Systems Architecture 125 126OpenBMC test infrastructure is proven capable of running on: 127 128- POWER 129- x86 systems running OpenBMC firmware stack. 130 131## Testing Setup Steps 132 133To verify the installation setup is completed and ready to execute. 134 135- Download the openbmc-test-automation repository: 136 ``` 137 $ git clone https://github.com/openbmc/openbmc-test-automation 138 $ cd openbmc-test-automation 139 ``` 140- Execute basic setup test run: 141 ``` 142 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 143 ``` 144 where xx.xx.xx.xx is the BMC hostname or IP. 145 146## Test Layout 147 148There are several sub-directories within the openbmc-test-automation base which 149contain test suites, tools, templates, etc. These sub-directories are classified 150as follows: 151 152`tests/`: Contains the general test cases for OpenBMC stack functional 153verification. The "tests" subdirectory uses legacy REST and will be deprecated 154at some point and therefore no longer supported. 155 156`extended/`: Contains test cases for boot testing, code update testing using 157legacy REST, etc. The "extended" subdirectory uses legacy REST and will be 158deprecated at some point and therefore no longer supported. 159 160`systest/`: Contains test cases for HTX bootme testing. 161 162`xcat/`: Contains test cases for XCAT automation. 163 164`gui/test/`: Contains test cases for testing web-based interface built on 165AngularJS. 166 167`gui/gui_test/`: Contains test cases for testing web-based user interface built 168on Vue.js. 169 170`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 171platform, OEM). 172 173`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 174configuration testing. 175 176`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 177Serviceability) for an OpenPOWER system. 178 179`openpower/secureboot/`: Contains test cases for secure boot testing on a secure 180boot feature enabled OpenPOWER system only. 181 182`tools/`: Contains various tools. 183 184`templates/`: Contains sample code examples and setup testing. 185 186`test_list/`: Contains the argument files used for skipping test cases (e.g 187"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 188"CT_basic_run", etc.). 189 190## Redfish Test Layout 191 192OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 193standard specification and schema that meets the expectations of end users for 194simple, modern and secure management of scalable platform hardware. 195 196`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 197OpenBMC. 198 199`redfish/extended/`: Contains test cases for combined legacy REST and DMTF 200Redfish-related feature supported on OpenBMC. 201 202Note: Work in progress test development parameter 203`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 204redfish mode only. 205 206## Quickstart 207 208To run openbmc-automation first you need to install the prerequisite Python 209packages which will help to invoke tests through tox (Note that tox version 2102.3.1 or greater is required) or via Robot CLI command. 211 212**Robot Command Line** 213 214- Execute all test suites for `tests/`: 215 216 ``` 217 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 218 ``` 219 220- Execute a test suite: 221 222 ``` 223 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 224 ``` 225 226- Initialize the following test variables which will be used during test 227 execution: 228 229 User can forward declare as environment variables: 230 231 ``` 232 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 233 $ export OPENBMC_USERNAME=<openbmc username> 234 $ export OPENBMC_PASSWORD=<openbmc password> 235 $ export IPMI_COMMAND=<Dbus/External> 236 ``` 237 238 or 239 240 User can input as robot variables as part of the CLI command: 241 242 ``` 243 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 244 -v OPENBMC_USERNAME:<openbmc username> 245 -v OPENBMC_PASSWORD:<openbmc password> 246 ``` 247 248- For QEMU tests, set the following environment variables as well: 249 250 ``` 251 $ export SSH_PORT=<ssh port number> 252 $ export HTTPS_PORT=<https port number> 253 ``` 254 255- Run tests: 256 257 ``` 258 $ tox tests 259 ``` 260 261- How to run individual test: 262 263 One specific test: 264 265 ``` 266 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 267 ``` 268 269 No preset environment variables, default configuration for all supported 270 systems: 271 272 ``` 273 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 274 ``` 275 276 No preset environment variables, one test case from a test suite: 277 278 ``` 279 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 280 ``` 281 282 No preset environment variables, the entire test suite: 283 284 ``` 285 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 286 ``` 287 288 No preset environment variables, the entire test suite excluding test cases 289 using argument file: 290 291 ``` 292 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 293 ``` 294 295 Exclude test list for supported systems: 296 297 ``` 298 Palmetto: test_lists/skip_test_palmetto 299 Witherspoon: test_lists/skip_test_witherspoon 300 ``` 301 302- Run IPMI tests: 303 304 Running only out-of-band IPMI tests: 305 306 ``` 307 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 308 ``` 309 310 Running only inband IPMI tests: 311 312 ``` 313 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 314 ``` 315 316- Run GUI tests: 317 318 By default, GUI runs with Firefox browser and headless mode. Example with 319 chrome browser and header mode: 320 321 ``` 322 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 323 ``` 324 325 Run GUI default CI test bucket: 326 327 ``` 328 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 329 ``` 330 331- Run LDAP tests: 332 333 Before using LDAP test functions, be sure appropriate LDAP user(s) and 334 group(s) have been created on your LDAP server. Note: There are multiple ways 335 to create LDAP users / groups and all depend on your LDAP server. One common 336 way for openldap is ldapadd / ldapmodify refer 337 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 338 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 339 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 340 341 Note: Currently, LDAP test automation for Redfish API is in progress. The 342 format to invoke LDAP test is as follows: 343 344 ``` 345 $ cd redfish/account_service/ 346 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 347 ``` 348 349- How to run CI and CT bucket test: 350 351 Default CI test bucket list: 352 353 ``` 354 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 355 ``` 356 357 Default CI smoke test bucket list: 358 359 ``` 360 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 361 ``` 362 363- Run extended tests: 364 365 For-loop test (default iteration is 10): 366 367 ``` 368 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 369 ``` 370 371 Example using tox testing a test suite for 5 iterations "witherspoon": 372 373 ``` 374 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 375 ``` 376 377- Host CPU architecture 378 379 By default openbmc-test-automation framework assumes that host CPU is based on 380 the POWER architecture. If your host CPU is x86 add 381 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 382 environment variable: 383 384 ``` 385 $ export PLATFORM_ARCH_TYPE=x86 386 ``` 387 388**Jenkins jobs tox commands** 389 390- HW CI tox command: 391 ``` 392 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 393 ``` 394