1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- REST 6- DMTF Redfish 7- Out-of-band IPMI 8- SSH to BMC and Host OS 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed intstructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82For Web UI ( GUI ) Testing setup instructions see 83[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 84 85Installing tox: 86 87``` 88 $ pip install -U tox 89``` 90 91Installing expect (Ubuntu example): 92 93``` 94 $ sudo apt-get install expect 95``` 96 97## OpenBMC Test Development 98 99These documents contain details on developing OpenBMC test code and debugging. 100 101- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 102- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 103- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 104 To check common code misspellings, syntax and standard checks. 105- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 106 Quick reference for some common curl commands required for legacy REST 107 testing. 108- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 109 Quick reference for some common curl commands required for redfish testing. 110- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 111 UI setup reference. 112- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 113 Submitting changes via Gerrit server 114 115## OpenBMC Test Documentation 116 117- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 118 Reference information for helper tools. 119- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 120 Currently supported BMC and PNOR update. 121- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 122 Steps to create and install CA signed certificate. 123 124## Supported Systems Architecture 125 126OpenBMC test infrastructure is proven capable of running on: 127 128- POWER 129- x86 systems running OpenBMC firmware stack. 130 131## Testing Setup Steps 132 133To verify the installation setup is completed and ready to execute. 134 135- Download the openbmc-test-automation repository: 136 137 ``` 138 $ git clone https://github.com/openbmc/openbmc-test-automation 139 $ cd openbmc-test-automation 140 ``` 141 142- Execute basic setup test run: 143 144 ``` 145 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 146 ``` 147 148 where xx.xx.xx.xx is the BMC hostname or IP. 149 150## Test Layout 151 152There are several sub-directories within the openbmc-test-automation base which 153contain test suites, tools, templates, etc. These sub-directories are classified 154as follows: 155 156`tests/`: Contains the general test cases for OpenBMC stack functional 157verification. The "tests" subdirectory uses legacy REST and will be deprecated 158at some point and therefore no longer supported. 159 160`extended/`: Contains test cases for boot testing, code update testing using 161legacy REST, etc. The "extended" subdirectory uses legacy REST and will be 162deprecated at some point and therefore no longer supported. 163 164`systest/`: Contains test cases for HTX bootme testing. 165 166`xcat/`: Contains test cases for XCAT automation. 167 168`gui/test/`: Contains test cases for testing web-based interface built on 169AngularJS. 170 171`gui/gui_test/`: Contains test cases for testing web-based user interface built 172on Vue.js. 173 174`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 175platform, OEM). 176 177`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 178configuration testing. 179 180`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 181Serviceability) for an OpenPOWER system. 182 183`openpower/secureboot/`: Contains test cases for secure boot testing on a secure 184boot feature enabled OpenPOWER system only. 185 186`tools/`: Contains various tools. 187 188`templates/`: Contains sample code examples and setup testing. 189 190`test_list/`: Contains the argument files used for skipping test cases (e.g 191"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 192"CT_basic_run", etc.). 193 194## Redfish Test Layout 195 196OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 197standard specification and schema that meets the expectations of end users for 198simple, modern and secure management of scalable platform hardware. 199 200`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 201OpenBMC. 202 203`redfish/extended/`: Contains test cases for combined legacy REST and DMTF 204Redfish-related feature supported on OpenBMC. 205 206Note: Work in progress test development parameter 207`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 208redfish mode only. 209 210## Quickstart 211 212To run openbmc-automation first you need to install the prerequisite Python 213packages which will help to invoke tests through tox (Note that tox version 2142.3.1 or greater is required) or via Robot CLI command. 215 216**Robot Command Line** 217 218- Execute all test suites for `tests/`: 219 220 ``` 221 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 222 ``` 223 224- Execute a test suite: 225 226 ``` 227 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 228 ``` 229 230- Initialize the following test variables which will be used during test 231 execution: 232 233 User can forward declare as environment variables: 234 235 ``` 236 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 237 $ export OPENBMC_USERNAME=<openbmc username> 238 $ export OPENBMC_PASSWORD=<openbmc password> 239 $ export IPMI_COMMAND=<Dbus/External> 240 ``` 241 242 or 243 244 User can input as robot variables as part of the CLI command: 245 246 ``` 247 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 248 -v OPENBMC_USERNAME:<openbmc username> 249 -v OPENBMC_PASSWORD:<openbmc password> 250 ``` 251 252- Testing in qemu: 253 254 Set extra environment variables: 255 256 ``` 257 $ export SSH_PORT=<ssh port number> 258 $ export HTTPS_PORT=<https port number> 259 ``` 260 261 Run the QEMU CI test suite (not all tests will pass in qemu): 262 263 ``` 264 $ OPENBMC_HOST=x.x.x.x SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI tests/ 265 ``` 266 267- Run tests: 268 269 ``` 270 $ tox tests 271 ``` 272 273- How to run an individual test: 274 275 ``` 276 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 277 ``` 278 279- No preset environment variables, default configuration for all supported 280 systems: 281 282 ``` 283 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 284 ``` 285 286- No preset environment variables, one test case from a test suite: 287 288 ``` 289 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 290 ``` 291 292- No preset environment variables, the entire test suite: 293 294 ``` 295 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 296 ``` 297 298- No preset environment variables, the entire test suite excluding test cases 299 using argument file: 300 301 ``` 302 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 303 ``` 304 305- Exclude test list for supported systems: 306 307 ``` 308 Palmetto: test_lists/skip_test_palmetto 309 Witherspoon: test_lists/skip_test_witherspoon 310 ``` 311 312 Using the exclude lists (example for Witherspoon) 313 314 ``` 315 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon tests/ 316 ``` 317 318- Run IPMI tests: 319 320 Running only out-of-band IPMI tests: 321 322 ``` 323 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 324 ``` 325 326 Running only inband IPMI tests: 327 328 ``` 329 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 330 ``` 331 332- Run GUI tests: 333 334 By default, GUI runs with Firefox browser and headless mode. Example with 335 Chrome browser and header mode: 336 337 ``` 338 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 339 ``` 340 341 Run GUI default CI test bucket: 342 343 ``` 344 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 345 ``` 346 347- Run LDAP tests: 348 349 Before using LDAP test functions, be sure appropriate LDAP user(s) and 350 group(s) have been created on your LDAP server. Note: There are multiple ways 351 to create LDAP users / groups and all depend on your LDAP server. One common 352 way for openldap is ldapadd / ldapmodify refer 353 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 354 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 355 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 356 357 Note: Currently, LDAP test automation for Redfish API is in progress. The 358 format to invoke LDAP test is as follows: 359 360 ``` 361 $ cd redfish/account_service/ 362 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 363 ``` 364 365- How to run CI and CT bucket test: 366 367 Default CI test bucket list: 368 369 ``` 370 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 371 ``` 372 373 Default CI smoke test bucket list: 374 375 ``` 376 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 377 ``` 378 379- Run extended tests: 380 381 For-loop test (default iteration is 10): 382 383 ``` 384 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 385 ``` 386 387 Example using tox testing a test suite for 5 iterations "witherspoon": 388 389 ``` 390 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 391 ``` 392 393- Host CPU architecture 394 395 By default openbmc-test-automation framework assumes that host CPU is based on 396 the POWER architecture. If your host CPU is x86 add 397 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 398 environment variable: 399 400 ``` 401 $ export PLATFORM_ARCH_TYPE=x86 402 ``` 403 404**Jenkins jobs tox commands** 405 406- HW CI tox command: 407 408 ``` 409 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 410 ``` 411