1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- DMTF Redfish 6- Out-of-band IPMI 7- SSH to BMC and Host OS 8- [Legacy REST](https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable) 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51REST base packages: 52 53``` 54 $ pip install -U requests 55 $ pip install -U robotframework-requests 56 $ pip install -U robotframework-httplibrary 57``` 58 59Python redfish library packages: For more detailed intstructions see 60[python-redfish-library](https://github.com/DMTF/python-redfish-library) 61 62``` 63 $ pip install redfish 64``` 65 66SSH and SCP base packages: For more detailed installation instructions see 67[robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 68 69``` 70 $ pip install robotframework-sshlibrary 71 $ pip install robotframework-scplibrary 72``` 73 74Installing requirement dependencies: 75 76``` 77 $ pip install -r requirements.txt 78``` 79 80you'll find this file once your clone openbmc-test-automation repository. 81 82For Web UI ( GUI ) Testing setup instructions see 83[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 84 85Installing tox: 86 87``` 88 $ pip install -U tox 89``` 90 91Installing expect (Ubuntu example): 92 93``` 94 $ sudo apt-get install expect 95``` 96 97## OpenBMC Test Development 98 99These documents contain details on developing OpenBMC test code and debugging. 100 101- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 102- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 103- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 104 To check common code misspellings, syntax and standard checks. 105- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 106 Quick reference for some common curl commands required for legacy REST 107 testing. 108- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 109 Quick reference for some common curl commands required for redfish testing. 110- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 111 UI setup reference. 112- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 113 Submitting changes via Gerrit server 114 115## OpenBMC Test Documentation 116 117- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 118 Reference information for helper tools. 119- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 120 Currently supported BMC and PNOR update. 121- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 122 Steps to create and install CA signed certificate. 123 124## Supported Systems Architecture 125 126OpenBMC test infrastructure is proven capable of running on: 127 128- POWER 129- x86 systems running OpenBMC firmware stack. 130 131## Testing Setup Steps 132 133To verify the installation setup is completed and ready to execute. 134 135- Download the openbmc-test-automation repository: 136 137 ``` 138 $ git clone https://github.com/openbmc/openbmc-test-automation 139 $ cd openbmc-test-automation 140 ``` 141 142- Execute basic setup test run: 143 144 ``` 145 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 146 ``` 147 148 where xx.xx.xx.xx is the BMC hostname or IP. 149 150## Test Layout 151 152There are several sub-directories within the openbmc-test-automation base which 153contain test suites, tools, templates, etc. These sub-directories are classified 154as follows: 155 156`redfish/`: Contains the general test cases for OpenBMC stack functional 157verification. 158 159`systest/`: Contains test cases for HTX bootme testing. 160 161`xcat/`: Contains test cases for XCAT automation. 162 163`gui/test/`: Contains test cases for testing web-based interface built on 164AngularJS. 165 166`gui/gui_test/`: Contains test cases for testing web-based user interface built 167on Vue.js. 168 169`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 170platform, OEM). 171 172`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 173configuration testing. 174 175`openpower/`: Contains test cases for an OpenPOWER based system. 176 177`tools/`: Contains various tools. 178 179`templates/`: Contains sample code examples and setup testing. 180 181`test_list/`: Contains the argument files used for skipping test cases (e.g 182"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 183"CT_basic_run", etc.). 184 185## Redfish Test Layout 186 187OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 188standard specification and schema that meets the expectations of end users for 189simple, modern and secure management of scalable platform hardware. 190 191`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 192OpenBMC. 193 194`redfish/extended/`: Contains test cases for combined DMTF Redfish-related 195feature supported on OpenBMC. Some of the test will be deprecated. 196 197Note: Work in progress test development parameter 198`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 199redfish mode only. 200 201## Quickstart 202 203To run openbmc-automation first you need to install the prerequisite Python 204packages which will help to invoke tests through tox (Note that tox version 2052.3.1 or greater is required) or via Robot CLI command. 206 207**Robot Command Line** 208 209- Execute all test suites for `tests/`: 210 211 ``` 212 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 213 ``` 214 215- Execute a test suite: 216 217 ``` 218 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 219 ``` 220 221- Initialize the following test variables which will be used during test 222 execution: 223 224 User can forward declare as environment variables: 225 226 ``` 227 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 228 $ export OPENBMC_USERNAME=<openbmc username> 229 $ export OPENBMC_PASSWORD=<openbmc password> 230 $ export IPMI_COMMAND=<Dbus/External> 231 ``` 232 233 or 234 235 User can input as robot variables as part of the CLI command: 236 237 ``` 238 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 239 -v OPENBMC_USERNAME:<openbmc username> 240 -v OPENBMC_PASSWORD:<openbmc password> 241 ``` 242 243- Testing in qemu: 244 245 Set extra environment variables: 246 247 ``` 248 $ export SSH_PORT=<ssh port number> 249 $ export HTTPS_PORT=<https port number> 250 ``` 251 252 Run the QEMU CI test suite (not all tests will pass in qemu): 253 254 ``` 255 $ OPENBMC_HOST=x.x.x.x SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI tests/ 256 ``` 257 258- Run tests: 259 260 ``` 261 $ tox tests 262 ``` 263 264- How to run an individual test: 265 266 ``` 267 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 268 ``` 269 270- No preset environment variables, default configuration for all supported 271 systems: 272 273 ``` 274 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 275 ``` 276 277- No preset environment variables, one test case from a test suite: 278 279 ``` 280 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 281 ``` 282 283- No preset environment variables, the entire test suite: 284 285 ``` 286 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 287 ``` 288 289- No preset environment variables, the entire test suite excluding test cases 290 using argument file: 291 292 ``` 293 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 294 ``` 295 296- Exclude test list for supported systems: 297 298 ``` 299 Palmetto: test_lists/skip_test_palmetto 300 Witherspoon: test_lists/skip_test_witherspoon 301 ``` 302 303 Using the exclude lists (example for Witherspoon) 304 305 ``` 306 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon tests/ 307 ``` 308 309- Run IPMI tests: 310 311 Running only out-of-band IPMI tests: 312 313 ``` 314 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 315 ``` 316 317 Running only inband IPMI tests: 318 319 ``` 320 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 321 ``` 322 323- Run GUI tests: 324 325 By default, GUI runs with Firefox browser and headless mode. Example with 326 Chrome browser and header mode: 327 328 ``` 329 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 330 ``` 331 332 Run GUI default CI test bucket: 333 334 ``` 335 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 336 ``` 337 338- Run LDAP tests: 339 340 Before using LDAP test functions, be sure appropriate LDAP user(s) and 341 group(s) have been created on your LDAP server. Note: There are multiple ways 342 to create LDAP users / groups and all depend on your LDAP server. One common 343 way for openldap is ldapadd / ldapmodify refer 344 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 345 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 346 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 347 348 Note: Currently, LDAP test automation for Redfish API is in progress. The 349 format to invoke LDAP test is as follows: 350 351 ``` 352 $ cd redfish/account_service/ 353 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 354 ``` 355 356- How to run CI and CT bucket test: 357 358 Default CI test bucket list: 359 360 ``` 361 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 362 ``` 363 364 Default CI smoke test bucket list: 365 366 ``` 367 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 368 ``` 369 370- Run extended tests: 371 372 For-loop test (default iteration is 10): 373 374 ``` 375 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 376 ``` 377 378 Example using tox testing a test suite for 5 iterations "witherspoon": 379 380 ``` 381 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 382 ``` 383 384- Host CPU architecture 385 386 By default openbmc-test-automation framework assumes that host CPU is based on 387 the POWER architecture. If your host CPU is x86 add 388 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 389 environment variable: 390 391 ``` 392 $ export PLATFORM_ARCH_TYPE=x86 393 ``` 394 395**Jenkins jobs tox commands** 396 397- HW CI tox command: 398 399 ``` 400 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 401 ``` 402