1## Features of OpenBMC Test Automation ## 2 3**Interface Feature List** 4* REST 5* DMTF Redfish 6* Out-of-band IPMI 7* SSH to BMC and Host OS 8 9**Key Feature List** 10* Power on/off 11* Reboot Host 12* Reset BMC 13* Code update BMC and host 14* Power management 15* Fan controller 16* HTX bootme 17* XCAT execution 18* Network 19* IPMI support (generic and DCMI compliant) 20* Factory reset 21* RAS (Reliability, availability and serviceability) 22* Web UI testing 23* Secure boot 24* SNMP (Simple Network Management Protocol) 25* Remote Logging via Rsyslog 26* LDAP (Lightweight Directory Access Protocol) 27* Certificate 28* Local User Management(Redfish/IPMI) 29* DateTime 30* Event Logging 31* PLDM (Platform Level Data Model) via pldmtool 32 33**Debugging Supported List** 34* SOL collection 35* FFDC collection 36* Error injection from host 37 38## Installation Setup Guide ## 39* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 40 41* Miscellaneous 42Packages required to be installed for OpenBmc Automation. 43Install the packages and it's dependencies via `pip` 44 45If using Python 3.x, use the corresponding `pip3` to install packages. 46Note: Older Python 2.x is not actively supported. 47 48REST base packages: 49``` 50 $ pip install -U requests 51 $ pip install -U robotframework-requests 52 $ pip install -U robotframework-httplibrary 53``` 54 55Python redfish library packages: 56For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library) 57``` 58 $ pip install redfish 59``` 60 61SSH and SCP base packages: 62For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 63``` 64 $ pip install robotframework-sshlibrary 65 $ pip install robotframework-scplibrary 66``` 67 68Installing requirement dependencies: 69``` 70 $ pip install -r requirements.txt 71``` 72you'll find this file once your clone openbmc-test-automation repository. 73 74Installing tox: 75``` 76 $ pip install -U tox 77``` 78 79Installing expect: 80``` 81 $ sudo apt-get install expect (Ubuntu example) 82``` 83 84## OpenBMC Test Development ## 85 86These documents contain details on developing OpenBMC test code and debugging. 87 88 - [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information. 89 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 90 - [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): To check common code misspellings, syntax and standard checks. 91 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 92 curl commands required for legacy REST testing. 93 - [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): Quick reference for some common 94 curl commands required for redfish testing. 95 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 96 - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server 97 98## OpenBMC Test Documentation ## 99 100 - [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools. 101 - [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): Currently supported BMC and PNOR update. 102 - [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): Steps to create and install CA signed certificate. 103 104## Supported Systems Architecture ## 105 106 OpenBMC test infrastructure is proven capable of running on: 107 - POWER 108 - x86 109 systems running OpenBMC firmware stack. 110 111## Testing Setup Steps ## 112 113To verify the installation setup is completed and ready to execute. 114 115* Download the openbmc-test-automation repository: 116 ``` 117 $ git clone https://github.com/openbmc/openbmc-test-automation 118 $ cd openbmc-test-automation 119 ``` 120* Execute basic setup test run: 121 ``` 122 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 123 ``` 124 where xx.xx.xx.xx is the BMC hostname or IP. 125 126## Test Layout ## 127 128There are several sub-directories within the openbmc-test-automation base which 129contain test suites, tools, templates, etc. These sub-directories are 130classified as follows: 131 132`tests/`: Contains the general test cases for OpenBMC stack functional 133 verification. The "tests" subdirectory uses legacy REST and will be 134 deprecated at some point and therefore no longer supported. 135 136`extended/`: Contains test cases for boot testing, code update testing using legacy REST, etc. 137 138`systest/`: Contains test cases for HTX bootme testing. 139 140`xcat/`: Contains test cases for XCAT automation. 141 142`gui/test/`: Contains test cases for testing web-based interface built on AngularJS. 143 144`gui/gui_test/`: Contains test cases for testing web-based user interface built on Vue.js. 145 146`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, platform, OEM). 147 148`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 149 configuration testing. 150 151`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 152 Serviceability) for an OpenPOWER system. 153 154`openpower/secureboot/`: Contains test cases for secure boot testing on a 155 secure boot feature enabled OpenPOWER system only. 156 157`tools/`: Contains various tools. 158 159`templates/`: Contains sample code examples and setup testing. 160 161`test_list/`: Contains the argument files used for skipping test cases 162 (e.g "skip_test", "skip_test_extended", etc.) or 163 grouping them (e.g "HW_CI", "CT_basic_run", etc.). 164 165 166## Redfish Test Layout ## 167 168OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 169specification and schema that meets the expectations of end users for simple, 170modern and secure management of scalable platform hardware. 171 172`redfish/`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC. 173 174`redfish/extended/`: Contains test cases for combined legacy REST and DMTF Redfish-related 175 feature supported on OpenBMC. 176 177Note: Work in progress test development parameter `-v REDFISH_SUPPORT_TRANS_STATE:1` to force 178 the test suites to execute in redfish mode only. 179 180 181## Quickstart ## 182To run openbmc-automation first you need to install the prerequisite Python 183packages which will help to invoke tests through tox (Note that tox 184version 2.3.1 or greater is required) or via Robot CLI command. 185 186**Robot Command Line** 187 188* Execute all test suites for `tests/`: 189 ``` 190 $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 191 ``` 192 193* Execute a test suite: 194 ``` 195 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 196 ``` 197 198* Initialize the following test variables which will be used during test execution: 199 200 User can forward declare as environment variables: 201 ``` 202 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 203 $ export OPENBMC_USERNAME=<openbmc username> 204 $ export OPENBMC_PASSWORD=<openbmc password> 205 $ export IPMI_COMMAND=<Dbus/External> 206 ``` 207 208 or 209 210 User can input as robot variables as part of the CLI command: 211 ``` 212 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 213 -v OPENBMC_USERNAME:<openbmc username> 214 -v OPENBMC_PASSWORD:<openbmc password> 215 ``` 216 217* For QEMU tests, set the following environment variables as well: 218 ``` 219 $ export SSH_PORT=<ssh port number> 220 $ export HTTPS_PORT=<https port number> 221 ``` 222 223* Run tests: 224 ``` 225 $ tox tests 226 ``` 227 228* How to run individual test: 229 230 One specific test: 231 ``` 232 $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 233 ``` 234 No preset environment variables, default configuration for all supported 235 systems: 236 ``` 237 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 238 ``` 239 No preset environment variables, one test case from a test suite: 240 ``` 241 $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 242 ``` 243 No preset environment variables, the entire test suite: 244 ``` 245 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 246 ``` 247 248 No preset environment variables, the entire test suite excluding test 249 cases using argument file: 250 ``` 251 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 252 ``` 253 254 Exclude test list for supported systems: 255 ``` 256 Palmetto: test_lists/skip_test_palmetto 257 Witherspoon: test_lists/skip_test_witherspoon 258 ``` 259 260 261* Run IPMI tests: 262 263 Running only out-of-band IPMI tests: 264 ``` 265 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 266 ``` 267 268 Running only inband IPMI tests: 269 ``` 270 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 271 ``` 272 273 274* Run GUI tests: 275 276 By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode: 277 ``` 278 $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 279 ``` 280 281 Run GUI default CI test bucket: 282 ``` 283 $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 284 ``` 285 286* Run LDAP tests: 287 288 Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server. 289 Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server. 290 One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd 291 For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch". 292 Microsoft ADS: refer to https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 293 294 Note: Currently, LDAP test automation for Redfish API is in progress. The format to invoke LDAP test is as follows: 295 ``` 296 $ cd redfish/account_service/ 297 $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 298 ``` 299 300* How to run CI and CT bucket test: 301 302 Default CI test bucket list: 303 ``` 304 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 305 ``` 306 307 Default CI smoke test bucket list: 308 ``` 309 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 310 ``` 311 312* Run extended tests: 313 314 For-loop test (default iteration is 10): 315 ``` 316 $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 317 ``` 318 319 Example using tox testing a test suite for 5 iterations "witherspoon": 320 ``` 321 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 322 ``` 323 324* Host CPU architecture 325 326 By default openbmc-test-automation framework assumes that host CPU is based on the POWER architecture. 327 If your host CPU is x86 add `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an environment variable: 328 ``` 329 $ export PLATFORM_ARCH_TYPE=x86 330 ``` 331 332**Jenkins jobs tox commands** 333* HW CI tox command: 334 ``` 335 $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 336 ``` 337