1## Features of OpenBMC Test Automation 2 3**Interface Feature List** 4 5- DMTF Redfish 6- Out-of-band IPMI 7- SSH to BMC and Host OS 8- [Legacy REST](https://github.com/openbmc/openbmc-test-automation/releases/tag/v4.0-stable) 9 10**Key Feature List** 11 12- Power on/off 13- Reboot Host 14- Reset BMC 15- Code update BMC and host 16- Power management 17- Fan controller 18- HTX bootme 19- XCAT execution 20- Network 21- IPMI support (generic and DCMI compliant) 22- Factory reset 23- RAS (Reliability, availability and serviceability) 24- Web UI testing 25- Secure boot 26- SNMP (Simple Network Management Protocol) 27- Remote Logging via Rsyslog 28- LDAP (Lightweight Directory Access Protocol) 29- Certificate 30- Local User Management(Redfish/IPMI) 31- DateTime 32- Event Logging 33- PLDM (Platform Level Data Model) via pldmtool 34 35**Debugging Supported List** 36 37- SOL collection 38- FFDC collection 39- Error injection from host 40 41## Installation Setup Guide 42 43- [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 44 45- Miscellaneous Packages required to be installed for OpenBmc Automation. 46 Install the packages and it's dependencies via `pip` 47 48If using Python 3.x, use the corresponding `pip3` to install packages. Note: 49Older Python 2.x is not actively supported. 50 51Installing requirement dependencies: 52 53``` 54 $ pip install -r requirements.txt 55``` 56 57Optional packages but required for `redfish/dmtf_tools/` 58 59``` 60 $ pip install -r requirements_optional.txt 61``` 62 63you'll find this file once your clone openbmc-test-automation repository. 64 65For Web UI ( GUI ) Testing setup, please the follow instructions mentioned in 66the OpenBMC GUI Test Setup Guide. Note: GUI test cases under `gui/` will not 67work without the above setup completed in your environment. 68[OpenBMC GUI Test Setup Guide](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/gui_setup_reference.md) 69 70Installing tox: 71 72``` 73 $ pip install -U tox 74``` 75 76Installing expect (Ubuntu example): 77 78``` 79 $ sudo apt-get install expect 80``` 81 82## OpenBMC Test Development 83 84These documents contain details on developing OpenBMC test code and debugging. 85 86- [MAINTAINERS](OWNERS): OpenBMC test code maintainers information. 87- [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 88- [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): 89 To check common code misspellings, syntax and standard checks. 90- [Redfish Coding Guidelines](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_coding_guidelines.md): 91 Refer for Redfish coding guidelines. 92- [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): 93 Quick reference for some common curl commands required for legacy REST 94 testing. 95- [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): 96 Quick reference for some common curl commands required for redfish testing. 97- [README.md](https://github.com/openbmc/webui-vue/blob/master/README.md): Web 98 UI setup reference. 99- [Redfish Request Via mTLS](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/redfish_request_via_mTLS.md): 100 Reference for Redfish Request Via mTLS . 101- [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): 102 Submitting changes via Gerrit server 103 104## OpenBMC Test Documentation 105 106- [OpenBMC Test Architecture](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_architecture.md): 107 Reference for OpenBMC Test Architecture. 108- [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): 109 Reference information for helper tools. 110- [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): 111 Currently supported BMC and PNOR update. 112- [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): 113 Steps to create and install CA signed certificate. 114- [Boot Test](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/boot_test.md): 115 Boot test for OpenBMC. 116 117## Supported Systems Architecture 118 119OpenBMC test infrastructure is proven capable of running on: 120 121- POWER 122- x86 systems running OpenBMC firmware stack. 123 124## Testing Setup Steps 125 126To verify the installation setup is completed and ready to execute. 127 128- Download the openbmc-test-automation repository: 129 130 ``` 131 $ git clone https://github.com/openbmc/openbmc-test-automation 132 $ cd openbmc-test-automation 133 ``` 134 135- Execute basic setup test run: 136 137 ``` 138 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 139 ``` 140 141 where xx.xx.xx.xx is the BMC hostname or IP. 142 143## Test Layout 144 145There are several sub-directories within the openbmc-test-automation base which 146contain test suites, tools, templates, etc. These sub-directories are classified 147as follows: 148 149`docs/`: Contains the documentation related to OpenBMC. 150 151`redfish/`: Contains the general test cases for OpenBMC stack functional 152verification. 153 154`systest/`: Contains test cases for HTX bootme testing. 155 156`xcat/`: Contains test cases for XCAT automation. 157 158`gui/test/`: Contains test cases for testing web-based interface built on 159AngularJS. 160 161`gui/gui_test/`: Contains test cases for testing web-based user interface built 162on Vue.js. 163 164`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, 165platform, OEM). 166 167`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 168configuration testing. 169 170`openpower/`: Contains test cases for an OpenPOWER based system. 171 172`tools/`: Contains various tools. 173 174`templates/`: Contains sample code examples and setup testing. 175 176`test_list/`: Contains the argument files used for skipping test cases (e.g 177"skip_test", "skip_test_extended", etc.) or grouping them (e.g "HW_CI", 178"CT_basic_run", etc.). 179 180## Redfish Test Layout 181 182OpenBMC is moving steadily towards DTMF Redfish, which is an open industry 183standard specification and schema that meets the expectations of end users for 184simple, modern and secure management of scalable platform hardware. 185 186`redfish/`: Contains test cases for DMTF Redfish-related feature supported on 187OpenBMC. 188 189`redfish/extended/`: Contains test cases for combined DMTF Redfish-related 190feature supported on OpenBMC. Some of the test will be deprecated. 191 192Note: Work in progress test development parameter 193`-v REDFISH_SUPPORT_TRANS_STATE:1` to force the test suites to execute in 194redfish mode only. 195 196## Quickstart 197 198To run openbmc-automation first you need to install the prerequisite Python 199packages which will help to invoke tests through tox (Note that tox version 2002.3.1 or greater is required) or via Robot CLI command. 201 202**Robot Command Line** 203 204- Execute all test suites for `redfish/` and `ipmi/`: 205 206 ``` 207 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish ipmi 208 ``` 209 210- Execute a test suite: 211 212 ``` 213 $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 214 ``` 215 216- Initialize the following test variables which will be used during test 217 execution: 218 219 User can forward declare as environment variables: 220 221 ``` 222 $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 223 $ export OPENBMC_USERNAME=<openbmc username> 224 $ export OPENBMC_PASSWORD=<openbmc password> 225 $ export IPMI_COMMAND=<Dbus/External> 226 ``` 227 228 or 229 230 User can input as robot variables as part of the CLI command: 231 232 ``` 233 -v OPENBMC_HOST:<openbmc machine IP address/hostname> 234 -v OPENBMC_USERNAME:<openbmc username> 235 -v OPENBMC_PASSWORD:<openbmc password> 236 ``` 237 238- Testing in qemu: 239 240 Set extra environment variables: 241 242 ``` 243 $ export SSH_PORT=<ssh port number> 244 $ export HTTPS_PORT=<https port number> 245 ``` 246 247 Run the QEMU CI test suite: 248 249 ``` 250 $ OPENBMC_HOST=xx.xx.xx.xx SSH_PORT=<port number> HTTPS_PORT=<port number> robot -A test_lists/QEMU_CI redfish/ ipmi/ 251 ``` 252 253- Run tests: 254 255- How to run an individual test: 256 257 ``` 258 $ OPENBMC_HOST=xx.xx.xx.xx tox -e default -- --include Test_SSH_And_IPMI_Connections redfish/extended/test_basic_ci.robot 259 ``` 260 261- How to run CI and CT bucket test: 262 263 Default CI test bucket list: 264 265 ``` 266 $ OPENBMC_HOST=xx.xx.xx.xx tox -e default -- --argumentfile test_lists/HW_CI redfish/ ipmi/ 267 ``` 268 269 Default CI smoke test bucket list: 270 271 ``` 272 $ OPENBMC_HOST=xx.xx.xx.xx tox -e default -- --argumentfile test_lists/CT_basic_run redfish/ ipmi/ 273 ``` 274 275- Exclude test list for supported systems: 276 277 ``` 278 Witherspoon: test_lists/skip_test_witherspoon 279 ``` 280 281 Using the exclude lists (example for Witherspoon) 282 283 ``` 284 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -A test_lists/skip_test_witherspoon redfish/ ipmi/ 285 ``` 286 287- Run IPMI tests via robot CLI interface: 288 289 Running only out-of-band IPMI tests: 290 291 ``` 292 $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:xx.xx.xx.xx --argumentfile test_lists/witherspoon/skip_inband_ipmi ipmi/ 293 ``` 294 295 Running only inband IPMI tests: 296 297 ``` 298 $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:xx.xx.xx.xx -v OS_HOST:xx.xx.xx.xx -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi ipmi/ 299 ``` 300 301- Run GUI tests via robot CLI interface: 302 303 By default, GUI runs with Firefox browser and headless mode. Example with 304 Chrome browser and header mode: 305 306 ``` 307 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 308 ``` 309 310 Run GUI default CI test bucket: 311 312 ``` 313 $ robot -v OPENBMC_HOST:xx.xx.xx.xx --argumentfile test_lists/BMC_WEB_CI gui/test/ 314 ``` 315 316- Run LDAP tests via robot CLI interface: 317 318 Before using LDAP test functions, be sure appropriate LDAP user(s) and 319 group(s) have been created on your LDAP server. Note: There are multiple ways 320 to create LDAP users / groups and all depend on your LDAP server. One common 321 way for openldap is ldapadd / ldapmodify refer 322 https://linux.die.net/man/1/ldapadd For ldapsearch, refer to 323 "https://linux.die.net/man/1/ldapsearch". Microsoft ADS: refer to 324 https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 325 326 Note: Currently, LDAP test automation for Redfish API is in progress. The 327 format to invoke LDAP test is as follows: 328 329 ``` 330 $ cd redfish/account_service/ 331 $ robot -v OPENBMC_HOST:xx.xx.xx.xx -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 332 ``` 333 334- Host CPU architecture 335 336 By default openbmc-test-automation framework assumes that host CPU is based on 337 the POWER architecture. If your host CPU is x86 add 338 `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an 339 environment variable: 340 341 ``` 342 $ export PLATFORM_ARCH_TYPE=x86 343 ``` 344 345**Jenkins jobs tox commands** 346 347- HW CI tox command: 348 349 ``` 350 $ OPENBMC_HOST=xx.xx.xx.xx tox -e default -- --argumentfile test_lists/HW_CI redfish/ ipmi/ 351 ``` 352