1## Features of OpenBMC Test Automation ## 2 3**Interface Feature List** 4* REST 5* Out-of-band IPMI 6* SSH to BMC and Host OS 7 8**Key Feature List** 9* Power on/off 10* Reboot Host 11* Reset BMC 12* Code update BMC and host 13* Power management 14* HTX bootme 15* XCAT execution 16* Network 17 18**Debugging Supported List** 19* SOL collection 20* FFDC collection 21* Error injection from host 22 23## Installation Setup Guide ## 24* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 25 26* Miscelleneous 27Packages required to be installed for OpenBmc Automation. 28Install the packages and it's dependencies via `pip` 29 30 REST base packages: 31 ``` 32 $ pip install -U requests 33 $ pip install -U robotframework-requests 34 ``` 35 SSH and SCP base packages: 36 For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 37 ``` 38 $ pip install robotframework-sshlibrary 39 $ pip install robotframework-scplibrary 40 ``` 41## Testing Setup Steps ## 42 43To verify the installation setup is completed and ready to execute. 44 45* Download the openbmc-test-automation repository: 46 ``` 47 $ git clone https://github.com/openbmc/openbmc-test-automation 48 $ cd openbmc-test-automation 49 ``` 50* Execute basic setup test run: 51 ``` 52 $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 53 ``` 54 where xx.xx.xx.xx is the BMC hostname or IP. 55 56## Quickstart ## 57To run openbmc-automation first you need to install the prerequisite python 58packages which will help to invoke tests through tox. Note that tox 59version 2.3.1 or greater is required. 60 61Install the python dependencies for tox 62```shell 63 $ easy_install tox 64 $ easy_install pip 65``` 66 67Initialize the following environment variable which will be used during testing 68```shell 69 $ export OPENBMC_HOST=<openbmc machine ip address> 70 $ export OPENBMC_PASSWORD=<openbmc password> 71 $ export OPENBMC_USERNAME=<openbmc username> 72 $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc] 73 $ export IPMI_COMMAND=<Dbus/External> 74 $ export IPMI_PASSWORD=<External IPMI password> 75``` 76 77There are two different set of test suite existing based on the usage. 78The test suites are distinctly separated by directory as under 79 tests/ 80 extended/ 81 82`tests`: directory contains the general test cases 83 84`extended`: directory contains the use cases for new IP network testing, PDU, 85BIOS and BMC code update. 86 87```shell 88Use Following Variables for networking test cases 89=========================================================== 90 $export NEW_BMC_IP=<openbmc machine ip address> 91 $export NEW_SUBNET_MASK=<openbmc new subnet mask> 92 $export NEW_GATEWAY=<openbmc new gateway> 93========================================================== 94 95 Use following parameters for PDU: 96 $ export PDU_IP=<PDU IP address> 97 $ export PDU_USERNAME=<PDU username> 98 $ export PDU_PASSWORD=<PDU password> 99 $ export PDU_TYPE=<PDU type> 100 $ export PDU_SLOT_NO=<SLOT number> 101 102 for PDU_TYPE we support only synaccess at the moment 103 104Use following variables for syslog test cases 105========================================================== 106 $ export SYSLOG_IP_ADDRESS=<remote syslog system ip> 107 $ export SYSLOG_PORT=<remote syslog system port> 108 109Use the following variables for Qemu test run 110========================================================== 111 $ export SSH_PORT=<ssh port number> 112 $ export HTTPS_PORT=<https port number> 113 114Use the following variables for BIOS update testing 115========================================================== 116 $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor 117 118``` 119 120Run tests 121```shell 122 $ tox -e tests 123``` 124 125How to test individual test 126```shell 127 One specific test 128 $ tox -e default -- -t '"DIMM0 no fault"' tests/test_sensors.robot 129 130 No preset environment variables, default configuration for all supported systems. 131 $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 132 133 No preset environment variables, one test case from one test suite 134 $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot 135 136 No preset environment variables, one test suite for a palmetto system 137 $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot 138 139 No preset environment variables, the entire test suite for a barreleye system 140 $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests 141 142 Default CI test bucket list: 143 No preset environment variables, the entire test suite excluding test 144 cases using argument file. 145 $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/skip_test tests 146 147 Exclude test list for supported systems: 148 Barrleye: test_lists/skip_test_barreleye 149 Palmetto: test_lists/skip_test_palmetto 150 Witherspoon: test_lists/skip_test_witherspoon 151``` 152 153It can also be run by passing variables from the cli... 154```shell 155 Run one test suite using using pybot 156 $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot 157 158 Run entire test suite using using pybot 159 $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests 160 161 Run entire test suite using external ipmitool 162 $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests 163``` 164 165Run extended tests 166```shell 167 Set the preset environment variables, run test suite for a barreleye system 168 $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot 169 170 Similarly for Network, PDU and update BIOS 171 172 For BMC code update, download the system type *.all.tar image from https://openpower.xyz 173 and run as follows: 174 175 For Barreleye system 176 python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar extended/code_update/update_bmc.robot 177 178 For loop test (Default iteration is 10) 179 python -m robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 180 Below is sample command using tox to test only fw version using Barreleye system for 5 times 181 OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot 182``` 183 184Jenkins jobs tox commands 185```shell 186 HW CI tox command 187 Set the preset environment variables, run HW CI test for a barreleye system 188 $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/HW_CI tests 189 190``` 191 192Template to be used to create new stand-alone python programs. 193```shell 194 195If you wish to create a new python stand-alone program, please copy bin/python_pgm_template to your new name and then begin your work. Example: 196 197cd bin 198cp python_pgm_template my_new_program 199 200This template has much of your preliminary work done for you and it will help us all follow a similar structure. 201 202Features: 203- Help text and argparsing started for you. 204- Support for "stock" parameters like quiet, debug, test_mode. 205- exit_function and signal_handler defined. 206- validate_parms function pre-created. 207- main function follows conventional startup: 208 209 if not gen_get_options(parser, stock_list): 210 return False 211 212 if not validate_parms(): 213 return False 214 215 qprint_pgm_header() 216 217 # Your code here. 218 219``` 220 221Command to get GitHub issues (any GitHub repository) report in CSV format 222Note: On Prompt "Enter your GitHub Password:" enter your GitHub password. 223```shell 224python ./tools/github_issues_to_csv <github user> <github repo> 225 226Example for getting openbmc issues 227python ./tools/github_issues_to_csv <github user> openbmc/openbmc 228 229Example for getting openbmc-test-automation issues 230python ./tools/github_issues_to_csv <github user> openbmc/openbmc-test-automation 231``` 232 233 234Command to generate Robot test cases test documentations 235 236```shell 237./tools/generate_test_document <Robot test cases directory path> <test case document file path> 238 239Example for generating tests cases documentation for tests directory 240./tools/generate_test_document tests testsdirectoryTCdocs.html 241 242Example for generating tests cases documentation (tests,gui,extended TCs) 243# Note: Invoke the tool with out argument 244./tools/generate_test_document 245``` 246