1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* Out-of-band IPMI
6* SSH to BMC and Host OS
7
8**Key Feature List**
9* Power on/off
10* Reboot Host
11* Reset BMC
12* Code update BMC and host
13* Power management
14* HTX bootme
15* XCAT execution
16* Network
17
18**Debugging Supported List**
19* SOL collection
20* FFDC collection
21* Error injection from host
22
23## Installation Setup Guide ##
24* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
25
26* Miscelleneous
27Packages required to be installed for OpenBmc Automation.
28Install the packages and it's dependencies via `pip`
29
30    REST base packages:
31    ```
32    $ pip install -U requests
33    $ pip install -U robotframework-requests
34    ```
35    SSH and SCP base packages:
36    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
37    ```
38    $ pip install robotframework-sshlibrary
39    $ pip install robotframework-scplibrary
40    ```
41
42## Quickstart ##
43To run openbmc-automation first you need to install the prerequisite python
44packages which will help to invoke tests through tox.  Note that tox
45version 2.3.1 or greater is required.
46
47Install the python dependencies for tox
48```shell
49    $ easy_install tox
50    $ easy_install pip
51```
52
53Initialize the following environment variable which will be used during testing
54```shell
55    $ export OPENBMC_HOST=<openbmc machine ip address>
56    $ export OPENBMC_PASSWORD=<openbmc password>
57    $ export OPENBMC_USERNAME=<openbmc username>
58    $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
59    $ export IPMI_COMMAND=<Dbus/External>
60    $ export IPMI_PASSWORD=<External IPMI password>
61```
62
63There are two different set of test suite existing based on the usage.
64The test suites are distinctly separated by directory as under
65    tests/
66    extended/
67
68`tests`: directory contains the general test cases
69
70`extended`: directory contains the use cases for new IP network testing, PDU,
71BIOS and BMC code update.
72
73```shell
74Use Following Variables for networking test cases
75===========================================================
76    $export NEW_BMC_IP=<openbmc machine ip address>
77    $export NEW_SUBNET_MASK=<openbmc new subnet mask>
78    $export NEW_GATEWAY=<openbmc new gateway>
79==========================================================
80
81    Use following parameters for PDU:
82    $ export PDU_IP=<PDU IP address>
83    $ export PDU_USERNAME=<PDU username>
84    $ export PDU_PASSWORD=<PDU password>
85    $ export PDU_TYPE=<PDU type>
86    $ export PDU_SLOT_NO=<SLOT number>
87
88    for PDU_TYPE we support only synaccess at the moment
89
90Use following variables for syslog test cases
91==========================================================
92    $ export SYSLOG_IP_ADDRESS=<remote syslog system ip>
93    $ export SYSLOG_PORT=<remote syslog system port>
94
95Use the following variables for Qemu test run
96==========================================================
97    $ export SSH_PORT=<ssh port number>
98    $ export HTTPS_PORT=<https port number>
99
100Use the following variables for BIOS update testing
101==========================================================
102    $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
103
104```
105
106Run tests
107```shell
108    $ tox -e tests
109```
110
111How to test individual test
112```shell
113    One specific test
114    $ tox -e default -- -t '"DIMM0 no fault"' tests/test_sensors.robot
115
116    No preset environment variables, default configuration for all supported systems.
117    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
118
119    No preset environment variables, one test case from one test suite
120    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot
121
122    No preset environment variables, one test suite  for a palmetto system
123    $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot
124
125    No preset environment variables, the entire test suite for a barreleye system
126    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests
127
128    Default CI test bucket list:
129    No preset environment variables, the entire test suite excluding test
130    cases using argument file.
131    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/skip_test tests
132
133    Exclude test list for supported systems:
134    Barrleye:  test_lists/skip_test_barreleye
135    Palmetto:  test_lists/skip_test_palmetto
136    Witherspoon:  test_lists/skip_test_witherspoon
137```
138
139It can also be run by passing variables from the cli...
140```shell
141    Run one test suite using using pybot
142    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot
143
144    Run entire test suite using using pybot
145    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests
146
147    Run entire test suite using external ipmitool
148    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests
149```
150
151Run extended tests
152```shell
153    Set the preset environment variables, run test suite for a barreleye system
154    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot
155
156    Similarly for Network, PDU and update BIOS
157
158    For BMC code update, download the system type *.all.tar image from https://openpower.xyz
159    and run as follows:
160
161    For Barreleye system
162    python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar  extended/code_update/update_bmc.robot
163
164    For loop test (Default iteration is 10)
165    python -m robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
166    Below is sample command using tox to test only fw version using Barreleye system for 5 times
167    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5  OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot
168```
169
170Jenkins jobs tox commands
171```shell
172    HW CI tox command
173    Set the preset environment variables, run HW CI test for a barreleye system
174    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/HW_CI tests
175
176```
177
178Template to be used to create new stand-alone python programs.
179```shell
180
181If you wish to create a new python stand-alone program, please copy bin/python_pgm_template to your new name and then begin your work.  Example:
182
183cd bin
184cp python_pgm_template my_new_program
185
186This template has much of your preliminary work done for you and it will help us all follow a similar structure.
187
188Features:
189- Help text and argparsing started for you.
190- Support for "stock" parameters like quiet, debug, test_mode.
191- exit_function and signal_handler defined.
192- validate_parms function pre-created.
193- main function follows conventional startup:
194
195    if not gen_get_options(parser, stock_list):
196        return False
197
198    if not validate_parms():
199        return False
200
201    qprint_pgm_header()
202
203    # Your code here.
204
205```
206
207Command to get GitHub issues (any GitHub repository) report in CSV format
208Note: On Prompt "Enter your GitHub Password:" enter your GitHub password.
209```shell
210python ./tools/github_issues_to_csv <github user> <github repo>
211
212Example for getting openbmc issues
213python ./tools/github_issues_to_csv <github user>  openbmc/openbmc
214
215Example for getting openbmc-test-automation issues
216python ./tools/github_issues_to_csv <github user>  openbmc/openbmc-test-automation
217```
218
219
220Command to generate Robot test cases test documentations
221
222```shell
223./tools/generate_test_document <Robot test cases directory path> <test case document file path>
224
225Example for generating tests cases documentation for tests directory
226./tools/generate_test_document tests testsdirectoryTCdocs.html
227
228Example for generating tests cases documentation (tests,gui,extended TCs)
229# Note: Invoke the tool with out argument
230./tools/generate_test_document
231```
232