1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* DMTF Redfish
6* Out-of-band IPMI
7* SSH to BMC and Host OS
8
9**Key Feature List**
10* Power on/off
11* Reboot Host
12* Reset BMC
13* Code update BMC and host
14* Power management
15* Fan controller
16* HTX bootme
17* XCAT execution
18* Network
19* IPMI support (generic and DCMI compliant)
20* Factory reset
21* RAS (Reliability, availability and serviceability)
22* Web UI testing
23* IO storage and EEH (Enhanced Error Handling) testing
24* Secure boot
25* SNMP (Simple Network Management Protocol)
26
27**Debugging Supported List**
28* SOL collection
29* FFDC collection
30* Error injection from host
31
32## Installation Setup Guide ##
33* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
34
35* Miscellaneous
36Packages required to be installed for OpenBmc Automation.
37Install the packages and it's dependencies via `pip`
38
39    REST base packages:
40    ```
41    $ pip install -U requests
42    $ pip install -U robotframework-requests
43    $ pip install -U robotframework-httplibrary
44    ```
45    SSH and SCP base packages:
46    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
47    ```
48    $ pip install robotframework-sshlibrary
49    $ pip install robotframework-scplibrary
50    ```
51
52## OpenBMC Test Development ##
53
54These documents contain details on developing OpenBMC test code and debugging.
55
56 - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information.
57 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
58 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
59   curl commands required for testing.
60 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
61 - [Tools.md](Tools.md): Reference information for helper tools.
62
63## Testing Setup Steps ##
64
65To verify the installation setup is completed and ready to execute.
66
67* Download the openbmc-test-automation repository:
68    ```
69    $ git clone https://github.com/openbmc/openbmc-test-automation
70    $ cd openbmc-test-automation
71    ```
72* Execute basic setup test run:
73    ```
74    $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
75    ```
76    where xx.xx.xx.xx is the BMC hostname or IP.
77
78## Test Layout ##
79
80There are several sub-directories within the openbmc-test-automation base which
81contain test suites, tools, templates, etc. These sub-directories are
82classified as follows:
83
84`tests/`: Contains the general test cases for OpenBMC stack functional
85          verification.
86
87`extended/`: Contains test cases for boot testing, network testing,
88             code update testing etc.
89
90`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing.
91
92`xcat/`: Contains test cases for XCAT automation.
93
94`gui/`: Contains test cases for web UI and security scanning tool automation.
95
96`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing.
97
98`network/`: Contains test cases for network testing.
99
100`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol)
101         configuration testing.
102
103`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and
104                  Serviceability) for an OpenPOWER system.
105
106`openpower/secureboot/`: Contains test cases for secure boot testing on a
107                         secure boot feature enabled OpenPOWER system only.
108
109`tools/`: Contains various tools.
110
111`templates/`: Contains sample code examples and setup testing.
112
113
114## Redfish Test Layout ##
115
116OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard
117specification and schema that meets the expectations of end users for simple,
118modern and secure management of scalable platform hardware.
119
120`redfish_test`: Contains test cases for DMTF Redfish-related feature supported
121                on OpenBMC.
122
123
124## Quickstart ##
125To run openbmc-automation first you need to install the prerequisite Python
126packages which will help to invoke tests through tox (Note that tox
127version 2.3.1 or greater is required) or via Robot CLI command.
128
129**Robot Command Line**
130
131* Execute all test suites for `tests/`:
132    ```
133    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
134    ```
135* Execute a test suite:
136    ```
137    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests/test_basic_poweron.robot
138    ```
139**Tox Command Line**
140
141* Install the python dependencies for tox:
142    ```
143    $ easy_install tox
144    $ easy_install pip
145    ```
146
147* Initialize the following environment variables which will be used during testing:
148    ```
149    $ export OPENBMC_HOST=<openbmc machine ip address>
150    $ export OPENBMC_PASSWORD=<openbmc password>
151    $ export OPENBMC_USERNAME=<openbmc username>
152    $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
153    $ export IPMI_COMMAND=<Dbus/External>
154    $ export IPMI_PASSWORD=<External IPMI password>
155    ```
156* For tests requiring PDU, set the following environment variables as well:
157    ```
158    $ export PDU_IP=<PDU IP address>
159    $ export PDU_USERNAME=<PDU username>
160    $ export PDU_PASSWORD=<PDU password>
161    $ export PDU_TYPE=<PDU type>
162    $ export PDU_SLOT_NO=<SLOT number>
163    ```
164    Note: For PDU_TYPE we support only synaccess at the moment.
165
166* For QEMU tests, set the following environment variables as well:
167    ```
168    $ export SSH_PORT=<ssh port number>
169    $ export HTTPS_PORT=<https port number>
170    ```
171* For BIOS tests, set the following environment variables as well:
172    ```
173    $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
174    ```
175
176* Run tests:
177    ```
178    $ tox -e tests
179    ```
180
181* How to run individual test:
182
183    One specific test:
184    ```
185    $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
186    ```
187    No preset environment variables, default configuration for all supported
188    systems:
189    ```
190    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
191    ```
192    No preset environment variables, one test case from a test suite:
193    ```
194    $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
195    ```
196    No preset environment variables, the entire test suite:
197    ```
198    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
199    ```
200
201    No preset environment variables, the entire test suite excluding test
202    cases using argument file:
203    ```
204    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
205    ```
206
207    Exclude test list for supported systems:
208    ```
209    Barrleye:  test_lists/skip_test_barreleye
210    Palmetto:  test_lists/skip_test_palmetto
211    Witherspoon:  test_lists/skip_test_witherspoon
212    ```
213
214* How to run CI and CT bucket test:
215
216    Default CI test bucket list:
217    ```
218    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
219    ```
220
221    Default CI smoke test bucket list:
222    ```
223    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
224    ```
225
226* Code update test:
227
228    For BMC code update, download the system type *.ubi.mdt.tar image from
229    https://openpower.xyz/job/openbmc-build/ and run as follows:
230
231    For Witherspoon system:
232    ```
233    $ cd extended/code_update/
234    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update  bmc_code_update.robot
235    ```
236
237    For host code update, download the system type *.pnor.squashfs.tar image
238    from https://openpower.xyz/job/openpower-op-build/ and run as follows:
239
240    For Witherspoon system:
241    ```
242    $ cd extended/code_update/
243    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update  host_code_update.robot
244    ```
245
246    For manual code update information please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md)
247
248* Run extended tests:
249
250    For loop test (default iteration is 10):
251    ```
252    $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
253    ```
254    Example using tox testing a test suite for 5 times:
255    ```
256    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5  OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot
257    ```
258
259**Jenkins jobs tox commands**
260* HW CI tox command:
261    ```
262    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
263    ```
264