1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* Out-of-band IPMI
6* SSH to BMC and Host OS
7
8**Key Feature List**
9* Power on/off
10* Reboot Host
11* Reset BMC
12* Code update BMC and host
13* Power management
14* Fan controller
15* HTX bootme
16* XCAT execution
17* Network
18* IPMI support (generic and DCMI compliant)
19* Factory reset
20* RAS (Reliability, availability and serviceability)
21* Web UI testing
22* IO storage and EEH (Enhanced Error Handling) testing
23* Secure boot
24
25**Debugging Supported List**
26* SOL collection
27* FFDC collection
28* Error injection from host
29
30## Installation Setup Guide ##
31* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
32
33* Miscellaneous
34Packages required to be installed for OpenBmc Automation.
35Install the packages and it's dependencies via `pip`
36
37    REST base packages:
38    ```
39    $ pip install -U requests
40    $ pip install -U robotframework-requests
41    ```
42    SSH and SCP base packages:
43    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
44    ```
45    $ pip install robotframework-sshlibrary
46    $ pip install robotframework-scplibrary
47    ```
48
49## OpenBMC Test Development ##
50
51These documents contain details on developing OpenBMC test code and debugging.
52
53 - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information.
54 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
55 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
56   curl commands required for testing.
57 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
58 - [Tools.md](Tools.md): Reference information for helper tools.
59
60## Testing Setup Steps ##
61
62To verify the installation setup is completed and ready to execute.
63
64* Download the openbmc-test-automation repository:
65    ```
66    $ git clone https://github.com/openbmc/openbmc-test-automation
67    $ cd openbmc-test-automation
68    ```
69* Execute basic setup test run:
70    ```
71    $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
72    ```
73    where xx.xx.xx.xx is the BMC hostname or IP.
74
75## Test Layout ##
76
77There are several sub-directories within the openbmc-test-automation base which
78contain test suites, tools, templates, etc. These sub-directories are
79classified as follows:
80
81`tests/`: Contains the general test cases for OpenBMC stack functional
82          verification.
83
84`extended/`: Contains test cases for boot testing, network testing,
85             code update testing etc.
86
87`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing.
88
89`xcat/`: Contains test cases for XCAT automation.
90
91`ras/`: Contains test cases for RAS (Reliability, availability and
92        serviceability).
93
94`gui/`: Contains test cases for web UI and security scanning tool automation.
95
96`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing.
97
98`network/`: Contains test cases for network testing.
99
100`secureboot/`: Contains test cases for secure boot testing on a secure boot
101               feature enabled system only.
102
103`tools/`: Contains various tools.
104
105`templates/`: Contains sample code examples and setup testing.
106`
107
108## Quickstart ##
109To run openbmc-automation first you need to install the prerequisite Python
110packages which will help to invoke tests through tox (Note that tox
111version 2.3.1 or greater is required) or via Robot CLI command.
112
113**Robot Command Line**
114
115* Execute all test suites for `tests/`:
116    ```
117    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
118    ```
119* Execute a test suite:
120    ```
121    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests/test_basic_poweron.robot
122    ```
123**Tox Command Line**
124
125* Install the python dependencies for tox:
126    ```
127    $ easy_install tox
128    $ easy_install pip
129    ```
130
131* Initialize the following environment variables which will be used during testing:
132    ```
133    $ export OPENBMC_HOST=<openbmc machine ip address>
134    $ export OPENBMC_PASSWORD=<openbmc password>
135    $ export OPENBMC_USERNAME=<openbmc username>
136    $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
137    $ export IPMI_COMMAND=<Dbus/External>
138    $ export IPMI_PASSWORD=<External IPMI password>
139    ```
140* For tests requiring PDU, set the following environment variables as well:
141    ```
142    $ export PDU_IP=<PDU IP address>
143    $ export PDU_USERNAME=<PDU username>
144    $ export PDU_PASSWORD=<PDU password>
145    $ export PDU_TYPE=<PDU type>
146    $ export PDU_SLOT_NO=<SLOT number>
147    ```
148    Note: For PDU_TYPE we support only synaccess at the moment.
149
150* For QEMU tests, set the following environment variables as well:
151    ```
152    $ export SSH_PORT=<ssh port number>
153    $ export HTTPS_PORT=<https port number>
154    ```
155* For BIOS tests, set the following environment variables as well:
156    ```
157    $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
158    ```
159
160* Run tests:
161    ```
162    $ tox -e tests
163    ```
164
165* How to run individual test:
166
167    One specific test:
168    ```
169    $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
170    ```
171    No preset environment variables, default configuration for all supported
172    systems:
173    ```
174    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
175    ```
176    No preset environment variables, one test case from a test suite:
177    ```
178    $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
179    ```
180    No preset environment variables, the entire test suite:
181    ```
182    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
183    ```
184
185    No preset environment variables, the entire test suite excluding test
186    cases using argument file:
187    ```
188    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
189    ```
190
191    Exclude test list for supported systems:
192    ```
193    Barrleye:  test_lists/skip_test_barreleye
194    Palmetto:  test_lists/skip_test_palmetto
195    Witherspoon:  test_lists/skip_test_witherspoon
196    ```
197
198* How to run CI and CT bucket test:
199
200    Default CI test bucket list:
201    ```
202    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
203    ```
204
205    Default CI smoke test bucket list:
206    ```
207    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
208    ```
209
210* Code update test:
211
212    For BMC code update, download the system type *.ubi.mdt.tar image from
213    https://openpower.xyz/job/openbmc-build/ and run as follows:
214
215    For Witherspoon system:
216    ```
217    $ cd extended/code_update/
218    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update  bmc_code_update.robot
219    ```
220
221    For host code update, download the system type *.pnor.squashfs.tar image
222    from https://openpower.xyz/job/openpower-op-build/ and run as follows:
223
224    For Witherspoon system:
225    ```
226    $ cd extended/code_update/
227    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update  host_code_update.robot
228    ```
229
230    For manual code update information please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md)
231
232* Run extended tests:
233
234    For loop test (default iteration is 10):
235    ```
236    $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
237    ```
238    Example using tox testing a test suite for 5 times:
239    ```
240    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5  OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot
241    ```
242
243**Jenkins jobs tox commands**
244* HW CI tox command:
245    ```
246    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
247    ```
248