1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* DMTF Redfish
6* Out-of-band IPMI
7* SSH to BMC and Host OS
8
9**Key Feature List**
10* Power on/off
11* Reboot Host
12* Reset BMC
13* Code update BMC and host
14* Power management
15* Fan controller
16* HTX bootme
17* XCAT execution
18* Network
19* IPMI support (generic and DCMI compliant)
20* Factory reset
21* RAS (Reliability, availability and serviceability)
22* Web UI testing
23* IO storage and EEH (Enhanced Error Handling) testing
24* Secure boot
25* SNMP (Simple Network Management Protocol)
26
27**Debugging Supported List**
28* SOL collection
29* FFDC collection
30* Error injection from host
31
32## Installation Setup Guide ##
33* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
34
35* Miscellaneous
36Packages required to be installed for OpenBmc Automation.
37Install the packages and it's dependencies via `pip`
38
39    REST base packages:
40    ```
41    $ pip install -U requests
42    $ pip install -U robotframework-requests
43    $ pip install -U robotframework-httplibrary
44    ```
45
46    SSH and SCP base packages:
47    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
48    ```
49    $ pip install robotframework-sshlibrary
50    $ pip install robotframework-scplibrary
51    ```
52
53    Installing tox:
54    ```
55    $ pip install -U tox
56    ```
57
58If using Python 3.x, use the corresponding `pip3` to install packages.
59
60## OpenBMC Test Development ##
61
62These documents contain details on developing OpenBMC test code and debugging.
63
64 - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information.
65 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
66 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
67   curl commands required for testing.
68 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
69 - [Tools.md](Tools.md): Reference information for helper tools.
70
71## Testing Setup Steps ##
72
73To verify the installation setup is completed and ready to execute.
74
75* Download the openbmc-test-automation repository:
76    ```
77    $ git clone https://github.com/openbmc/openbmc-test-automation
78    $ cd openbmc-test-automation
79    ```
80* Execute basic setup test run:
81    ```
82    $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
83    ```
84    where xx.xx.xx.xx is the BMC hostname or IP.
85
86## Test Layout ##
87
88There are several sub-directories within the openbmc-test-automation base which
89contain test suites, tools, templates, etc. These sub-directories are
90classified as follows:
91
92`tests/`: Contains the general test cases for OpenBMC stack functional
93          verification.
94
95`extended/`: Contains test cases for boot testing, network testing,
96             code update testing etc.
97
98`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing.
99
100`xcat/`: Contains test cases for XCAT automation.
101
102`gui/`: Contains test cases for web UI and security scanning tool automation.
103
104`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing.
105
106`network/`: Contains test cases for network testing.
107
108`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol)
109         configuration testing.
110
111`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and
112                  Serviceability) for an OpenPOWER system.
113
114`openpower/secureboot/`: Contains test cases for secure boot testing on a
115                         secure boot feature enabled OpenPOWER system only.
116
117`tools/`: Contains various tools.
118
119`templates/`: Contains sample code examples and setup testing.
120
121`test_list/`: Contains the argument files used for skipping test cases
122              (e.g "skip_test", "skip_test_extended", etc.) or
123              grouping them (e.g "HW_CI", "CT_basic_run", etc.).
124
125
126## Redfish Test Layout ##
127
128OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard
129specification and schema that meets the expectations of end users for simple,
130modern and secure management of scalable platform hardware.
131
132`redfish_test`: Contains test cases for DMTF Redfish-related feature supported
133                on OpenBMC.
134
135
136## Quickstart ##
137To run openbmc-automation first you need to install the prerequisite Python
138packages which will help to invoke tests through tox (Note that tox
139version 2.3.1 or greater is required) or via Robot CLI command.
140
141**Robot Command Line**
142
143* Execute all test suites for `tests/`:
144    ```
145    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
146    ```
147
148* Execute a test suite:
149    ```
150    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests/test_basic_poweron.robot
151    ```
152
153* Initialize the following environment variables which will be used during testing:
154    ```
155    $ export OPENBMC_HOST=<openbmc machine ip address>
156    $ export OPENBMC_PASSWORD=<openbmc password>
157    $ export OPENBMC_USERNAME=<openbmc username>
158    $ export OPENBMC_MODEL=[./data/Witherspoon.py, ./data/Palmetto.py, etc]
159    $ export IPMI_COMMAND=<Dbus/External>
160    $ export IPMI_PASSWORD=<External IPMI password>
161    ```
162
163* For QEMU tests, set the following environment variables as well:
164    ```
165    $ export SSH_PORT=<ssh port number>
166    $ export HTTPS_PORT=<https port number>
167    ```
168
169* Run tests:
170    ```
171    $ tox -e tests
172    ```
173
174* How to run individual test:
175
176    One specific test:
177    ```
178    $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
179    ```
180    No preset environment variables, default configuration for all supported
181    systems:
182    ```
183    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
184    ```
185    No preset environment variables, one test case from a test suite:
186    ```
187    $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
188    ```
189    No preset environment variables, the entire test suite:
190    ```
191    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
192    ```
193
194    No preset environment variables, the entire test suite excluding test
195    cases using argument file:
196    ```
197    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
198    ```
199
200    Exclude test list for supported systems:
201    ```
202    Palmetto:  test_lists/skip_test_palmetto
203    Witherspoon:  test_lists/skip_test_witherspoon
204    ```
205
206* How to run CI and CT bucket test:
207
208    Default CI test bucket list:
209    ```
210    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
211    ```
212
213    Default CI smoke test bucket list:
214    ```
215    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
216    ```
217
218* Run extended tests:
219
220    For-loop test (default iteration is 10):
221    ```
222    $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
223    ```
224
225    Example using tox testing a test suite for 5 iterations "witherspoon":
226    ```
227    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot
228    ```
229
230**Jenkins jobs tox commands**
231* HW CI tox command:
232    ```
233    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
234    ```
235
236## Code Update ##
237
238Currently supported BMC and PNOR update formats are UBI and non-UBI.
239For code update information, please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md)
240
241
242* UBI Format *
243
244    For BMC code update, download the system type *.ubi.mdt.tar image from
245    https://openpower.xyz/job/openbmc-build/ and run as follows:
246
247    For Witherspoon system:
248    ```
249    $ cd extended/code_update/
250    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update  bmc_code_update.robot
251    ```
252
253    For host code update, download the system type *.pnor.squashfs.tar image
254    from https://openpower.xyz/job/openpower-op-build/ and run as follows:
255
256    For Witherspoon system:
257    ```
258    $ cd extended/code_update/
259    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update  host_code_update.robot
260    ```
261
262* Non-UBI Format *
263
264    For BMC code update, download the system type *all.tar image from
265    https://openpower.xyz/job/openbmc-build/ and run as follows:
266
267    For a Zaius system:
268    ```
269    $ cd extended/code_update/
270    $ robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:<image path>/zaius-<date time>.all.tar --include Initiate_Code_Update_BMC update_bmc.robot
271    ```
272
273    For host code update, download the system type *.pnor from
274    https://openpower.xyz/job/openpower-op-build/ and run as follows:
275
276    For a Zaius system:
277    ```
278    $ cd extended/
279    $ robot -v OPENBMC_HOST:x.x.x.x -v PNOR_IMAGE_PATH:<image path>/zaius.pnor test_bios_update.robot
280    ```
281