1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* DMTF Redfish
6* Out-of-band IPMI
7* SSH to BMC and Host OS
8
9**Key Feature List**
10* Power on/off
11* Reboot Host
12* Reset BMC
13* Code update BMC and host
14* Power management
15* Fan controller
16* HTX bootme
17* XCAT execution
18* Network
19* IPMI support (generic and DCMI compliant)
20* Factory reset
21* RAS (Reliability, availability and serviceability)
22* Web UI testing
23* IO storage and EEH (Enhanced Error Handling) testing
24* Secure boot
25* SNMP (Simple Network Management Protocol)
26* Remote Logging via Rsyslog
27* LDAP (Lightweight Directory Access Protocol)
28* Certificate
29
30**Debugging Supported List**
31* SOL collection
32* FFDC collection
33* Error injection from host
34
35## Installation Setup Guide ##
36* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
37
38* Miscellaneous
39Packages required to be installed for OpenBmc Automation.
40Install the packages and it's dependencies via `pip`
41
42    REST base packages:
43    ```
44    $ pip install -U requests
45    $ pip install -U robotframework-requests
46    $ pip install -U robotframework-httplibrary
47    ```
48
49    SSH and SCP base packages:
50    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
51    ```
52    $ pip install robotframework-sshlibrary
53    $ pip install robotframework-scplibrary
54    ```
55
56    Installing tox:
57    ```
58    $ pip install -U tox
59    ```
60
61If using Python 3.x, use the corresponding `pip3` to install packages.
62
63## OpenBMC Test Development ##
64
65These documents contain details on developing OpenBMC test code and debugging.
66
67 - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information.
68 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
69 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
70   curl commands required for testing.
71 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
72 - [Tools.md](Tools.md): Reference information for helper tools.
73
74## Testing Setup Steps ##
75
76To verify the installation setup is completed and ready to execute.
77
78* Download the openbmc-test-automation repository:
79    ```
80    $ git clone https://github.com/openbmc/openbmc-test-automation
81    $ cd openbmc-test-automation
82    ```
83* Execute basic setup test run:
84    ```
85    $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
86    ```
87    where xx.xx.xx.xx is the BMC hostname or IP.
88
89## Test Layout ##
90
91There are several sub-directories within the openbmc-test-automation base which
92contain test suites, tools, templates, etc. These sub-directories are
93classified as follows:
94
95`tests/`: Contains the general test cases for OpenBMC stack functional
96          verification.
97
98`extended/`: Contains test cases for boot testing, network testing,
99             code update testing etc.
100
101`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing.
102
103`xcat/`: Contains test cases for XCAT automation.
104
105`gui/`: Contains test cases for web UI and security scanning tool automation.
106
107`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing.
108
109`network/`: Contains test cases for network testing. It covers IPv4 in static mode
110            and in DHCP mode.
111
112`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol)
113         configuration testing.
114
115`remote_logging`: Contains test cases for remote logging via rsyslog.
116
117`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and
118                  Serviceability) for an OpenPOWER system.
119
120`openpower/secureboot/`: Contains test cases for secure boot testing on a
121                         secure boot feature enabled OpenPOWER system only.
122
123`tools/`: Contains various tools.
124
125`templates/`: Contains sample code examples and setup testing.
126
127`test_list/`: Contains the argument files used for skipping test cases
128              (e.g "skip_test", "skip_test_extended", etc.) or
129              grouping them (e.g "HW_CI", "CT_basic_run", etc.).
130
131
132## Redfish Test Layout ##
133
134OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard
135specification and schema that meets the expectations of end users for simple,
136modern and secure management of scalable platform hardware.
137
138`redfish_test`: Contains test cases for DMTF Redfish-related feature supported
139                on OpenBMC.
140
141
142## Quickstart ##
143To run openbmc-automation first you need to install the prerequisite Python
144packages which will help to invoke tests through tox (Note that tox
145version 2.3.1 or greater is required) or via Robot CLI command.
146
147**Robot Command Line**
148
149* Execute all test suites for `tests/`:
150    ```
151    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
152    ```
153
154* Execute a test suite:
155    ```
156    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests/test_basic_poweron.robot
157    ```
158
159* Initialize the following environment variables which will be used during testing:
160    ```
161    $ export OPENBMC_HOST=<openbmc machine ip address>
162    $ export OPENBMC_PASSWORD=<openbmc password>
163    $ export OPENBMC_USERNAME=<openbmc username>
164    $ export OPENBMC_MODEL=[./data/Witherspoon.py, ./data/Palmetto.py, etc]
165    $ export IPMI_COMMAND=<Dbus/External>
166    $ export IPMI_PASSWORD=<External IPMI password>
167    ```
168
169* For QEMU tests, set the following environment variables as well:
170    ```
171    $ export SSH_PORT=<ssh port number>
172    $ export HTTPS_PORT=<https port number>
173    ```
174
175* Run tests:
176    ```
177    $ tox -e tests
178    ```
179
180* How to run individual test:
181
182    One specific test:
183    ```
184    $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
185    ```
186    No preset environment variables, default configuration for all supported
187    systems:
188    ```
189    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
190    ```
191    No preset environment variables, one test case from a test suite:
192    ```
193    $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
194    ```
195    No preset environment variables, the entire test suite:
196    ```
197    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
198    ```
199
200    No preset environment variables, the entire test suite excluding test
201    cases using argument file:
202    ```
203    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
204    ```
205
206    Exclude test list for supported systems:
207    ```
208    Palmetto:  test_lists/skip_test_palmetto
209    Witherspoon:  test_lists/skip_test_witherspoon
210    ```
211
212* Run GUI tests:
213
214    By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode:
215    ```
216    $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/obmc_gui/test/
217    ```
218
219    Run GUI default CI test bucket:
220    ```
221    $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/obmc_gui/test/
222    ```
223
224* Run LDAP tests:
225
226    Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server.
227    Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server.
228    One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd
229    For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch".
230    Microsoft ADS: refer to  https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS
231    ```
232    $ cd tests_ldap/
233    $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege>  test_user_ldap_management.robot
234    ```
235
236* How to run CI and CT bucket test:
237
238    Default CI test bucket list:
239    ```
240    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
241    ```
242
243    Default CI smoke test bucket list:
244    ```
245    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
246    ```
247
248* Run extended tests:
249
250    For-loop test (default iteration is 10):
251    ```
252    $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
253    ```
254
255    Example using tox testing a test suite for 5 iterations "witherspoon":
256    ```
257    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot
258    ```
259
260**Jenkins jobs tox commands**
261* HW CI tox command:
262    ```
263    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
264    ```
265
266## Code Update ##
267
268Currently supported BMC and PNOR update formats are UBI and non-UBI.
269For code update information, please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md)
270
271
272* UBI Format *
273
274    For BMC code update, download the system type *.ubi.mdt.tar image from
275    https://openpower.xyz/job/openbmc-build/ and run as follows:
276
277    For Witherspoon system:
278    ```
279    $ cd extended/code_update/
280    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update  bmc_code_update.robot
281    ```
282
283    For host code update, download the system type *.pnor.squashfs.tar image
284    from https://openpower.xyz/job/openpower-op-build/ and run as follows:
285
286    For Witherspoon system:
287    ```
288    $ cd extended/code_update/
289    $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update  host_code_update.robot
290    ```
291
292* Non-UBI Format *
293
294    For BMC code update, download the system type *all.tar image from
295    https://openpower.xyz/job/openbmc-build/ and run as follows:
296
297    For a Zaius system:
298    ```
299    $ cd extended/code_update/
300    $ robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:<image path>/zaius-<date time>.all.tar --include Initiate_Code_Update_BMC update_bmc.robot
301    ```
302
303    For host code update, download the system type *.pnor from
304    https://openpower.xyz/job/openpower-op-build/ and run as follows:
305
306    For a Zaius system:
307    ```
308    $ cd extended/
309    $ robot -v OPENBMC_HOST:x.x.x.x -v PNOR_IMAGE_PATH:<image path>/zaius.pnor test_bios_update.robot
310    ```
311