1## Features of OpenBMC Test Automation ##
2
3**Interface Feature List**
4* REST
5* DMTF Redfish
6* Out-of-band IPMI
7* SSH to BMC and Host OS
8
9**Key Feature List**
10* Power on/off
11* Reboot Host
12* Reset BMC
13* Code update BMC and host
14* Power management
15* Fan controller
16* HTX bootme
17* XCAT execution
18* Network
19* IPMI support (generic and DCMI compliant)
20* Factory reset
21* RAS (Reliability, availability and serviceability)
22* Web UI testing
23* Secure boot
24* SNMP (Simple Network Management Protocol)
25* Remote Logging via Rsyslog
26* LDAP (Lightweight Directory Access Protocol)
27* Certificate
28* Local User Management(Redfish/IPMI)
29* DateTime
30* Event Logging
31* PLDM (Platform Level Data Model) via pldmtool
32
33**Debugging Supported List**
34* SOL collection
35* FFDC collection
36* Error injection from host
37
38## Installation Setup Guide ##
39* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
40
41* Miscellaneous
42Packages required to be installed for OpenBmc Automation.
43Install the packages and it's dependencies via `pip`
44
45    REST base packages:
46    ```
47    $ pip install -U requests
48    $ pip install -U robotframework-requests
49    $ pip install -U robotframework-httplibrary
50    ```
51
52    Python redfish library packages:
53    For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library)
54    ```
55    $ pip install redfish
56    ```
57
58    SSH and SCP base packages:
59    For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
60    ```
61    $ pip install robotframework-sshlibrary
62    $ pip install robotframework-scplibrary
63    ```
64
65    Installing tox:
66    ```
67    $ pip install -U tox
68    ```
69
70If using Python 3.x, use the corresponding `pip3` to install packages.
71
72## OpenBMC Test Development ##
73
74These documents contain details on developing OpenBMC test code and debugging.
75
76 - [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information.
77 - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
78 - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
79   curl commands required for legacy REST testing.
80 - [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): Quick reference for some common
81   curl commands required for redfish testing.
82 - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
83 - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server
84
85## OpenBMC Test Documentation ##
86
87 - [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools.
88 - [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): Currently supported BMC and PNOR update.
89 - [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): Steps to create and install CA signed certificate.
90
91## Supported Systems Architecture ##
92
93  OpenBMC test infrastructure is proven capable of running on:
94  - POWER
95  - x86
96  systems running OpenBMC firmware stack.
97
98## Testing Setup Steps ##
99
100To verify the installation setup is completed and ready to execute.
101
102* Download the openbmc-test-automation repository:
103    ```
104    $ git clone https://github.com/openbmc/openbmc-test-automation
105    $ cd openbmc-test-automation
106    ```
107* Execute basic setup test run:
108    ```
109    $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
110    ```
111    where xx.xx.xx.xx is the BMC hostname or IP.
112
113## Test Layout ##
114
115There are several sub-directories within the openbmc-test-automation base which
116contain test suites, tools, templates, etc. These sub-directories are
117classified as follows:
118
119`tests/`: Contains the general test cases for OpenBMC stack functional
120          verification. The "tests" subdirectory uses legacy REST and will be
121          deprecated at some point and therefore no longer supported.
122
123`extended/`: Contains test cases for boot testing, code update testing using legacy REST, etc.
124
125`systest/`: Contains test cases for HTX bootme testing.
126
127`xcat/`: Contains test cases for XCAT automation.
128
129`gui/`: Contains test cases for web UI and security scanning tool automation.
130
131`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, platform, OEM).
132
133`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol)
134         configuration testing.
135
136`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and
137                  Serviceability) for an OpenPOWER system.
138
139`openpower/secureboot/`: Contains test cases for secure boot testing on a
140                         secure boot feature enabled OpenPOWER system only.
141
142`tools/`: Contains various tools.
143
144`templates/`: Contains sample code examples and setup testing.
145
146`test_list/`: Contains the argument files used for skipping test cases
147              (e.g "skip_test", "skip_test_extended", etc.) or
148              grouping them (e.g "HW_CI", "CT_basic_run", etc.).
149
150
151## Redfish Test Layout ##
152
153OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard
154specification and schema that meets the expectations of end users for simple,
155modern and secure management of scalable platform hardware.
156
157`redfish/`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC.
158
159`redfish/extended/`: Contains test cases for combined legacy REST and DMTF Redfish-related
160                     feature supported on OpenBMC.
161
162Note: Work in progress test development parameter `-v REDFISH_SUPPORT_TRANS_STATE:1` to force
163      the test suites to execute in redfish mode only.
164
165
166## Quickstart ##
167To run openbmc-automation first you need to install the prerequisite Python
168packages which will help to invoke tests through tox (Note that tox
169version 2.3.1 or greater is required) or via Robot CLI command.
170
171**Robot Command Line**
172
173* Execute all test suites for `tests/`:
174    ```
175    $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
176    ```
177
178* Execute a test suite:
179    ```
180    $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot
181    ```
182
183* Initialize the following test variables which will be used during test execution:
184
185    User can forward declare as environment variables:
186    ```
187    $ export OPENBMC_HOST=<openbmc machine IP address/hostname>
188    $ export OPENBMC_USERNAME=<openbmc username>
189    $ export OPENBMC_PASSWORD=<openbmc password>
190    $ export IPMI_COMMAND=<Dbus/External>
191    ```
192
193    or
194
195    User can input as robot variables as part of the CLI command:
196    ```
197    -v OPENBMC_HOST:<openbmc machine IP address/hostname>
198    -v OPENBMC_USERNAME:<openbmc username>
199    -v OPENBMC_PASSWORD:<openbmc password>
200    ```
201
202* For QEMU tests, set the following environment variables as well:
203    ```
204    $ export SSH_PORT=<ssh port number>
205    $ export HTTPS_PORT=<https port number>
206    ```
207
208* Run tests:
209    ```
210    $ tox tests
211    ```
212
213* How to run individual test:
214
215    One specific test:
216    ```
217    $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
218    ```
219    No preset environment variables, default configuration for all supported
220    systems:
221    ```
222    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
223    ```
224    No preset environment variables, one test case from a test suite:
225    ```
226    $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
227    ```
228    No preset environment variables, the entire test suite:
229    ```
230    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
231    ```
232
233    No preset environment variables, the entire test suite excluding test
234    cases using argument file:
235    ```
236    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
237    ```
238
239    Exclude test list for supported systems:
240    ```
241    Palmetto:  test_lists/skip_test_palmetto
242    Witherspoon:  test_lists/skip_test_witherspoon
243    ```
244
245
246* Run IPMI tests:
247
248    Running only out-of-band IPMI tests:
249    ```
250    $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/
251    ```
252
253    Running only inband IPMI tests:
254    ```
255    $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/
256    ```
257
258
259* Run GUI tests:
260
261    By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode:
262    ```
263    $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/
264    ```
265
266    Run GUI default CI test bucket:
267    ```
268    $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/
269    ```
270
271* Run LDAP tests:
272
273    Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server.
274    Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server.
275    One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd
276    For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch".
277    Microsoft ADS: refer to  https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS
278
279   Note: Currently, LDAP test automation for Redfish API is in progress. The format to invoke LDAP test is as follows:
280    ```
281    $ cd redfish/account_service/
282    $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege>  ./test_ldap_configuration.robot
283    ```
284
285* How to run CI and CT bucket test:
286
287    Default CI test bucket list:
288    ```
289    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
290    ```
291
292    Default CI smoke test bucket list:
293    ```
294    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
295    ```
296
297* Run extended tests:
298
299    For-loop test (default iteration is 10):
300    ```
301    $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
302    ```
303
304    Example using tox testing a test suite for 5 iterations "witherspoon":
305    ```
306    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot
307    ```
308
309* Host CPU architecture
310
311    By default openbmc-test-automation framework assumes that host CPU is based on the POWER architecture.
312    If your host CPU is x86 add `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an environment variable:
313    ```
314    $ export PLATFORM_ARCH_TYPE=x86
315    ```
316
317**Jenkins jobs tox commands**
318* HW CI tox command:
319    ```
320    $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
321    ```
322