xref: /openbmc/openbmc-test-automation/README.md (revision 438fd3b6c69c11da78a5b01123a0b4ce50e7face)
1 ## Features of OpenBMC Test Automation ##
2 
3 **Interface Feature List**
4 * REST
5 * DMTF Redfish
6 * Out-of-band IPMI
7 * SSH to BMC and Host OS
8 
9 **Key Feature List**
10 * Power on/off
11 * Reboot Host
12 * Reset BMC
13 * Code update BMC and host
14 * Power management
15 * Fan controller
16 * HTX bootme
17 * XCAT execution
18 * Network
19 * IPMI support (generic and DCMI compliant)
20 * Factory reset
21 * RAS (Reliability, availability and serviceability)
22 * Web UI testing
23 * Secure boot
24 * SNMP (Simple Network Management Protocol)
25 * Remote Logging via Rsyslog
26 * LDAP (Lightweight Directory Access Protocol)
27 * Certificate
28 * Local User Management(Redfish/IPMI)
29 * DateTime
30 * Event Logging
31 * PLDM (Platform Level Data Model) via pldmtool
32 
33 **Debugging Supported List**
34 * SOL collection
35 * FFDC collection
36 * Error injection from host
37 
38 ## Installation Setup Guide ##
39 * [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst)
40 
41 * Miscellaneous
42 Packages required to be installed for OpenBmc Automation.
43 Install the packages and it's dependencies via `pip`
44 
45 If using Python 3.x, use the corresponding `pip3` to install packages.
46 Note: Older Python 2.x is not actively supported.
47 
48 REST base packages:
49 ```
50     $ pip install -U requests
51     $ pip install -U robotframework-requests
52     $ pip install -U robotframework-httplibrary
53 ```
54 
55 Python redfish library packages:
56 For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library)
57 ```
58     $ pip install redfish
59 ```
60 
61 SSH and SCP base packages:
62 For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary)
63 ```
64     $ pip install robotframework-sshlibrary
65     $ pip install robotframework-scplibrary
66 ```
67 
68 Installing requirement dependencies:
69 ```
70     $ pip install -r requirements.txt
71 ```
72 you'll find this file once your clone openbmc-test-automation repository.
73 
74 Installing tox:
75 ```
76     $ pip install -U tox
77 ```
78 
79 Installing expect:
80 ```
81     $ sudo apt-get install expect (Ubuntu example)
82 ```
83 
84 ## OpenBMC Test Development ##
85 
86 These documents contain details on developing OpenBMC test code and debugging.
87 
88  - [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information.
89  - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines.
90  - [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): To check common code misspellings, syntax and standard checks.
91  - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common
92    curl commands required for legacy REST testing.
93  - [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): Quick reference for some common
94    curl commands required for redfish testing.
95  - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference.
96  - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server
97 
98 ## OpenBMC Test Documentation ##
99 
100  - [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools.
101  - [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): Currently supported BMC and PNOR update.
102  - [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): Steps to create and install CA signed certificate.
103 
104 ## Supported Systems Architecture ##
105 
106   OpenBMC test infrastructure is proven capable of running on:
107   - POWER
108   - x86
109   systems running OpenBMC firmware stack.
110 
111 ## Testing Setup Steps ##
112 
113 To verify the installation setup is completed and ready to execute.
114 
115 * Download the openbmc-test-automation repository:
116     ```
117     $ git clone https://github.com/openbmc/openbmc-test-automation
118     $ cd openbmc-test-automation
119     ```
120 * Execute basic setup test run:
121     ```
122     $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot
123     ```
124     where xx.xx.xx.xx is the BMC hostname or IP.
125 
126 ## Test Layout ##
127 
128 There are several sub-directories within the openbmc-test-automation base which
129 contain test suites, tools, templates, etc. These sub-directories are
130 classified as follows:
131 
132 `tests/`: Contains the general test cases for OpenBMC stack functional
133           verification. The "tests" subdirectory uses legacy REST and will be
134           deprecated at some point and therefore no longer supported.
135 
136 `extended/`: Contains test cases for boot testing, code update testing using legacy REST, etc.
137 
138 `systest/`: Contains test cases for HTX bootme testing.
139 
140 `xcat/`: Contains test cases for XCAT automation.
141 
142 `gui/`: Contains test cases for testing web-based interface built on AngularJS.
143 
144 `gui/gui_test/`: Contains test cases for testing web-based user interface built on Vue.js.
145 
146 `pldm/`: Contains test cases for platform management subsystem (base, bios, fru, platform, OEM).
147 
148 `snmp/`: Contains test cases for SNMP (Simple Network Management Protocol)
149          configuration testing.
150 
151 `openpower/ras/`: Contains test cases for RAS (Reliability, Availability and
152                   Serviceability) for an OpenPOWER system.
153 
154 `openpower/secureboot/`: Contains test cases for secure boot testing on a
155                          secure boot feature enabled OpenPOWER system only.
156 
157 `tools/`: Contains various tools.
158 
159 `templates/`: Contains sample code examples and setup testing.
160 
161 `test_list/`: Contains the argument files used for skipping test cases
162               (e.g "skip_test", "skip_test_extended", etc.) or
163               grouping them (e.g "HW_CI", "CT_basic_run", etc.).
164 
165 
166 ## Redfish Test Layout ##
167 
168 OpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard
169 specification and schema that meets the expectations of end users for simple,
170 modern and secure management of scalable platform hardware.
171 
172 `redfish/`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC.
173 
174 `redfish/extended/`: Contains test cases for combined legacy REST and DMTF Redfish-related
175                      feature supported on OpenBMC.
176 
177 Note: Work in progress test development parameter `-v REDFISH_SUPPORT_TRANS_STATE:1` to force
178       the test suites to execute in redfish mode only.
179 
180 
181 ## Quickstart ##
182 To run openbmc-automation first you need to install the prerequisite Python
183 packages which will help to invoke tests through tox (Note that tox
184 version 2.3.1 or greater is required) or via Robot CLI command.
185 
186 **Robot Command Line**
187 
188 * Execute all test suites for `tests/`:
189     ```
190     $ robot -v OPENBMC_HOST:xx.xx.xx.xx  tests
191     ```
192 
193 * Execute a test suite:
194     ```
195     $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot
196     ```
197 
198 * Initialize the following test variables which will be used during test execution:
199 
200     User can forward declare as environment variables:
201     ```
202     $ export OPENBMC_HOST=<openbmc machine IP address/hostname>
203     $ export OPENBMC_USERNAME=<openbmc username>
204     $ export OPENBMC_PASSWORD=<openbmc password>
205     $ export IPMI_COMMAND=<Dbus/External>
206     ```
207 
208     or
209 
210     User can input as robot variables as part of the CLI command:
211     ```
212     -v OPENBMC_HOST:<openbmc machine IP address/hostname>
213     -v OPENBMC_USERNAME:<openbmc username>
214     -v OPENBMC_PASSWORD:<openbmc password>
215     ```
216 
217 * For QEMU tests, set the following environment variables as well:
218     ```
219     $ export SSH_PORT=<ssh port number>
220     $ export HTTPS_PORT=<https port number>
221     ```
222 
223 * Run tests:
224     ```
225     $ tox tests
226     ```
227 
228 * How to run individual test:
229 
230     One specific test:
231     ```
232     $ tox -e default -- --include Power_On_Test  tests/test_basic_poweron.robot
233     ```
234     No preset environment variables, default configuration for all supported
235     systems:
236     ```
237     $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
238     ```
239     No preset environment variables, one test case from a test suite:
240     ```
241     $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot
242     ```
243     No preset environment variables, the entire test suite:
244     ```
245     $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
246     ```
247 
248     No preset environment variables, the entire test suite excluding test
249     cases using argument file:
250     ```
251     $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests
252     ```
253 
254     Exclude test list for supported systems:
255     ```
256     Palmetto:  test_lists/skip_test_palmetto
257     Witherspoon:  test_lists/skip_test_witherspoon
258     ```
259 
260 
261 * Run IPMI tests:
262 
263     Running only out-of-band IPMI tests:
264     ```
265     $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/
266     ```
267 
268     Running only inband IPMI tests:
269     ```
270     $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/
271     ```
272 
273 
274 * Run GUI tests:
275 
276     By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode:
277     ```
278     $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/
279     ```
280 
281     Run GUI default CI test bucket:
282     ```
283     $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/
284     ```
285 
286 * Run LDAP tests:
287 
288     Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server.
289     Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server.
290     One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd
291     For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch".
292     Microsoft ADS: refer to  https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS
293 
294    Note: Currently, LDAP test automation for Redfish API is in progress. The format to invoke LDAP test is as follows:
295     ```
296     $ cd redfish/account_service/
297     $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege>  ./test_ldap_configuration.robot
298     ```
299 
300 * How to run CI and CT bucket test:
301 
302     Default CI test bucket list:
303     ```
304     $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
305     ```
306 
307     Default CI smoke test bucket list:
308     ```
309     $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests
310     ```
311 
312 * Run extended tests:
313 
314     For-loop test (default iteration is 10):
315     ```
316     $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
317     ```
318 
319     Example using tox testing a test suite for 5 iterations "witherspoon":
320     ```
321     OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot
322     ```
323 
324 * Host CPU architecture
325 
326     By default openbmc-test-automation framework assumes that host CPU is based on the POWER architecture.
327     If your host CPU is x86 add `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an environment variable:
328     ```
329     $ export PLATFORM_ARCH_TYPE=x86
330     ```
331 
332 **Jenkins jobs tox commands**
333 * HW CI tox command:
334     ```
335     $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests
336     ```
337