1#openbmc-automation
2
3Quickstart
4----------
5
6To run openbmc-automation first you need to install the prerequisite python
7packages which will help to invoke tests through tox.  Note that tox
8version 2.3.1 or greater is required.
9
10Install the python dependencies for tox
11```shell
12    $ easy_install tox
13    $ easy_install pip
14```
15
16Initialize the following environment variable which will be used during testing
17```shell
18    $ export OPENBMC_HOST=<openbmc machine ip address>
19    $ export OPENBMC_PASSWORD=<openbmc password>
20    $ export OPENBMC_USERNAME=<openbmc username>
21    $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
22    $ export IPMI_COMMAND=<Dbus/External>
23    $ export IPMI_PASSWORD=<External IPMI password>
24```
25
26There are two different set of test suite existing based on the usage.
27The test suites are distinctly separated by directory as under
28    tests/
29    extended/
30
31`tests`: directory contains the general test cases
32
33`extended`: directory contains the use cases for new IP network testing, PDU,
34BIOS and BMC code update.
35
36```shell
37Use Following Variables for networking test cases
38===========================================================
39    $export NEW_BMC_IP=<openbmc machine ip address>
40    $export NEW_SUBNET_MASK=<openbmc new subnet mask>
41    $export NEW_GATEWAY=<openbmc new gateway>
42==========================================================
43
44    Use following parameters for PDU:
45    $ export PDU_IP=<PDU IP address>
46    $ export PDU_USERNAME=<PDU username>
47    $ export PDU_PASSWORD=<PDU password>
48    $ export PDU_TYPE=<PDU type>
49    $ export PDU_SLOT_NO=<SLOT number>
50
51    for PDU_TYPE we support only synaccess at the moment
52
53Use following variables for syslog test cases
54==========================================================
55    $ export SYSLOG_IP_ADDRESS=<remote syslog system ip>
56    $ export SYSLOG_PORT=<remote syslog system port>
57
58Use the following variables for Qemu test run
59==========================================================
60    $ export SSH_PORT=<ssh port number>
61    $ export HTTPS_PORT=<https port number>
62
63Use the following variables for BIOS update testing
64==========================================================
65    $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
66
67```
68
69Run tests
70```shell
71    $ tox -e tests
72```
73
74How to test individual test
75```shell
76    One specific test
77    $ tox -e default -- -t '"DIMM0 no fault"' tests/test_sensors.robot
78
79    No preset environment variables, default configuration for all supported systems.
80    $ OPENBMC_HOST=x.x.x.x tox -e default -- tests
81
82    No preset environment variables, one test case from one test suite
83    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot
84
85    No preset environment variables, one test suite  for a palmetto system
86    $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot
87
88    No preset environment variables, the entire test suite for a barreleye system
89    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests
90
91    Default CI test bucket list:
92    No preset environment variables, the entire test suite excluding test
93    cases using argument file.
94    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/skip_test tests
95
96    Exclude test list for supported systems:
97    Barrleye:  test_lists/skip_test_barreleye
98    Palmetto:  test_lists/skip_test_palmetto
99    Witherspoon:  test_lists/skip_test_witherspoon
100```
101
102It can also be run by passing variables from the cli...
103```shell
104    Run one test suite using using pybot
105    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot
106
107    Run entire test suite using using pybot
108    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests
109
110    Run entire test suite using external ipmitool
111    $  pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests
112```
113
114Run extended tests
115```shell
116    Set the preset environment variables, run test suite for a barreleye system
117    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot
118
119    Similarly for Network, PDU and update BIOS
120
121    For BMC code update, download the system type *.all.tar image from https://openpower.xyz
122    and run as follows:
123
124    For Barreleye system
125    python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar  extended/code_update/update_bmc.robot
126
127    For loop test (Default iteration is 10)
128    python -m robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot
129    Below is sample command using tox to test only fw version using Barreleye system for 5 times
130    OPENBMC_HOST=x.x.x.x  LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5  OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot
131```
132
133Jenkins jobs tox commands
134```shell
135    HW CI tox command
136    Set the preset environment variables, run HW CI test for a barreleye system
137    $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/HW_CI tests
138
139```
140
141Template to be used to create new stand-alone python programs.
142```shell
143
144If you wish to create a new python stand-alone program, please copy bin/python_pgm_template to your new name and then begin your work.  Example:
145
146cd bin
147cp python_pgm_template my_new_program
148
149This template has much of your preliminary work done for you and it will help us all follow a similar structure.
150
151Features:
152- Help text and argparsing started for you.
153- Support for "stock" parameters like quiet, debug, test_mode.
154- exit_function and signal_handler defined.
155- validate_parms function pre-created.
156- main function follows conventional startup:
157
158    if not gen_get_options(parser, stock_list):
159        return False
160
161    if not validate_parms():
162        return False
163
164    qprint_pgm_header()
165
166    # Your code here.
167
168```
169
170Command to get GitHub issues (any GitHub repository) report in CSV format
171Note: On Prompt "Enter your GitHub Password:" enter your GitHub password.
172```shell
173python ./tools/github_issues_to_csv <github user> <github repo>
174
175Example for getting openbmc issues
176python ./tools/github_issues_to_csv <github user>  openbmc/openbmc
177
178Example for getting openbmc-test-automation issues
179python ./tools/github_issues_to_csv <github user>  openbmc/openbmc-test-automation
180```
181
182
183Command to generate Robot test cases test documentations
184
185```shell
186./tools/generate_test_document <Robot test cases directory path> <test case document file path>
187
188Example for generating tests cases documentation for tests directory
189./tools/generate_test_document tests testsdirectoryTCdocs.html
190
191Example for generating tests cases documentation (tests,gui,extended TCs)
192# Note: Invoke the tool with out argument
193./tools/generate_test_document
194```
195