124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 524c9f88dSGeorge Keishing* Out-of-band IPMI 624c9f88dSGeorge Keishing* SSH to BMC and Host OS 724c9f88dSGeorge Keishing 824c9f88dSGeorge Keishing**Key Feature List** 924c9f88dSGeorge Keishing* Power on/off 1024c9f88dSGeorge Keishing* Reboot Host 1124c9f88dSGeorge Keishing* Reset BMC 1224c9f88dSGeorge Keishing* Code update BMC and host 1324c9f88dSGeorge Keishing* Power management 1424c9f88dSGeorge Keishing* HTX bootme 1524c9f88dSGeorge Keishing* XCAT execution 1624c9f88dSGeorge Keishing* Network 17a52159ffSGeorge Keishing* IPMI support 18a52159ffSGeorge Keishing* Factory reset 19a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2029814f0eSGeorge Keishing* Web UI testing 2124c9f88dSGeorge Keishing 2224c9f88dSGeorge Keishing**Debugging Supported List** 2324c9f88dSGeorge Keishing* SOL collection 2424c9f88dSGeorge Keishing* FFDC collection 2524c9f88dSGeorge Keishing* Error injection from host 2624c9f88dSGeorge Keishing 27a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 28a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 29b29d2e84SChris Austen 30a5c6e9d6SGeorge Keishing* Miscelleneous 31a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 32a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 33a5c6e9d6SGeorge Keishing 34a5c6e9d6SGeorge Keishing REST base packages: 35a5c6e9d6SGeorge Keishing ``` 36a5c6e9d6SGeorge Keishing $ pip install -U requests 37a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 38a5c6e9d6SGeorge Keishing ``` 39a5c6e9d6SGeorge Keishing SSH and SCP base packages: 40a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 41a5c6e9d6SGeorge Keishing ``` 42a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 43a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 44a5c6e9d6SGeorge Keishing ``` 452de157e4SGeorge Keishing 462de157e4SGeorge Keishing## OpenBMC Test Development ## 472de157e4SGeorge Keishing 482de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 492de157e4SGeorge Keishing 502de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 512de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 522de157e4SGeorge Keishing curl commands required for testing. 5329814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 54*fa6da2f1SGeorge Keishing - [Tools.md](Tools.md): Reference information for helper tools. 552de157e4SGeorge Keishing 563df030faSGeorge Keishing## Testing Setup Steps ## 573df030faSGeorge Keishing 583df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 593df030faSGeorge Keishing 603df030faSGeorge Keishing* Download the openbmc-test-automation repository: 613df030faSGeorge Keishing ``` 623df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 633df030faSGeorge Keishing $ cd openbmc-test-automation 643df030faSGeorge Keishing ``` 653df030faSGeorge Keishing* Execute basic setup test run: 663df030faSGeorge Keishing ``` 673df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 683df030faSGeorge Keishing ``` 693df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 70a5c6e9d6SGeorge Keishing 715977d2aeSGeorge Keishing## Test Layout ## 725977d2aeSGeorge Keishing 735977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 745977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 755977d2aeSGeorge Keishingclassified as follows: 765977d2aeSGeorge Keishing 775977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 785977d2aeSGeorge Keishing verification. 795977d2aeSGeorge Keishing 805977d2aeSGeorge Keishing`extended/`: Contains test cases for boot testing, network testing, 81a52159ffSGeorge Keishing code update testing etc. 825977d2aeSGeorge Keishing 835977d2aeSGeorge Keishing`systest/`: Contains test cases for HTX bootme. 845977d2aeSGeorge Keishing 855977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 865977d2aeSGeorge Keishing 87a52159ffSGeorge Keishing`ras/`: Contains test cases for RAS (Reliability, availability and 88a52159ffSGeorge Keishing serviceability). 89a52159ffSGeorge Keishing 9029814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 9129814f0eSGeorge Keishing 925977d2aeSGeorge Keishing`tools/`: Contains various tools. 935977d2aeSGeorge Keishing 945977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 955977d2aeSGeorge Keishing` 965977d2aeSGeorge Keishing 97a5c6e9d6SGeorge Keishing## Quickstart ## 985977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 995977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1005977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1015977d2aeSGeorge Keishing 1025977d2aeSGeorge Keishing**Robot Command Line** 1035977d2aeSGeorge Keishing 1045977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1055977d2aeSGeorge Keishing ``` 1065977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1075977d2aeSGeorge Keishing ``` 1085977d2aeSGeorge Keishing* Execute a test suite: 1095977d2aeSGeorge Keishing ``` 1105977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests/test_basic_poweron.robot 1115977d2aeSGeorge Keishing ``` 1125977d2aeSGeorge Keishing**Tox Command Line** 113b29d2e84SChris Austen 114a7c46e39SGeorge Keishing* Install the python dependencies for tox: 115a7c46e39SGeorge Keishing ``` 1164319d194SJay Azurin $ easy_install tox 117b29d2e84SChris Austen $ easy_install pip 118b29d2e84SChris Austen ``` 119b29d2e84SChris Austen 120a7c46e39SGeorge Keishing* Initialize the following environment variables which will be used during testing: 121a7c46e39SGeorge Keishing ``` 122b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 1237a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 1247a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 125b29d2e84SChris Austen $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc] 1267a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 1277a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 128359f8039SGeorge Keishing ``` 129a7c46e39SGeorge Keishing* For tests requiring PDU, set the following environment variables as well: 130a7c46e39SGeorge Keishing ``` 131b29d2e84SChris Austen $ export PDU_IP=<PDU IP address> 132b29d2e84SChris Austen $ export PDU_USERNAME=<PDU username> 133b29d2e84SChris Austen $ export PDU_PASSWORD=<PDU password> 134b29d2e84SChris Austen $ export PDU_TYPE=<PDU type> 135b29d2e84SChris Austen $ export PDU_SLOT_NO=<SLOT number> 136a7c46e39SGeorge Keishing ``` 137a7c46e39SGeorge Keishing Note: For PDU_TYPE we support only synaccess at the moment. 138b29d2e84SChris Austen 139a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 140a7c46e39SGeorge Keishing ``` 1414d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 1424d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 143a7c46e39SGeorge Keishing ``` 144a7c46e39SGeorge Keishing* For BIOS tests, set the following environment variables as well: 145a7c46e39SGeorge Keishing ``` 146e4c52eb2SJay Azurin $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor 147b29d2e84SChris Austen ``` 148b29d2e84SChris Austen 149a7c46e39SGeorge Keishing* Run tests: 150a7c46e39SGeorge Keishing ``` 151b29d2e84SChris Austen $ tox -e tests 152b29d2e84SChris Austen ``` 153b29d2e84SChris Austen 154a7c46e39SGeorge Keishing* How to run individual test: 15588acd8daSGeorge Keishing 156a7c46e39SGeorge Keishing One specific test: 157a7c46e39SGeorge Keishing ``` 158a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 159a7c46e39SGeorge Keishing ``` 160a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 161a7c46e39SGeorge Keishing systems: 162a7c46e39SGeorge Keishing ``` 16388acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 164a7c46e39SGeorge Keishing ``` 165a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 166a7c46e39SGeorge Keishing ``` 167a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 168a7c46e39SGeorge Keishing ``` 169a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 170a7c46e39SGeorge Keishing ``` 171a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 172a7c46e39SGeorge Keishing ``` 173a7c46e39SGeorge Keishing 17408fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 175a7c46e39SGeorge Keishing cases using argument file: 176a7c46e39SGeorge Keishing ``` 177a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 178a7c46e39SGeorge Keishing ``` 179ddf0cc2aSGeorge Keishing 180ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 181a7c46e39SGeorge Keishing ``` 182ddf0cc2aSGeorge Keishing Barrleye: test_lists/skip_test_barreleye 183ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 184ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 185b29d2e84SChris Austen ``` 186b29d2e84SChris Austen 187d7c26840SGeorge Keishing* How to run CI and CT bucket test: 188d7c26840SGeorge Keishing 189d7c26840SGeorge Keishing Default CI test bucket list: 190d7c26840SGeorge Keishing ``` 191d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 192d7c26840SGeorge Keishing ``` 193d7c26840SGeorge Keishing 194d7c26840SGeorge Keishing Default CI smoke test bucket list: 195d7c26840SGeorge Keishing ``` 196d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 197d7c26840SGeorge Keishing ``` 198d7c26840SGeorge Keishing 1997904eb37SGeorge Keishing* Code update test: 2007904eb37SGeorge Keishing 2017904eb37SGeorge Keishing For BMC code update, download the system type *.ubi.mdt.tar image from 2027904eb37SGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 2037904eb37SGeorge Keishing 2047904eb37SGeorge Keishing For Witherspoon system: 2057904eb37SGeorge Keishing ``` 2067904eb37SGeorge Keishing $ cd extended/code_update/ 2077904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update bmc_code_update.robot 2087904eb37SGeorge Keishing ``` 2097904eb37SGeorge Keishing 2107904eb37SGeorge Keishing For host code update, download the system type *.pnor.squashfs.tar image 2117904eb37SGeorge Keishing from https://openpower.xyz/job/openpower-op-build/ and run as follows: 2127904eb37SGeorge Keishing 2137904eb37SGeorge Keishing For Witherspoon system: 2147904eb37SGeorge Keishing ``` 2157904eb37SGeorge Keishing $ cd extended/code_update/ 2167904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update host_code_update.robot 2177904eb37SGeorge Keishing ``` 2187904eb37SGeorge Keishing 2197904eb37SGeorge Keishing For manual code update information please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md) 2207904eb37SGeorge Keishing 221a7c46e39SGeorge Keishing* Run extended tests: 222359f8039SGeorge Keishing 223a7c46e39SGeorge Keishing For loop test (default iteration is 10): 224a7c46e39SGeorge Keishing ``` 225a7c46e39SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 226a7c46e39SGeorge Keishing ``` 227a7c46e39SGeorge Keishing Example using tox testing a test suite for 5 times: 228a7c46e39SGeorge Keishing ``` 229594a6c9dSSivas SRR OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot 230359f8039SGeorge Keishing ``` 23197651c75SGeorge Keishing 232a7c46e39SGeorge Keishing**Jenkins jobs tox commands** 233a7c46e39SGeorge Keishing* HW CI tox command: 234a7c46e39SGeorge Keishing ``` 235a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 23697651c75SGeorge Keishing ``` 237