124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 524c9f88dSGeorge Keishing* Out-of-band IPMI 624c9f88dSGeorge Keishing* SSH to BMC and Host OS 724c9f88dSGeorge Keishing 824c9f88dSGeorge Keishing**Key Feature List** 924c9f88dSGeorge Keishing* Power on/off 1024c9f88dSGeorge Keishing* Reboot Host 1124c9f88dSGeorge Keishing* Reset BMC 1224c9f88dSGeorge Keishing* Code update BMC and host 1324c9f88dSGeorge Keishing* Power management 14dd9471e0SGeorge Keishing* Fan controller 1524c9f88dSGeorge Keishing* HTX bootme 1624c9f88dSGeorge Keishing* XCAT execution 1724c9f88dSGeorge Keishing* Network 18dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 19a52159ffSGeorge Keishing* Factory reset 20a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2129814f0eSGeorge Keishing* Web UI testing 228b4f4034SGeorge Keishing* IO storage and EEH (Enhanced Error Handling) testing 2338f566bbSGeorge Keishing* Secure boot 24*0465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 2524c9f88dSGeorge Keishing 2624c9f88dSGeorge Keishing**Debugging Supported List** 2724c9f88dSGeorge Keishing* SOL collection 2824c9f88dSGeorge Keishing* FFDC collection 2924c9f88dSGeorge Keishing* Error injection from host 3024c9f88dSGeorge Keishing 31a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 32a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 33b29d2e84SChris Austen 34917ba1a2SGunnar Mills* Miscellaneous 35a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 36a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 37a5c6e9d6SGeorge Keishing 38a5c6e9d6SGeorge Keishing REST base packages: 39a5c6e9d6SGeorge Keishing ``` 40a5c6e9d6SGeorge Keishing $ pip install -U requests 41a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 42a5c6e9d6SGeorge Keishing ``` 43a5c6e9d6SGeorge Keishing SSH and SCP base packages: 44a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 45a5c6e9d6SGeorge Keishing ``` 46a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 47a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 48a5c6e9d6SGeorge Keishing ``` 492de157e4SGeorge Keishing 502de157e4SGeorge Keishing## OpenBMC Test Development ## 512de157e4SGeorge Keishing 522de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 532de157e4SGeorge Keishing 540b9bd82cSGeorge Keishing - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information. 552de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 562de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 572de157e4SGeorge Keishing curl commands required for testing. 5829814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 59fa6da2f1SGeorge Keishing - [Tools.md](Tools.md): Reference information for helper tools. 602de157e4SGeorge Keishing 613df030faSGeorge Keishing## Testing Setup Steps ## 623df030faSGeorge Keishing 633df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 643df030faSGeorge Keishing 653df030faSGeorge Keishing* Download the openbmc-test-automation repository: 663df030faSGeorge Keishing ``` 673df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 683df030faSGeorge Keishing $ cd openbmc-test-automation 693df030faSGeorge Keishing ``` 703df030faSGeorge Keishing* Execute basic setup test run: 713df030faSGeorge Keishing ``` 723df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 733df030faSGeorge Keishing ``` 743df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 75a5c6e9d6SGeorge Keishing 765977d2aeSGeorge Keishing## Test Layout ## 775977d2aeSGeorge Keishing 785977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 795977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 805977d2aeSGeorge Keishingclassified as follows: 815977d2aeSGeorge Keishing 825977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 835977d2aeSGeorge Keishing verification. 845977d2aeSGeorge Keishing 855977d2aeSGeorge Keishing`extended/`: Contains test cases for boot testing, network testing, 86a52159ffSGeorge Keishing code update testing etc. 875977d2aeSGeorge Keishing 888b4f4034SGeorge Keishing`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing. 895977d2aeSGeorge Keishing 905977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 915977d2aeSGeorge Keishing 9229814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 9329814f0eSGeorge Keishing 94dd9471e0SGeorge Keishing`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing. 95dd9471e0SGeorge Keishing 96df62e5c4SGeorge Keishing`network/`: Contains test cases for network testing. 97df62e5c4SGeorge Keishing 98*0465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 99*0465b9cdSGeorge Keishing configuration testing. 100*0465b9cdSGeorge Keishing 101*0465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 102*0465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 103*0465b9cdSGeorge Keishing 104*0465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 105*0465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 10638f566bbSGeorge Keishing 1075977d2aeSGeorge Keishing`tools/`: Contains various tools. 1085977d2aeSGeorge Keishing 1095977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 1105977d2aeSGeorge Keishing` 1115977d2aeSGeorge Keishing 112a5c6e9d6SGeorge Keishing## Quickstart ## 1135977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1145977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1155977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1165977d2aeSGeorge Keishing 1175977d2aeSGeorge Keishing**Robot Command Line** 1185977d2aeSGeorge Keishing 1195977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1205977d2aeSGeorge Keishing ``` 1215977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1225977d2aeSGeorge Keishing ``` 1235977d2aeSGeorge Keishing* Execute a test suite: 1245977d2aeSGeorge Keishing ``` 1255977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests/test_basic_poweron.robot 1265977d2aeSGeorge Keishing ``` 1275977d2aeSGeorge Keishing**Tox Command Line** 128b29d2e84SChris Austen 129a7c46e39SGeorge Keishing* Install the python dependencies for tox: 130a7c46e39SGeorge Keishing ``` 1314319d194SJay Azurin $ easy_install tox 132b29d2e84SChris Austen $ easy_install pip 133b29d2e84SChris Austen ``` 134b29d2e84SChris Austen 135a7c46e39SGeorge Keishing* Initialize the following environment variables which will be used during testing: 136a7c46e39SGeorge Keishing ``` 137b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 1387a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 1397a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 140b29d2e84SChris Austen $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc] 1417a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 1427a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 143359f8039SGeorge Keishing ``` 144a7c46e39SGeorge Keishing* For tests requiring PDU, set the following environment variables as well: 145a7c46e39SGeorge Keishing ``` 146b29d2e84SChris Austen $ export PDU_IP=<PDU IP address> 147b29d2e84SChris Austen $ export PDU_USERNAME=<PDU username> 148b29d2e84SChris Austen $ export PDU_PASSWORD=<PDU password> 149b29d2e84SChris Austen $ export PDU_TYPE=<PDU type> 150b29d2e84SChris Austen $ export PDU_SLOT_NO=<SLOT number> 151a7c46e39SGeorge Keishing ``` 152a7c46e39SGeorge Keishing Note: For PDU_TYPE we support only synaccess at the moment. 153b29d2e84SChris Austen 154a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 155a7c46e39SGeorge Keishing ``` 1564d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 1574d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 158a7c46e39SGeorge Keishing ``` 159a7c46e39SGeorge Keishing* For BIOS tests, set the following environment variables as well: 160a7c46e39SGeorge Keishing ``` 161e4c52eb2SJay Azurin $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor 162b29d2e84SChris Austen ``` 163b29d2e84SChris Austen 164a7c46e39SGeorge Keishing* Run tests: 165a7c46e39SGeorge Keishing ``` 166b29d2e84SChris Austen $ tox -e tests 167b29d2e84SChris Austen ``` 168b29d2e84SChris Austen 169a7c46e39SGeorge Keishing* How to run individual test: 17088acd8daSGeorge Keishing 171a7c46e39SGeorge Keishing One specific test: 172a7c46e39SGeorge Keishing ``` 173a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 174a7c46e39SGeorge Keishing ``` 175a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 176a7c46e39SGeorge Keishing systems: 177a7c46e39SGeorge Keishing ``` 17888acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 179a7c46e39SGeorge Keishing ``` 180a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 181a7c46e39SGeorge Keishing ``` 182a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 183a7c46e39SGeorge Keishing ``` 184a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 185a7c46e39SGeorge Keishing ``` 186a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 187a7c46e39SGeorge Keishing ``` 188a7c46e39SGeorge Keishing 18908fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 190a7c46e39SGeorge Keishing cases using argument file: 191a7c46e39SGeorge Keishing ``` 192a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 193a7c46e39SGeorge Keishing ``` 194ddf0cc2aSGeorge Keishing 195ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 196a7c46e39SGeorge Keishing ``` 197ddf0cc2aSGeorge Keishing Barrleye: test_lists/skip_test_barreleye 198ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 199ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 200b29d2e84SChris Austen ``` 201b29d2e84SChris Austen 202d7c26840SGeorge Keishing* How to run CI and CT bucket test: 203d7c26840SGeorge Keishing 204d7c26840SGeorge Keishing Default CI test bucket list: 205d7c26840SGeorge Keishing ``` 206d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 207d7c26840SGeorge Keishing ``` 208d7c26840SGeorge Keishing 209d7c26840SGeorge Keishing Default CI smoke test bucket list: 210d7c26840SGeorge Keishing ``` 211d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 212d7c26840SGeorge Keishing ``` 213d7c26840SGeorge Keishing 2147904eb37SGeorge Keishing* Code update test: 2157904eb37SGeorge Keishing 2167904eb37SGeorge Keishing For BMC code update, download the system type *.ubi.mdt.tar image from 2177904eb37SGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 2187904eb37SGeorge Keishing 2197904eb37SGeorge Keishing For Witherspoon system: 2207904eb37SGeorge Keishing ``` 2217904eb37SGeorge Keishing $ cd extended/code_update/ 2227904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update bmc_code_update.robot 2237904eb37SGeorge Keishing ``` 2247904eb37SGeorge Keishing 2257904eb37SGeorge Keishing For host code update, download the system type *.pnor.squashfs.tar image 2267904eb37SGeorge Keishing from https://openpower.xyz/job/openpower-op-build/ and run as follows: 2277904eb37SGeorge Keishing 2287904eb37SGeorge Keishing For Witherspoon system: 2297904eb37SGeorge Keishing ``` 2307904eb37SGeorge Keishing $ cd extended/code_update/ 2317904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update host_code_update.robot 2327904eb37SGeorge Keishing ``` 2337904eb37SGeorge Keishing 2347904eb37SGeorge Keishing For manual code update information please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md) 2357904eb37SGeorge Keishing 236a7c46e39SGeorge Keishing* Run extended tests: 237359f8039SGeorge Keishing 238a7c46e39SGeorge Keishing For loop test (default iteration is 10): 239a7c46e39SGeorge Keishing ``` 240a7c46e39SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 241a7c46e39SGeorge Keishing ``` 242a7c46e39SGeorge Keishing Example using tox testing a test suite for 5 times: 243a7c46e39SGeorge Keishing ``` 244594a6c9dSSivas SRR OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot 245359f8039SGeorge Keishing ``` 24697651c75SGeorge Keishing 247a7c46e39SGeorge Keishing**Jenkins jobs tox commands** 248a7c46e39SGeorge Keishing* HW CI tox command: 249a7c46e39SGeorge Keishing ``` 250a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 25197651c75SGeorge Keishing ``` 252