124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 5acb3c1efSGeorge Keishing* DMTF Redfish 624c9f88dSGeorge Keishing* Out-of-band IPMI 724c9f88dSGeorge Keishing* SSH to BMC and Host OS 824c9f88dSGeorge Keishing 924c9f88dSGeorge Keishing**Key Feature List** 1024c9f88dSGeorge Keishing* Power on/off 1124c9f88dSGeorge Keishing* Reboot Host 1224c9f88dSGeorge Keishing* Reset BMC 1324c9f88dSGeorge Keishing* Code update BMC and host 1424c9f88dSGeorge Keishing* Power management 15dd9471e0SGeorge Keishing* Fan controller 1624c9f88dSGeorge Keishing* HTX bootme 1724c9f88dSGeorge Keishing* XCAT execution 1824c9f88dSGeorge Keishing* Network 19dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 20a52159ffSGeorge Keishing* Factory reset 21a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2229814f0eSGeorge Keishing* Web UI testing 238b4f4034SGeorge Keishing* IO storage and EEH (Enhanced Error Handling) testing 2438f566bbSGeorge Keishing* Secure boot 250465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 262c8c33fdSGeorge Keishing* Remote Logging via Rsyslog 2724c9f88dSGeorge Keishing 2824c9f88dSGeorge Keishing**Debugging Supported List** 2924c9f88dSGeorge Keishing* SOL collection 3024c9f88dSGeorge Keishing* FFDC collection 3124c9f88dSGeorge Keishing* Error injection from host 3224c9f88dSGeorge Keishing 33a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 34a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 35b29d2e84SChris Austen 36917ba1a2SGunnar Mills* Miscellaneous 37a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 38a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 39a5c6e9d6SGeorge Keishing 40a5c6e9d6SGeorge Keishing REST base packages: 41a5c6e9d6SGeorge Keishing ``` 42a5c6e9d6SGeorge Keishing $ pip install -U requests 43a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 441da95695SGeorge Keishing $ pip install -U robotframework-httplibrary 45a5c6e9d6SGeorge Keishing ``` 46aba958dcSGeorge Keishing 47a5c6e9d6SGeorge Keishing SSH and SCP base packages: 48a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 49a5c6e9d6SGeorge Keishing ``` 50a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 51a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 52a5c6e9d6SGeorge Keishing ``` 532de157e4SGeorge Keishing 54aba958dcSGeorge Keishing Installing tox: 55aba958dcSGeorge Keishing ``` 56aba958dcSGeorge Keishing $ pip install -U tox 57aba958dcSGeorge Keishing ``` 58aba958dcSGeorge Keishing 59aba958dcSGeorge KeishingIf using Python 3.x, use the corresponding `pip3` to install packages. 60aba958dcSGeorge Keishing 612de157e4SGeorge Keishing## OpenBMC Test Development ## 622de157e4SGeorge Keishing 632de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 642de157e4SGeorge Keishing 650b9bd82cSGeorge Keishing - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information. 662de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 672de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 682de157e4SGeorge Keishing curl commands required for testing. 6929814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 70fa6da2f1SGeorge Keishing - [Tools.md](Tools.md): Reference information for helper tools. 712de157e4SGeorge Keishing 723df030faSGeorge Keishing## Testing Setup Steps ## 733df030faSGeorge Keishing 743df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 753df030faSGeorge Keishing 763df030faSGeorge Keishing* Download the openbmc-test-automation repository: 773df030faSGeorge Keishing ``` 783df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 793df030faSGeorge Keishing $ cd openbmc-test-automation 803df030faSGeorge Keishing ``` 813df030faSGeorge Keishing* Execute basic setup test run: 823df030faSGeorge Keishing ``` 833df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 843df030faSGeorge Keishing ``` 853df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 86a5c6e9d6SGeorge Keishing 875977d2aeSGeorge Keishing## Test Layout ## 885977d2aeSGeorge Keishing 895977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 905977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 915977d2aeSGeorge Keishingclassified as follows: 925977d2aeSGeorge Keishing 935977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 945977d2aeSGeorge Keishing verification. 955977d2aeSGeorge Keishing 965977d2aeSGeorge Keishing`extended/`: Contains test cases for boot testing, network testing, 97a52159ffSGeorge Keishing code update testing etc. 985977d2aeSGeorge Keishing 998b4f4034SGeorge Keishing`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing. 1005977d2aeSGeorge Keishing 1015977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 1025977d2aeSGeorge Keishing 10329814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 10429814f0eSGeorge Keishing 105dd9471e0SGeorge Keishing`mnfg/`: Contains test cases for factory reset (DHCP mode) and PGOOD testing. 106dd9471e0SGeorge Keishing 107*39b8e96eSPrashanth Katti`network/`: Contains test cases for network testing. It covers IPv4 in static mode 108*39b8e96eSPrashanth Katti and in DHCP mode. 109df62e5c4SGeorge Keishing 1100465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 1110465b9cdSGeorge Keishing configuration testing. 1120465b9cdSGeorge Keishing 1132c8c33fdSGeorge Keishing`remote_logging`: Contains test cases for remote logging via rsyslog. 1142c8c33fdSGeorge Keishing 1150465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 1160465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 1170465b9cdSGeorge Keishing 1180465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 1190465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 12038f566bbSGeorge Keishing 1215977d2aeSGeorge Keishing`tools/`: Contains various tools. 1225977d2aeSGeorge Keishing 1235977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 124acb3c1efSGeorge Keishing 125aba958dcSGeorge Keishing`test_list/`: Contains the argument files used for skipping test cases 126aba958dcSGeorge Keishing (e.g "skip_test", "skip_test_extended", etc.) or 127aba958dcSGeorge Keishing grouping them (e.g "HW_CI", "CT_basic_run", etc.). 128aba958dcSGeorge Keishing 129acb3c1efSGeorge Keishing 130acb3c1efSGeorge Keishing## Redfish Test Layout ## 131acb3c1efSGeorge Keishing 132acb3c1efSGeorge KeishingOpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 133acb3c1efSGeorge Keishingspecification and schema that meets the expectations of end users for simple, 134acb3c1efSGeorge Keishingmodern and secure management of scalable platform hardware. 135acb3c1efSGeorge Keishing 136acb3c1efSGeorge Keishing`redfish_test`: Contains test cases for DMTF Redfish-related feature supported 137acb3c1efSGeorge Keishing on OpenBMC. 138acb3c1efSGeorge Keishing 1395977d2aeSGeorge Keishing 140a5c6e9d6SGeorge Keishing## Quickstart ## 1415977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1425977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1435977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1445977d2aeSGeorge Keishing 1455977d2aeSGeorge Keishing**Robot Command Line** 1465977d2aeSGeorge Keishing 1475977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1485977d2aeSGeorge Keishing ``` 1495977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1505977d2aeSGeorge Keishing ``` 151aba958dcSGeorge Keishing 1525977d2aeSGeorge Keishing* Execute a test suite: 1535977d2aeSGeorge Keishing ``` 1545977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests/test_basic_poweron.robot 1555977d2aeSGeorge Keishing ``` 156b29d2e84SChris Austen 157a7c46e39SGeorge Keishing* Initialize the following environment variables which will be used during testing: 158a7c46e39SGeorge Keishing ``` 159b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 1607a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 1617a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 162aba958dcSGeorge Keishing $ export OPENBMC_MODEL=[./data/Witherspoon.py, ./data/Palmetto.py, etc] 1637a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 1647a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 165359f8039SGeorge Keishing ``` 166b29d2e84SChris Austen 167a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 168a7c46e39SGeorge Keishing ``` 1694d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 1704d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 171a7c46e39SGeorge Keishing ``` 172b29d2e84SChris Austen 173a7c46e39SGeorge Keishing* Run tests: 174a7c46e39SGeorge Keishing ``` 175b29d2e84SChris Austen $ tox -e tests 176b29d2e84SChris Austen ``` 177b29d2e84SChris Austen 178a7c46e39SGeorge Keishing* How to run individual test: 17988acd8daSGeorge Keishing 180a7c46e39SGeorge Keishing One specific test: 181a7c46e39SGeorge Keishing ``` 182a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 183a7c46e39SGeorge Keishing ``` 184a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 185a7c46e39SGeorge Keishing systems: 186a7c46e39SGeorge Keishing ``` 18788acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 188a7c46e39SGeorge Keishing ``` 189a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 190a7c46e39SGeorge Keishing ``` 191a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 192a7c46e39SGeorge Keishing ``` 193a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 194a7c46e39SGeorge Keishing ``` 195a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 196a7c46e39SGeorge Keishing ``` 197a7c46e39SGeorge Keishing 19808fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 199a7c46e39SGeorge Keishing cases using argument file: 200a7c46e39SGeorge Keishing ``` 201a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 202a7c46e39SGeorge Keishing ``` 203ddf0cc2aSGeorge Keishing 204ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 205a7c46e39SGeorge Keishing ``` 206ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 207ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 208b29d2e84SChris Austen ``` 209b29d2e84SChris Austen 210d7c26840SGeorge Keishing* How to run CI and CT bucket test: 211d7c26840SGeorge Keishing 212d7c26840SGeorge Keishing Default CI test bucket list: 213d7c26840SGeorge Keishing ``` 214d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 215d7c26840SGeorge Keishing ``` 216d7c26840SGeorge Keishing 217d7c26840SGeorge Keishing Default CI smoke test bucket list: 218d7c26840SGeorge Keishing ``` 219d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 220d7c26840SGeorge Keishing ``` 221d7c26840SGeorge Keishing 222aba958dcSGeorge Keishing* Run extended tests: 223aba958dcSGeorge Keishing 224aba958dcSGeorge Keishing For-loop test (default iteration is 10): 225aba958dcSGeorge Keishing ``` 226aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 227aba958dcSGeorge Keishing ``` 228aba958dcSGeorge Keishing 229aba958dcSGeorge Keishing Example using tox testing a test suite for 5 iterations "witherspoon": 230aba958dcSGeorge Keishing ``` 231aba958dcSGeorge Keishing OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 232aba958dcSGeorge Keishing ``` 233aba958dcSGeorge Keishing 234aba958dcSGeorge Keishing**Jenkins jobs tox commands** 235aba958dcSGeorge Keishing* HW CI tox command: 236aba958dcSGeorge Keishing ``` 237aba958dcSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 238aba958dcSGeorge Keishing ``` 239aba958dcSGeorge Keishing 240aba958dcSGeorge Keishing## Code Update ## 241aba958dcSGeorge Keishing 242aba958dcSGeorge KeishingCurrently supported BMC and PNOR update formats are UBI and non-UBI. 243aba958dcSGeorge KeishingFor code update information, please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md) 244aba958dcSGeorge Keishing 245aba958dcSGeorge Keishing 246aba958dcSGeorge Keishing* UBI Format * 2477904eb37SGeorge Keishing 2487904eb37SGeorge Keishing For BMC code update, download the system type *.ubi.mdt.tar image from 2497904eb37SGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 2507904eb37SGeorge Keishing 2517904eb37SGeorge Keishing For Witherspoon system: 2527904eb37SGeorge Keishing ``` 2537904eb37SGeorge Keishing $ cd extended/code_update/ 2547904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update bmc_code_update.robot 2557904eb37SGeorge Keishing ``` 2567904eb37SGeorge Keishing 2577904eb37SGeorge Keishing For host code update, download the system type *.pnor.squashfs.tar image 2587904eb37SGeorge Keishing from https://openpower.xyz/job/openpower-op-build/ and run as follows: 2597904eb37SGeorge Keishing 2607904eb37SGeorge Keishing For Witherspoon system: 2617904eb37SGeorge Keishing ``` 2627904eb37SGeorge Keishing $ cd extended/code_update/ 2637904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update host_code_update.robot 2647904eb37SGeorge Keishing ``` 2657904eb37SGeorge Keishing 266aba958dcSGeorge Keishing* Non-UBI Format * 2677904eb37SGeorge Keishing 268aba958dcSGeorge Keishing For BMC code update, download the system type *all.tar image from 269aba958dcSGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 270359f8039SGeorge Keishing 271aba958dcSGeorge Keishing For a Zaius system: 272a7c46e39SGeorge Keishing ``` 273aba958dcSGeorge Keishing $ cd extended/code_update/ 274aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:<image path>/zaius-<date time>.all.tar --include Initiate_Code_Update_BMC update_bmc.robot 275359f8039SGeorge Keishing ``` 27697651c75SGeorge Keishing 277aba958dcSGeorge Keishing For host code update, download the system type *.pnor from 278aba958dcSGeorge Keishing https://openpower.xyz/job/openpower-op-build/ and run as follows: 279aba958dcSGeorge Keishing 280aba958dcSGeorge Keishing For a Zaius system: 281a7c46e39SGeorge Keishing ``` 282aba958dcSGeorge Keishing $ cd extended/ 283aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v PNOR_IMAGE_PATH:<image path>/zaius.pnor test_bios_update.robot 28497651c75SGeorge Keishing ``` 285