124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 5acb3c1efSGeorge Keishing* DMTF Redfish 624c9f88dSGeorge Keishing* Out-of-band IPMI 724c9f88dSGeorge Keishing* SSH to BMC and Host OS 824c9f88dSGeorge Keishing 924c9f88dSGeorge Keishing**Key Feature List** 1024c9f88dSGeorge Keishing* Power on/off 1124c9f88dSGeorge Keishing* Reboot Host 1224c9f88dSGeorge Keishing* Reset BMC 1324c9f88dSGeorge Keishing* Code update BMC and host 1424c9f88dSGeorge Keishing* Power management 15dd9471e0SGeorge Keishing* Fan controller 1624c9f88dSGeorge Keishing* HTX bootme 1724c9f88dSGeorge Keishing* XCAT execution 1824c9f88dSGeorge Keishing* Network 19dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 20a52159ffSGeorge Keishing* Factory reset 21a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2229814f0eSGeorge Keishing* Web UI testing 238b4f4034SGeorge Keishing* IO storage and EEH (Enhanced Error Handling) testing 2438f566bbSGeorge Keishing* Secure boot 250465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 262c8c33fdSGeorge Keishing* Remote Logging via Rsyslog 27f0dc6ecfSGeorge Keishing* LDAP (Lightweight Directory Access Protocol) 28f0dc6ecfSGeorge Keishing* Certificate 2924c9f88dSGeorge Keishing 3024c9f88dSGeorge Keishing**Debugging Supported List** 3124c9f88dSGeorge Keishing* SOL collection 3224c9f88dSGeorge Keishing* FFDC collection 3324c9f88dSGeorge Keishing* Error injection from host 3424c9f88dSGeorge Keishing 35a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 36a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 37b29d2e84SChris Austen 38917ba1a2SGunnar Mills* Miscellaneous 39a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 40a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 41a5c6e9d6SGeorge Keishing 42a5c6e9d6SGeorge Keishing REST base packages: 43a5c6e9d6SGeorge Keishing ``` 44a5c6e9d6SGeorge Keishing $ pip install -U requests 45a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 461da95695SGeorge Keishing $ pip install -U robotframework-httplibrary 47a5c6e9d6SGeorge Keishing ``` 48aba958dcSGeorge Keishing 494e64eaddSGeorge Keishing Python redfish library packages: 504e64eaddSGeorge Keishing For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library) 514e64eaddSGeorge Keishing ``` 524e64eaddSGeorge Keishing $ pip install redfish 534e64eaddSGeorge Keishing ``` 544e64eaddSGeorge Keishing 55a5c6e9d6SGeorge Keishing SSH and SCP base packages: 56a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 57a5c6e9d6SGeorge Keishing ``` 58a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 59a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 60a5c6e9d6SGeorge Keishing ``` 612de157e4SGeorge Keishing 62aba958dcSGeorge Keishing Installing tox: 63aba958dcSGeorge Keishing ``` 64aba958dcSGeorge Keishing $ pip install -U tox 65aba958dcSGeorge Keishing ``` 66aba958dcSGeorge Keishing 67aba958dcSGeorge KeishingIf using Python 3.x, use the corresponding `pip3` to install packages. 68aba958dcSGeorge Keishing 692de157e4SGeorge Keishing## OpenBMC Test Development ## 702de157e4SGeorge Keishing 712de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 722de157e4SGeorge Keishing 730b9bd82cSGeorge Keishing - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information. 742de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 752de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 762de157e4SGeorge Keishing curl commands required for testing. 7729814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 78fa6da2f1SGeorge Keishing - [Tools.md](Tools.md): Reference information for helper tools. 792de157e4SGeorge Keishing 803df030faSGeorge Keishing## Testing Setup Steps ## 813df030faSGeorge Keishing 823df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 833df030faSGeorge Keishing 843df030faSGeorge Keishing* Download the openbmc-test-automation repository: 853df030faSGeorge Keishing ``` 863df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 873df030faSGeorge Keishing $ cd openbmc-test-automation 883df030faSGeorge Keishing ``` 893df030faSGeorge Keishing* Execute basic setup test run: 903df030faSGeorge Keishing ``` 913df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 923df030faSGeorge Keishing ``` 933df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 94a5c6e9d6SGeorge Keishing 955977d2aeSGeorge Keishing## Test Layout ## 965977d2aeSGeorge Keishing 975977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 985977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 995977d2aeSGeorge Keishingclassified as follows: 1005977d2aeSGeorge Keishing 1015977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 1025977d2aeSGeorge Keishing verification. 1035977d2aeSGeorge Keishing 1045977d2aeSGeorge Keishing`extended/`: Contains test cases for boot testing, network testing, 105a52159ffSGeorge Keishing code update testing etc. 1065977d2aeSGeorge Keishing 1078b4f4034SGeorge Keishing`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing. 1085977d2aeSGeorge Keishing 1095977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 1105977d2aeSGeorge Keishing 11129814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 11229814f0eSGeorge Keishing 11339b8e96eSPrashanth Katti`network/`: Contains test cases for network testing. It covers IPv4 in static mode 11439b8e96eSPrashanth Katti and in DHCP mode. 115df62e5c4SGeorge Keishing 1160465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 1170465b9cdSGeorge Keishing configuration testing. 1180465b9cdSGeorge Keishing 1192c8c33fdSGeorge Keishing`remote_logging`: Contains test cases for remote logging via rsyslog. 1202c8c33fdSGeorge Keishing 1210465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 1220465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 1230465b9cdSGeorge Keishing 1240465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 1250465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 12638f566bbSGeorge Keishing 1275977d2aeSGeorge Keishing`tools/`: Contains various tools. 1285977d2aeSGeorge Keishing 1295977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 130acb3c1efSGeorge Keishing 131aba958dcSGeorge Keishing`test_list/`: Contains the argument files used for skipping test cases 132aba958dcSGeorge Keishing (e.g "skip_test", "skip_test_extended", etc.) or 133aba958dcSGeorge Keishing grouping them (e.g "HW_CI", "CT_basic_run", etc.). 134aba958dcSGeorge Keishing 135acb3c1efSGeorge Keishing 136acb3c1efSGeorge Keishing## Redfish Test Layout ## 137acb3c1efSGeorge Keishing 138acb3c1efSGeorge KeishingOpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 139acb3c1efSGeorge Keishingspecification and schema that meets the expectations of end users for simple, 140acb3c1efSGeorge Keishingmodern and secure management of scalable platform hardware. 141acb3c1efSGeorge Keishing 142*3dc23cc9SGeorge Keishing`redfish`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC. 143acb3c1efSGeorge Keishing 1445977d2aeSGeorge Keishing 145a5c6e9d6SGeorge Keishing## Quickstart ## 1465977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1475977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1485977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1495977d2aeSGeorge Keishing 1505977d2aeSGeorge Keishing**Robot Command Line** 1515977d2aeSGeorge Keishing 1525977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1535977d2aeSGeorge Keishing ``` 1545977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1555977d2aeSGeorge Keishing ``` 156aba958dcSGeorge Keishing 1575977d2aeSGeorge Keishing* Execute a test suite: 1585977d2aeSGeorge Keishing ``` 1595977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests/test_basic_poweron.robot 1605977d2aeSGeorge Keishing ``` 161b29d2e84SChris Austen 162a7c46e39SGeorge Keishing* Initialize the following environment variables which will be used during testing: 163a7c46e39SGeorge Keishing ``` 164b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 1657a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 1667a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 167aba958dcSGeorge Keishing $ export OPENBMC_MODEL=[./data/Witherspoon.py, ./data/Palmetto.py, etc] 1687a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 1697a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 170359f8039SGeorge Keishing ``` 171b29d2e84SChris Austen 172a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 173a7c46e39SGeorge Keishing ``` 1744d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 1754d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 176a7c46e39SGeorge Keishing ``` 177b29d2e84SChris Austen 178a7c46e39SGeorge Keishing* Run tests: 179a7c46e39SGeorge Keishing ``` 180b29d2e84SChris Austen $ tox -e tests 181b29d2e84SChris Austen ``` 182b29d2e84SChris Austen 183a7c46e39SGeorge Keishing* How to run individual test: 18488acd8daSGeorge Keishing 185a7c46e39SGeorge Keishing One specific test: 186a7c46e39SGeorge Keishing ``` 187a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 188a7c46e39SGeorge Keishing ``` 189a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 190a7c46e39SGeorge Keishing systems: 191a7c46e39SGeorge Keishing ``` 19288acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 193a7c46e39SGeorge Keishing ``` 194a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 195a7c46e39SGeorge Keishing ``` 196a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 197a7c46e39SGeorge Keishing ``` 198a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 199a7c46e39SGeorge Keishing ``` 200a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 201a7c46e39SGeorge Keishing ``` 202a7c46e39SGeorge Keishing 20308fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 204a7c46e39SGeorge Keishing cases using argument file: 205a7c46e39SGeorge Keishing ``` 206a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 207a7c46e39SGeorge Keishing ``` 208ddf0cc2aSGeorge Keishing 209ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 210a7c46e39SGeorge Keishing ``` 211ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 212ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 213b29d2e84SChris Austen ``` 214b29d2e84SChris Austen 2157eea8ed0SRahul Maheshwari 2167eea8ed0SRahul Maheshwari* Run IPMI tests: 2177eea8ed0SRahul Maheshwari 2187eea8ed0SRahul Maheshwari Running only out-of-band IPMI tests: 2197eea8ed0SRahul Maheshwari ``` 2207eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 2217eea8ed0SRahul Maheshwari ``` 2227eea8ed0SRahul Maheshwari 2237eea8ed0SRahul Maheshwari Running only inband IPMI tests: 2247eea8ed0SRahul Maheshwari ``` 2257eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 2267eea8ed0SRahul Maheshwari ``` 2277eea8ed0SRahul Maheshwari 2287eea8ed0SRahul Maheshwari 229dd38aef3SRahul Maheshwari* Run GUI tests: 230dd38aef3SRahul Maheshwari 231dd38aef3SRahul Maheshwari By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode: 232dd38aef3SRahul Maheshwari ``` 233b6903f27SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/obmc_gui/test/ 234dd38aef3SRahul Maheshwari ``` 235dd38aef3SRahul Maheshwari 2362bd32ff1SRahul Maheshwari Run GUI default CI test bucket: 2372bd32ff1SRahul Maheshwari ``` 2382bd32ff1SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/obmc_gui/test/ 2392bd32ff1SRahul Maheshwari ``` 2402bd32ff1SRahul Maheshwari 2415d00a3c8SSivas SRR* Run LDAP tests: 2425d00a3c8SSivas SRR 2435d00a3c8SSivas SRR Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server. 2445d00a3c8SSivas SRR Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server. 2455d00a3c8SSivas SRR One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd 2465d00a3c8SSivas SRR For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch". 2475d00a3c8SSivas SRR Microsoft ADS: refer to https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 2485d00a3c8SSivas SRR ``` 249f0dc6ecfSGeorge Keishing $ cd tests_ldap/ 250b6903f27SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> test_user_ldap_management.robot 2515d00a3c8SSivas SRR ``` 2525d00a3c8SSivas SRR 253d7c26840SGeorge Keishing* How to run CI and CT bucket test: 254d7c26840SGeorge Keishing 255d7c26840SGeorge Keishing Default CI test bucket list: 256d7c26840SGeorge Keishing ``` 257d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 258d7c26840SGeorge Keishing ``` 259d7c26840SGeorge Keishing 260d7c26840SGeorge Keishing Default CI smoke test bucket list: 261d7c26840SGeorge Keishing ``` 262d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 263d7c26840SGeorge Keishing ``` 264d7c26840SGeorge Keishing 265aba958dcSGeorge Keishing* Run extended tests: 266aba958dcSGeorge Keishing 267aba958dcSGeorge Keishing For-loop test (default iteration is 10): 268aba958dcSGeorge Keishing ``` 269aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 270aba958dcSGeorge Keishing ``` 271aba958dcSGeorge Keishing 272aba958dcSGeorge Keishing Example using tox testing a test suite for 5 iterations "witherspoon": 273aba958dcSGeorge Keishing ``` 274aba958dcSGeorge Keishing OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 275aba958dcSGeorge Keishing ``` 276aba958dcSGeorge Keishing 277aba958dcSGeorge Keishing**Jenkins jobs tox commands** 278aba958dcSGeorge Keishing* HW CI tox command: 279aba958dcSGeorge Keishing ``` 280aba958dcSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 281aba958dcSGeorge Keishing ``` 282aba958dcSGeorge Keishing 283aba958dcSGeorge Keishing## Code Update ## 284aba958dcSGeorge Keishing 285aba958dcSGeorge KeishingCurrently supported BMC and PNOR update formats are UBI and non-UBI. 286aba958dcSGeorge KeishingFor code update information, please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md) 287aba958dcSGeorge Keishing 288aba958dcSGeorge Keishing 289aba958dcSGeorge Keishing* UBI Format * 2907904eb37SGeorge Keishing 2917904eb37SGeorge Keishing For BMC code update, download the system type *.ubi.mdt.tar image from 2927904eb37SGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 2937904eb37SGeorge Keishing 2947904eb37SGeorge Keishing For Witherspoon system: 2957904eb37SGeorge Keishing ``` 2967904eb37SGeorge Keishing $ cd extended/code_update/ 2977904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update bmc_code_update.robot 2987904eb37SGeorge Keishing ``` 2997904eb37SGeorge Keishing 3007904eb37SGeorge Keishing For host code update, download the system type *.pnor.squashfs.tar image 3017904eb37SGeorge Keishing from https://openpower.xyz/job/openpower-op-build/ and run as follows: 3027904eb37SGeorge Keishing 3037904eb37SGeorge Keishing For Witherspoon system: 3047904eb37SGeorge Keishing ``` 3057904eb37SGeorge Keishing $ cd extended/code_update/ 3067904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update host_code_update.robot 3077904eb37SGeorge Keishing ``` 3087904eb37SGeorge Keishing 309aba958dcSGeorge Keishing* Non-UBI Format * 3107904eb37SGeorge Keishing 311aba958dcSGeorge Keishing For BMC code update, download the system type *all.tar image from 312aba958dcSGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 313359f8039SGeorge Keishing 314aba958dcSGeorge Keishing For a Zaius system: 315a7c46e39SGeorge Keishing ``` 316aba958dcSGeorge Keishing $ cd extended/code_update/ 317aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:<image path>/zaius-<date time>.all.tar --include Initiate_Code_Update_BMC update_bmc.robot 318359f8039SGeorge Keishing ``` 31997651c75SGeorge Keishing 320aba958dcSGeorge Keishing For host code update, download the system type *.pnor from 321aba958dcSGeorge Keishing https://openpower.xyz/job/openpower-op-build/ and run as follows: 322aba958dcSGeorge Keishing 323aba958dcSGeorge Keishing For a Zaius system: 324a7c46e39SGeorge Keishing ``` 325aba958dcSGeorge Keishing $ cd extended/ 326aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v PNOR_IMAGE_PATH:<image path>/zaius.pnor test_bios_update.robot 32797651c75SGeorge Keishing ``` 328