124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 5acb3c1efSGeorge Keishing* DMTF Redfish 624c9f88dSGeorge Keishing* Out-of-band IPMI 724c9f88dSGeorge Keishing* SSH to BMC and Host OS 824c9f88dSGeorge Keishing 924c9f88dSGeorge Keishing**Key Feature List** 1024c9f88dSGeorge Keishing* Power on/off 1124c9f88dSGeorge Keishing* Reboot Host 1224c9f88dSGeorge Keishing* Reset BMC 1324c9f88dSGeorge Keishing* Code update BMC and host 1424c9f88dSGeorge Keishing* Power management 15dd9471e0SGeorge Keishing* Fan controller 1624c9f88dSGeorge Keishing* HTX bootme 1724c9f88dSGeorge Keishing* XCAT execution 1824c9f88dSGeorge Keishing* Network 19dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 20a52159ffSGeorge Keishing* Factory reset 21a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2229814f0eSGeorge Keishing* Web UI testing 238b4f4034SGeorge Keishing* IO storage and EEH (Enhanced Error Handling) testing 2438f566bbSGeorge Keishing* Secure boot 250465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 262c8c33fdSGeorge Keishing* Remote Logging via Rsyslog 27f0dc6ecfSGeorge Keishing* LDAP (Lightweight Directory Access Protocol) 28f0dc6ecfSGeorge Keishing* Certificate 2924c9f88dSGeorge Keishing 3024c9f88dSGeorge Keishing**Debugging Supported List** 3124c9f88dSGeorge Keishing* SOL collection 3224c9f88dSGeorge Keishing* FFDC collection 3324c9f88dSGeorge Keishing* Error injection from host 3424c9f88dSGeorge Keishing 35a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 36a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 37b29d2e84SChris Austen 38917ba1a2SGunnar Mills* Miscellaneous 39a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 40a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 41a5c6e9d6SGeorge Keishing 42a5c6e9d6SGeorge Keishing REST base packages: 43a5c6e9d6SGeorge Keishing ``` 44a5c6e9d6SGeorge Keishing $ pip install -U requests 45a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 461da95695SGeorge Keishing $ pip install -U robotframework-httplibrary 47a5c6e9d6SGeorge Keishing ``` 48aba958dcSGeorge Keishing 494e64eaddSGeorge Keishing Python redfish library packages: 504e64eaddSGeorge Keishing For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library) 514e64eaddSGeorge Keishing ``` 524e64eaddSGeorge Keishing $ pip install redfish 534e64eaddSGeorge Keishing ``` 544e64eaddSGeorge Keishing 55a5c6e9d6SGeorge Keishing SSH and SCP base packages: 56a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 57a5c6e9d6SGeorge Keishing ``` 58a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 59a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 60a5c6e9d6SGeorge Keishing ``` 612de157e4SGeorge Keishing 62aba958dcSGeorge Keishing Installing tox: 63aba958dcSGeorge Keishing ``` 64aba958dcSGeorge Keishing $ pip install -U tox 65aba958dcSGeorge Keishing ``` 66aba958dcSGeorge Keishing 67aba958dcSGeorge KeishingIf using Python 3.x, use the corresponding `pip3` to install packages. 68aba958dcSGeorge Keishing 692de157e4SGeorge Keishing## OpenBMC Test Development ## 702de157e4SGeorge Keishing 712de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 722de157e4SGeorge Keishing 730b9bd82cSGeorge Keishing - [MAINTAINERS](https://github.com/openbmc/docs/blob/master/MAINTAINERS): OpenBMC code maintainers information. 742de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 752de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 762de157e4SGeorge Keishing curl commands required for testing. 7729814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 78171180dcSGeorge Keishing - [Tools.md](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools. 79*79385f7bSGeorge Keishing - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server 802de157e4SGeorge Keishing 813df030faSGeorge Keishing## Testing Setup Steps ## 823df030faSGeorge Keishing 833df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 843df030faSGeorge Keishing 853df030faSGeorge Keishing* Download the openbmc-test-automation repository: 863df030faSGeorge Keishing ``` 873df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 883df030faSGeorge Keishing $ cd openbmc-test-automation 893df030faSGeorge Keishing ``` 903df030faSGeorge Keishing* Execute basic setup test run: 913df030faSGeorge Keishing ``` 923df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 933df030faSGeorge Keishing ``` 943df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 95a5c6e9d6SGeorge Keishing 965977d2aeSGeorge Keishing## Test Layout ## 975977d2aeSGeorge Keishing 985977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 995977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 1005977d2aeSGeorge Keishingclassified as follows: 1015977d2aeSGeorge Keishing 1025977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 1035977d2aeSGeorge Keishing verification. 1045977d2aeSGeorge Keishing 1055977d2aeSGeorge Keishing`extended/`: Contains test cases for boot testing, network testing, 106a52159ffSGeorge Keishing code update testing etc. 1075977d2aeSGeorge Keishing 1088b4f4034SGeorge Keishing`systest/`: Contains test cases for HTX bootme, IO storage and EEH testing. 1095977d2aeSGeorge Keishing 1105977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 1115977d2aeSGeorge Keishing 11229814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 11329814f0eSGeorge Keishing 11439b8e96eSPrashanth Katti`network/`: Contains test cases for network testing. It covers IPv4 in static mode 11539b8e96eSPrashanth Katti and in DHCP mode. 116df62e5c4SGeorge Keishing 1170465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 1180465b9cdSGeorge Keishing configuration testing. 1190465b9cdSGeorge Keishing 1202c8c33fdSGeorge Keishing`remote_logging`: Contains test cases for remote logging via rsyslog. 1212c8c33fdSGeorge Keishing 1220465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 1230465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 1240465b9cdSGeorge Keishing 1250465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 1260465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 12738f566bbSGeorge Keishing 1285977d2aeSGeorge Keishing`tools/`: Contains various tools. 1295977d2aeSGeorge Keishing 1305977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 131acb3c1efSGeorge Keishing 132aba958dcSGeorge Keishing`test_list/`: Contains the argument files used for skipping test cases 133aba958dcSGeorge Keishing (e.g "skip_test", "skip_test_extended", etc.) or 134aba958dcSGeorge Keishing grouping them (e.g "HW_CI", "CT_basic_run", etc.). 135aba958dcSGeorge Keishing 136acb3c1efSGeorge Keishing 137acb3c1efSGeorge Keishing## Redfish Test Layout ## 138acb3c1efSGeorge Keishing 139acb3c1efSGeorge KeishingOpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 140acb3c1efSGeorge Keishingspecification and schema that meets the expectations of end users for simple, 141acb3c1efSGeorge Keishingmodern and secure management of scalable platform hardware. 142acb3c1efSGeorge Keishing 1433dc23cc9SGeorge Keishing`redfish`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC. 144acb3c1efSGeorge Keishing 1455977d2aeSGeorge Keishing 146a5c6e9d6SGeorge Keishing## Quickstart ## 1475977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1485977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1495977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1505977d2aeSGeorge Keishing 1515977d2aeSGeorge Keishing**Robot Command Line** 1525977d2aeSGeorge Keishing 1535977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1545977d2aeSGeorge Keishing ``` 1555977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1565977d2aeSGeorge Keishing ``` 157aba958dcSGeorge Keishing 1585977d2aeSGeorge Keishing* Execute a test suite: 1595977d2aeSGeorge Keishing ``` 1605977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests/test_basic_poweron.robot 1615977d2aeSGeorge Keishing ``` 162b29d2e84SChris Austen 163a7c46e39SGeorge Keishing* Initialize the following environment variables which will be used during testing: 164a7c46e39SGeorge Keishing ``` 165b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 1667a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 1677a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 168aba958dcSGeorge Keishing $ export OPENBMC_MODEL=[./data/Witherspoon.py, ./data/Palmetto.py, etc] 1697a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 1707a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 171359f8039SGeorge Keishing ``` 172b29d2e84SChris Austen 173a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 174a7c46e39SGeorge Keishing ``` 1754d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 1764d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 177a7c46e39SGeorge Keishing ``` 178b29d2e84SChris Austen 179a7c46e39SGeorge Keishing* Run tests: 180a7c46e39SGeorge Keishing ``` 181b29d2e84SChris Austen $ tox -e tests 182b29d2e84SChris Austen ``` 183b29d2e84SChris Austen 184a7c46e39SGeorge Keishing* How to run individual test: 18588acd8daSGeorge Keishing 186a7c46e39SGeorge Keishing One specific test: 187a7c46e39SGeorge Keishing ``` 188a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 189a7c46e39SGeorge Keishing ``` 190a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 191a7c46e39SGeorge Keishing systems: 192a7c46e39SGeorge Keishing ``` 19388acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 194a7c46e39SGeorge Keishing ``` 195a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 196a7c46e39SGeorge Keishing ``` 197a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 198a7c46e39SGeorge Keishing ``` 199a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 200a7c46e39SGeorge Keishing ``` 201a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 202a7c46e39SGeorge Keishing ``` 203a7c46e39SGeorge Keishing 20408fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 205a7c46e39SGeorge Keishing cases using argument file: 206a7c46e39SGeorge Keishing ``` 207a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 208a7c46e39SGeorge Keishing ``` 209ddf0cc2aSGeorge Keishing 210ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 211a7c46e39SGeorge Keishing ``` 212ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 213ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 214b29d2e84SChris Austen ``` 215b29d2e84SChris Austen 2167eea8ed0SRahul Maheshwari 2177eea8ed0SRahul Maheshwari* Run IPMI tests: 2187eea8ed0SRahul Maheshwari 2197eea8ed0SRahul Maheshwari Running only out-of-band IPMI tests: 2207eea8ed0SRahul Maheshwari ``` 2217eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 2227eea8ed0SRahul Maheshwari ``` 2237eea8ed0SRahul Maheshwari 2247eea8ed0SRahul Maheshwari Running only inband IPMI tests: 2257eea8ed0SRahul Maheshwari ``` 2267eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 2277eea8ed0SRahul Maheshwari ``` 2287eea8ed0SRahul Maheshwari 2297eea8ed0SRahul Maheshwari 230dd38aef3SRahul Maheshwari* Run GUI tests: 231dd38aef3SRahul Maheshwari 232dd38aef3SRahul Maheshwari By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode: 233dd38aef3SRahul Maheshwari ``` 234b6903f27SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/obmc_gui/test/ 235dd38aef3SRahul Maheshwari ``` 236dd38aef3SRahul Maheshwari 2372bd32ff1SRahul Maheshwari Run GUI default CI test bucket: 2382bd32ff1SRahul Maheshwari ``` 2392bd32ff1SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/obmc_gui/test/ 2402bd32ff1SRahul Maheshwari ``` 2412bd32ff1SRahul Maheshwari 2425d00a3c8SSivas SRR* Run LDAP tests: 2435d00a3c8SSivas SRR 2445d00a3c8SSivas SRR Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server. 2455d00a3c8SSivas SRR Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server. 2465d00a3c8SSivas SRR One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd 2475d00a3c8SSivas SRR For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch". 2485d00a3c8SSivas SRR Microsoft ADS: refer to https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 2495d00a3c8SSivas SRR ``` 250f0dc6ecfSGeorge Keishing $ cd tests_ldap/ 251b6903f27SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> test_user_ldap_management.robot 2525d00a3c8SSivas SRR ``` 2535d00a3c8SSivas SRR 254d7c26840SGeorge Keishing* How to run CI and CT bucket test: 255d7c26840SGeorge Keishing 256d7c26840SGeorge Keishing Default CI test bucket list: 257d7c26840SGeorge Keishing ``` 258d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 259d7c26840SGeorge Keishing ``` 260d7c26840SGeorge Keishing 261d7c26840SGeorge Keishing Default CI smoke test bucket list: 262d7c26840SGeorge Keishing ``` 263d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 264d7c26840SGeorge Keishing ``` 265d7c26840SGeorge Keishing 266aba958dcSGeorge Keishing* Run extended tests: 267aba958dcSGeorge Keishing 268aba958dcSGeorge Keishing For-loop test (default iteration is 10): 269aba958dcSGeorge Keishing ``` 270aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 271aba958dcSGeorge Keishing ``` 272aba958dcSGeorge Keishing 273aba958dcSGeorge Keishing Example using tox testing a test suite for 5 iterations "witherspoon": 274aba958dcSGeorge Keishing ``` 275aba958dcSGeorge Keishing OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 276aba958dcSGeorge Keishing ``` 277aba958dcSGeorge Keishing 278aba958dcSGeorge Keishing**Jenkins jobs tox commands** 279aba958dcSGeorge Keishing* HW CI tox command: 280aba958dcSGeorge Keishing ``` 281aba958dcSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 282aba958dcSGeorge Keishing ``` 283aba958dcSGeorge Keishing 284aba958dcSGeorge Keishing## Code Update ## 285aba958dcSGeorge Keishing 286aba958dcSGeorge KeishingCurrently supported BMC and PNOR update formats are UBI and non-UBI. 287aba958dcSGeorge KeishingFor code update information, please refer to [code-update.md](https://github.com/openbmc/docs/blob/master/code-update/code-update.md) 288aba958dcSGeorge Keishing 289aba958dcSGeorge Keishing 290aba958dcSGeorge Keishing* UBI Format * 2917904eb37SGeorge Keishing 2927904eb37SGeorge Keishing For BMC code update, download the system type *.ubi.mdt.tar image from 2937904eb37SGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 2947904eb37SGeorge Keishing 2957904eb37SGeorge Keishing For Witherspoon system: 2967904eb37SGeorge Keishing ``` 2977904eb37SGeorge Keishing $ cd extended/code_update/ 2987904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/obmc-phosphor-image-witherspoon.ubi.mtd.tar --include REST_BMC_Code_Update bmc_code_update.robot 2997904eb37SGeorge Keishing ``` 3007904eb37SGeorge Keishing 3017904eb37SGeorge Keishing For host code update, download the system type *.pnor.squashfs.tar image 3027904eb37SGeorge Keishing from https://openpower.xyz/job/openpower-op-build/ and run as follows: 3037904eb37SGeorge Keishing 3047904eb37SGeorge Keishing For Witherspoon system: 3057904eb37SGeorge Keishing ``` 3067904eb37SGeorge Keishing $ cd extended/code_update/ 3077904eb37SGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v IMAGE_FILE_PATH:<image path>/witherspoon.pnor.squashfs.tar --include REST_Host_Code_Update host_code_update.robot 3087904eb37SGeorge Keishing ``` 3097904eb37SGeorge Keishing 310aba958dcSGeorge Keishing* Non-UBI Format * 3117904eb37SGeorge Keishing 312aba958dcSGeorge Keishing For BMC code update, download the system type *all.tar image from 313aba958dcSGeorge Keishing https://openpower.xyz/job/openbmc-build/ and run as follows: 314359f8039SGeorge Keishing 315aba958dcSGeorge Keishing For a Zaius system: 316a7c46e39SGeorge Keishing ``` 317aba958dcSGeorge Keishing $ cd extended/code_update/ 318aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:<image path>/zaius-<date time>.all.tar --include Initiate_Code_Update_BMC update_bmc.robot 319359f8039SGeorge Keishing ``` 32097651c75SGeorge Keishing 321aba958dcSGeorge Keishing For host code update, download the system type *.pnor from 322aba958dcSGeorge Keishing https://openpower.xyz/job/openpower-op-build/ and run as follows: 323aba958dcSGeorge Keishing 324aba958dcSGeorge Keishing For a Zaius system: 325a7c46e39SGeorge Keishing ``` 326aba958dcSGeorge Keishing $ cd extended/ 327aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v PNOR_IMAGE_PATH:<image path>/zaius.pnor test_bios_update.robot 32897651c75SGeorge Keishing ``` 329