124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 5acb3c1efSGeorge Keishing* DMTF Redfish 624c9f88dSGeorge Keishing* Out-of-band IPMI 724c9f88dSGeorge Keishing* SSH to BMC and Host OS 824c9f88dSGeorge Keishing 924c9f88dSGeorge Keishing**Key Feature List** 1024c9f88dSGeorge Keishing* Power on/off 1124c9f88dSGeorge Keishing* Reboot Host 1224c9f88dSGeorge Keishing* Reset BMC 1324c9f88dSGeorge Keishing* Code update BMC and host 1424c9f88dSGeorge Keishing* Power management 15dd9471e0SGeorge Keishing* Fan controller 1624c9f88dSGeorge Keishing* HTX bootme 1724c9f88dSGeorge Keishing* XCAT execution 1824c9f88dSGeorge Keishing* Network 19dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 20a52159ffSGeorge Keishing* Factory reset 21a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2229814f0eSGeorge Keishing* Web UI testing 2338f566bbSGeorge Keishing* Secure boot 240465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 252c8c33fdSGeorge Keishing* Remote Logging via Rsyslog 26f0dc6ecfSGeorge Keishing* LDAP (Lightweight Directory Access Protocol) 27f0dc6ecfSGeorge Keishing* Certificate 288a1c686aSGeorge Keishing* Local User Management(Redfish/IPMI) 29f07f5b13SGeorge Keishing* DateTime 30f07f5b13SGeorge Keishing* Event Logging 31cb858f7aSGeorge Keishing* PLDM (Platform Level Data Model) via pldmtool 3224c9f88dSGeorge Keishing 3324c9f88dSGeorge Keishing**Debugging Supported List** 3424c9f88dSGeorge Keishing* SOL collection 3524c9f88dSGeorge Keishing* FFDC collection 3624c9f88dSGeorge Keishing* Error injection from host 3724c9f88dSGeorge Keishing 38a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 39a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 40b29d2e84SChris Austen 41917ba1a2SGunnar Mills* Miscellaneous 42a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 43a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 44a5c6e9d6SGeorge Keishing 45a5c6e9d6SGeorge Keishing REST base packages: 46a5c6e9d6SGeorge Keishing ``` 47a5c6e9d6SGeorge Keishing $ pip install -U requests 48a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 491da95695SGeorge Keishing $ pip install -U robotframework-httplibrary 50a5c6e9d6SGeorge Keishing ``` 51aba958dcSGeorge Keishing 524e64eaddSGeorge Keishing Python redfish library packages: 534e64eaddSGeorge Keishing For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library) 544e64eaddSGeorge Keishing ``` 554e64eaddSGeorge Keishing $ pip install redfish 564e64eaddSGeorge Keishing ``` 574e64eaddSGeorge Keishing 58a5c6e9d6SGeorge Keishing SSH and SCP base packages: 59a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 60a5c6e9d6SGeorge Keishing ``` 61a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 62a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 63a5c6e9d6SGeorge Keishing ``` 642de157e4SGeorge Keishing 65aba958dcSGeorge Keishing Installing tox: 66aba958dcSGeorge Keishing ``` 67aba958dcSGeorge Keishing $ pip install -U tox 68aba958dcSGeorge Keishing ``` 69aba958dcSGeorge Keishing 70aba958dcSGeorge KeishingIf using Python 3.x, use the corresponding `pip3` to install packages. 71aba958dcSGeorge Keishing 722de157e4SGeorge Keishing## OpenBMC Test Development ## 732de157e4SGeorge Keishing 742de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 752de157e4SGeorge Keishing 768a1c686aSGeorge Keishing - [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information. 772de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 78*02d25493SGeorge Keishing - [Code Check Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_standards_check.md): To check common code misspellings, syntax and standard checks. 792de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 808a1c686aSGeorge Keishing curl commands required for legacy REST testing. 818a1c686aSGeorge Keishing - [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): Quick reference for some common 828a1c686aSGeorge Keishing curl commands required for redfish testing. 8329814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 8479385f7bSGeorge Keishing - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server 852de157e4SGeorge Keishing 868c7eec37SGeorge Keishing## OpenBMC Test Documentation ## 878c7eec37SGeorge Keishing 888c7eec37SGeorge Keishing - [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools. 898c7eec37SGeorge Keishing - [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): Currently supported BMC and PNOR update. 908c7eec37SGeorge Keishing - [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): Steps to create and install CA signed certificate. 918c7eec37SGeorge Keishing 9287a3ff53SGeorge Keishing## Supported Systems Architecture ## 9387a3ff53SGeorge Keishing 9487a3ff53SGeorge Keishing OpenBMC test infrastructure is proven capable of running on: 9587a3ff53SGeorge Keishing - POWER 9687a3ff53SGeorge Keishing - x86 9787a3ff53SGeorge Keishing systems running OpenBMC firmware stack. 9887a3ff53SGeorge Keishing 993df030faSGeorge Keishing## Testing Setup Steps ## 1003df030faSGeorge Keishing 1013df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 1023df030faSGeorge Keishing 1033df030faSGeorge Keishing* Download the openbmc-test-automation repository: 1043df030faSGeorge Keishing ``` 1053df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 1063df030faSGeorge Keishing $ cd openbmc-test-automation 1073df030faSGeorge Keishing ``` 1083df030faSGeorge Keishing* Execute basic setup test run: 1093df030faSGeorge Keishing ``` 1103df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 1113df030faSGeorge Keishing ``` 1123df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 113a5c6e9d6SGeorge Keishing 1145977d2aeSGeorge Keishing## Test Layout ## 1155977d2aeSGeorge Keishing 1165977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 1175977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 1185977d2aeSGeorge Keishingclassified as follows: 1195977d2aeSGeorge Keishing 1205977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 12106d35d7dSGeorge Keishing verification. The "tests" subdirectory uses legacy REST and will be 12206d35d7dSGeorge Keishing deprecated at some point and therefore no longer supported. 1235977d2aeSGeorge Keishing 124f07f5b13SGeorge Keishing`extended/`: Contains test cases for boot testing, code update testing using legacy REST, etc. 1255977d2aeSGeorge Keishing 126b7ec2790SGeorge Keishing`systest/`: Contains test cases for HTX bootme testing. 1275977d2aeSGeorge Keishing 1285977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 1295977d2aeSGeorge Keishing 13029814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 13129814f0eSGeorge Keishing 132cb858f7aSGeorge Keishing`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, platform, OEM). 133cb858f7aSGeorge Keishing 1340465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 1350465b9cdSGeorge Keishing configuration testing. 1360465b9cdSGeorge Keishing 1370465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 1380465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 1390465b9cdSGeorge Keishing 1400465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 1410465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 14238f566bbSGeorge Keishing 1435977d2aeSGeorge Keishing`tools/`: Contains various tools. 1445977d2aeSGeorge Keishing 1455977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 146acb3c1efSGeorge Keishing 147aba958dcSGeorge Keishing`test_list/`: Contains the argument files used for skipping test cases 148aba958dcSGeorge Keishing (e.g "skip_test", "skip_test_extended", etc.) or 149aba958dcSGeorge Keishing grouping them (e.g "HW_CI", "CT_basic_run", etc.). 150aba958dcSGeorge Keishing 151acb3c1efSGeorge Keishing 152acb3c1efSGeorge Keishing## Redfish Test Layout ## 153acb3c1efSGeorge Keishing 154acb3c1efSGeorge KeishingOpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 155acb3c1efSGeorge Keishingspecification and schema that meets the expectations of end users for simple, 156acb3c1efSGeorge Keishingmodern and secure management of scalable platform hardware. 157acb3c1efSGeorge Keishing 158f07f5b13SGeorge Keishing`redfish/`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC. 1593292d4acSGeorge Keishing 160f32ee4cbSGeorge Keishing`redfish/extended/`: Contains test cases for combined legacy REST and DMTF Redfish-related 161f32ee4cbSGeorge Keishing feature supported on OpenBMC. 162f32ee4cbSGeorge Keishing 163a53c3419SGeorge KeishingNote: Work in progress test development parameter `-v REDFISH_SUPPORT_TRANS_STATE:1` to force 16487a3ff53SGeorge Keishing the test suites to execute in redfish mode only. 165acb3c1efSGeorge Keishing 1665977d2aeSGeorge Keishing 167a5c6e9d6SGeorge Keishing## Quickstart ## 1685977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1695977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1705977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1715977d2aeSGeorge Keishing 1725977d2aeSGeorge Keishing**Robot Command Line** 1735977d2aeSGeorge Keishing 1745977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1755977d2aeSGeorge Keishing ``` 1765977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1775977d2aeSGeorge Keishing ``` 178aba958dcSGeorge Keishing 1795977d2aeSGeorge Keishing* Execute a test suite: 1805977d2aeSGeorge Keishing ``` 181088da102SGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 1825977d2aeSGeorge Keishing ``` 183b29d2e84SChris Austen 18449a80604SGeorge Keishing* Initialize the following test variables which will be used during test execution: 18549a80604SGeorge Keishing 18649a80604SGeorge Keishing User can forward declare as environment variables: 187a7c46e39SGeorge Keishing ``` 18849a80604SGeorge Keishing $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 1897a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 19049a80604SGeorge Keishing $ export OPENBMC_PASSWORD=<openbmc password> 1917a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 19249a80604SGeorge Keishing ``` 19349a80604SGeorge Keishing 19449a80604SGeorge Keishing or 19549a80604SGeorge Keishing 19649a80604SGeorge Keishing User can input as robot variables as part of the CLI command: 19749a80604SGeorge Keishing ``` 19849a80604SGeorge Keishing -v OPENBMC_HOST:<openbmc machine IP address/hostname> 19949a80604SGeorge Keishing -v OPENBMC_USERNAME:<openbmc username> 20049a80604SGeorge Keishing -v OPENBMC_PASSWORD:<openbmc password> 201359f8039SGeorge Keishing ``` 202b29d2e84SChris Austen 203a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 204a7c46e39SGeorge Keishing ``` 2054d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 2064d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 207a7c46e39SGeorge Keishing ``` 208b29d2e84SChris Austen 209a7c46e39SGeorge Keishing* Run tests: 210a7c46e39SGeorge Keishing ``` 2115e021124SAlvin Wang $ tox tests 212b29d2e84SChris Austen ``` 213b29d2e84SChris Austen 214a7c46e39SGeorge Keishing* How to run individual test: 21588acd8daSGeorge Keishing 216a7c46e39SGeorge Keishing One specific test: 217a7c46e39SGeorge Keishing ``` 218a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 219a7c46e39SGeorge Keishing ``` 220a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 221a7c46e39SGeorge Keishing systems: 222a7c46e39SGeorge Keishing ``` 22388acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 224a7c46e39SGeorge Keishing ``` 225a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 226a7c46e39SGeorge Keishing ``` 227a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 228a7c46e39SGeorge Keishing ``` 229a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 230a7c46e39SGeorge Keishing ``` 231a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 232a7c46e39SGeorge Keishing ``` 233a7c46e39SGeorge Keishing 23408fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 235a7c46e39SGeorge Keishing cases using argument file: 236a7c46e39SGeorge Keishing ``` 237a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 238a7c46e39SGeorge Keishing ``` 239ddf0cc2aSGeorge Keishing 240ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 241a7c46e39SGeorge Keishing ``` 242ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 243ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 244b29d2e84SChris Austen ``` 245b29d2e84SChris Austen 2467eea8ed0SRahul Maheshwari 2477eea8ed0SRahul Maheshwari* Run IPMI tests: 2487eea8ed0SRahul Maheshwari 2497eea8ed0SRahul Maheshwari Running only out-of-band IPMI tests: 2507eea8ed0SRahul Maheshwari ``` 2517eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 2527eea8ed0SRahul Maheshwari ``` 2537eea8ed0SRahul Maheshwari 2547eea8ed0SRahul Maheshwari Running only inband IPMI tests: 2557eea8ed0SRahul Maheshwari ``` 2567eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 2577eea8ed0SRahul Maheshwari ``` 2587eea8ed0SRahul Maheshwari 2597eea8ed0SRahul Maheshwari 260dd38aef3SRahul Maheshwari* Run GUI tests: 261dd38aef3SRahul Maheshwari 262dd38aef3SRahul Maheshwari By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode: 263dd38aef3SRahul Maheshwari ``` 264eeb93c31SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 265dd38aef3SRahul Maheshwari ``` 266dd38aef3SRahul Maheshwari 2672bd32ff1SRahul Maheshwari Run GUI default CI test bucket: 2682bd32ff1SRahul Maheshwari ``` 269eeb93c31SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 2702bd32ff1SRahul Maheshwari ``` 2712bd32ff1SRahul Maheshwari 2725d00a3c8SSivas SRR* Run LDAP tests: 2735d00a3c8SSivas SRR 2745d00a3c8SSivas SRR Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server. 2755d00a3c8SSivas SRR Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server. 2765d00a3c8SSivas SRR One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd 2775d00a3c8SSivas SRR For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch". 2785d00a3c8SSivas SRR Microsoft ADS: refer to https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 279962ff298SSivas SRR 280962ff298SSivas SRR Note: Currently, LDAP test automation for Redfish API is in progress. The format to invoke LDAP test is as follows: 2815d00a3c8SSivas SRR ``` 282962ff298SSivas SRR $ cd redfish/account_service/ 283962ff298SSivas SRR $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 2845d00a3c8SSivas SRR ``` 2855d00a3c8SSivas SRR 286d7c26840SGeorge Keishing* How to run CI and CT bucket test: 287d7c26840SGeorge Keishing 288d7c26840SGeorge Keishing Default CI test bucket list: 289d7c26840SGeorge Keishing ``` 290d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 291d7c26840SGeorge Keishing ``` 292d7c26840SGeorge Keishing 293d7c26840SGeorge Keishing Default CI smoke test bucket list: 294d7c26840SGeorge Keishing ``` 295d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 296d7c26840SGeorge Keishing ``` 297d7c26840SGeorge Keishing 298aba958dcSGeorge Keishing* Run extended tests: 299aba958dcSGeorge Keishing 300aba958dcSGeorge Keishing For-loop test (default iteration is 10): 301aba958dcSGeorge Keishing ``` 302aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 303aba958dcSGeorge Keishing ``` 304aba958dcSGeorge Keishing 305aba958dcSGeorge Keishing Example using tox testing a test suite for 5 iterations "witherspoon": 306aba958dcSGeorge Keishing ``` 307aba958dcSGeorge Keishing OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 308aba958dcSGeorge Keishing ``` 309aba958dcSGeorge Keishing 310d61a64baSKonstantin Aladyshev* Host CPU architecture 311d61a64baSKonstantin Aladyshev 312d61a64baSKonstantin Aladyshev By default openbmc-test-automation framework assumes that host CPU is based on the POWER architecture. 313c056c2bbSKonstantin Aladyshev If your host CPU is x86 add `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an environment variable: 314c056c2bbSKonstantin Aladyshev ``` 315c056c2bbSKonstantin Aladyshev $ export PLATFORM_ARCH_TYPE=x86 316c056c2bbSKonstantin Aladyshev ``` 317d61a64baSKonstantin Aladyshev 318aba958dcSGeorge Keishing**Jenkins jobs tox commands** 319aba958dcSGeorge Keishing* HW CI tox command: 320aba958dcSGeorge Keishing ``` 321aba958dcSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 322aba958dcSGeorge Keishing ``` 323