124c9f88dSGeorge Keishing## Features of OpenBMC Test Automation ## 224c9f88dSGeorge Keishing 324c9f88dSGeorge Keishing**Interface Feature List** 424c9f88dSGeorge Keishing* REST 5acb3c1efSGeorge Keishing* DMTF Redfish 624c9f88dSGeorge Keishing* Out-of-band IPMI 724c9f88dSGeorge Keishing* SSH to BMC and Host OS 824c9f88dSGeorge Keishing 924c9f88dSGeorge Keishing**Key Feature List** 1024c9f88dSGeorge Keishing* Power on/off 1124c9f88dSGeorge Keishing* Reboot Host 1224c9f88dSGeorge Keishing* Reset BMC 1324c9f88dSGeorge Keishing* Code update BMC and host 1424c9f88dSGeorge Keishing* Power management 15dd9471e0SGeorge Keishing* Fan controller 1624c9f88dSGeorge Keishing* HTX bootme 1724c9f88dSGeorge Keishing* XCAT execution 1824c9f88dSGeorge Keishing* Network 19dd9471e0SGeorge Keishing* IPMI support (generic and DCMI compliant) 20a52159ffSGeorge Keishing* Factory reset 21a52159ffSGeorge Keishing* RAS (Reliability, availability and serviceability) 2229814f0eSGeorge Keishing* Web UI testing 2338f566bbSGeorge Keishing* Secure boot 240465b9cdSGeorge Keishing* SNMP (Simple Network Management Protocol) 252c8c33fdSGeorge Keishing* Remote Logging via Rsyslog 26f0dc6ecfSGeorge Keishing* LDAP (Lightweight Directory Access Protocol) 27f0dc6ecfSGeorge Keishing* Certificate 288a1c686aSGeorge Keishing* Local User Management(Redfish/IPMI) 29f07f5b13SGeorge Keishing* DateTime 30f07f5b13SGeorge Keishing* Event Logging 31cb858f7aSGeorge Keishing* PLDM (Platform Level Data Model) via pldmtool 3224c9f88dSGeorge Keishing 3324c9f88dSGeorge Keishing**Debugging Supported List** 3424c9f88dSGeorge Keishing* SOL collection 3524c9f88dSGeorge Keishing* FFDC collection 3624c9f88dSGeorge Keishing* Error injection from host 3724c9f88dSGeorge Keishing 38a5c6e9d6SGeorge Keishing## Installation Setup Guide ## 39a5c6e9d6SGeorge Keishing* [Robot Framework Install Instruction](https://github.com/robotframework/robotframework/blob/master/INSTALL.rst) 40b29d2e84SChris Austen 41917ba1a2SGunnar Mills* Miscellaneous 42a5c6e9d6SGeorge KeishingPackages required to be installed for OpenBmc Automation. 43a5c6e9d6SGeorge KeishingInstall the packages and it's dependencies via `pip` 44a5c6e9d6SGeorge Keishing 45a5c6e9d6SGeorge Keishing REST base packages: 46a5c6e9d6SGeorge Keishing ``` 47a5c6e9d6SGeorge Keishing $ pip install -U requests 48a5c6e9d6SGeorge Keishing $ pip install -U robotframework-requests 491da95695SGeorge Keishing $ pip install -U robotframework-httplibrary 50a5c6e9d6SGeorge Keishing ``` 51aba958dcSGeorge Keishing 524e64eaddSGeorge Keishing Python redfish library packages: 534e64eaddSGeorge Keishing For more detailed intstructions see [python-redfish-library](https://github.com/DMTF/python-redfish-library) 544e64eaddSGeorge Keishing ``` 554e64eaddSGeorge Keishing $ pip install redfish 564e64eaddSGeorge Keishing ``` 574e64eaddSGeorge Keishing 58a5c6e9d6SGeorge Keishing SSH and SCP base packages: 59a5c6e9d6SGeorge Keishing For more detailed installation instructions see [robotframework-sshlibrary](https://pypi.python.org/pypi/robotframework-sshlibrary) 60a5c6e9d6SGeorge Keishing ``` 61a5c6e9d6SGeorge Keishing $ pip install robotframework-sshlibrary 62a5c6e9d6SGeorge Keishing $ pip install robotframework-scplibrary 63a5c6e9d6SGeorge Keishing ``` 642de157e4SGeorge Keishing 65aba958dcSGeorge Keishing Installing tox: 66aba958dcSGeorge Keishing ``` 67aba958dcSGeorge Keishing $ pip install -U tox 68aba958dcSGeorge Keishing ``` 69aba958dcSGeorge Keishing 70aba958dcSGeorge KeishingIf using Python 3.x, use the corresponding `pip3` to install packages. 71aba958dcSGeorge Keishing 722de157e4SGeorge Keishing## OpenBMC Test Development ## 732de157e4SGeorge Keishing 742de157e4SGeorge KeishingThese documents contain details on developing OpenBMC test code and debugging. 752de157e4SGeorge Keishing 768a1c686aSGeorge Keishing - [MAINTAINERS](MAINTAINERS): OpenBMC test code maintainers information. 772de157e4SGeorge Keishing - [CONTRIBUTING.md](CONTRIBUTING.md): Coding guidelines. 782de157e4SGeorge Keishing - [REST-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REST-cheatsheet.md): Quick reference for some common 798a1c686aSGeorge Keishing curl commands required for legacy REST testing. 808a1c686aSGeorge Keishing - [REDFISH-cheatsheet.md](https://github.com/openbmc/docs/blob/master/REDFISH-cheatsheet.md): Quick reference for some common 818a1c686aSGeorge Keishing curl commands required for redfish testing. 8229814f0eSGeorge Keishing - [README.md](https://github.com/openbmc/phosphor-webui/blob/master/README.md): Web UI setup reference. 8379385f7bSGeorge Keishing - [Corporate CLA and Individual CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server): Submitting changes via Gerrit server 842de157e4SGeorge Keishing 858c7eec37SGeorge Keishing## OpenBMC Test Documentation ## 868c7eec37SGeorge Keishing 878c7eec37SGeorge Keishing - [Tools](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/openbmc_test_tools.md): Reference information for helper tools. 888c7eec37SGeorge Keishing - [Code Update](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/code_update.md): Currently supported BMC and PNOR update. 898c7eec37SGeorge Keishing - [Certificate Generate](https://github.com/openbmc/openbmc-test-automation/blob/master/docs/certificate_generate.md): Steps to create and install CA signed certificate. 908c7eec37SGeorge Keishing 91*87a3ff53SGeorge Keishing## Supported Systems Architecture ## 92*87a3ff53SGeorge Keishing 93*87a3ff53SGeorge Keishing OpenBMC test infrastructure is proven capable of running on: 94*87a3ff53SGeorge Keishing - POWER 95*87a3ff53SGeorge Keishing - x86 96*87a3ff53SGeorge Keishing systems running OpenBMC firmware stack. 97*87a3ff53SGeorge Keishing 983df030faSGeorge Keishing## Testing Setup Steps ## 993df030faSGeorge Keishing 1003df030faSGeorge KeishingTo verify the installation setup is completed and ready to execute. 1013df030faSGeorge Keishing 1023df030faSGeorge Keishing* Download the openbmc-test-automation repository: 1033df030faSGeorge Keishing ``` 1043df030faSGeorge Keishing $ git clone https://github.com/openbmc/openbmc-test-automation 1053df030faSGeorge Keishing $ cd openbmc-test-automation 1063df030faSGeorge Keishing ``` 1073df030faSGeorge Keishing* Execute basic setup test run: 1083df030faSGeorge Keishing ``` 1093df030faSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx templates/test_openbmc_setup.robot 1103df030faSGeorge Keishing ``` 1113df030faSGeorge Keishing where xx.xx.xx.xx is the BMC hostname or IP. 112a5c6e9d6SGeorge Keishing 1135977d2aeSGeorge Keishing## Test Layout ## 1145977d2aeSGeorge Keishing 1155977d2aeSGeorge KeishingThere are several sub-directories within the openbmc-test-automation base which 1165977d2aeSGeorge Keishingcontain test suites, tools, templates, etc. These sub-directories are 1175977d2aeSGeorge Keishingclassified as follows: 1185977d2aeSGeorge Keishing 1195977d2aeSGeorge Keishing`tests/`: Contains the general test cases for OpenBMC stack functional 12006d35d7dSGeorge Keishing verification. The "tests" subdirectory uses legacy REST and will be 12106d35d7dSGeorge Keishing deprecated at some point and therefore no longer supported. 1225977d2aeSGeorge Keishing 123f07f5b13SGeorge Keishing`extended/`: Contains test cases for boot testing, code update testing using legacy REST, etc. 1245977d2aeSGeorge Keishing 125b7ec2790SGeorge Keishing`systest/`: Contains test cases for HTX bootme testing. 1265977d2aeSGeorge Keishing 1275977d2aeSGeorge Keishing`xcat/`: Contains test cases for XCAT automation. 1285977d2aeSGeorge Keishing 12929814f0eSGeorge Keishing`gui/`: Contains test cases for web UI and security scanning tool automation. 13029814f0eSGeorge Keishing 131cb858f7aSGeorge Keishing`pldm/`: Contains test cases for platform management subsystem (base, bios, fru, platform, OEM). 132cb858f7aSGeorge Keishing 1330465b9cdSGeorge Keishing`snmp/`: Contains test cases for SNMP (Simple Network Management Protocol) 1340465b9cdSGeorge Keishing configuration testing. 1350465b9cdSGeorge Keishing 1360465b9cdSGeorge Keishing`openpower/ras/`: Contains test cases for RAS (Reliability, Availability and 1370465b9cdSGeorge Keishing Serviceability) for an OpenPOWER system. 1380465b9cdSGeorge Keishing 1390465b9cdSGeorge Keishing`openpower/secureboot/`: Contains test cases for secure boot testing on a 1400465b9cdSGeorge Keishing secure boot feature enabled OpenPOWER system only. 14138f566bbSGeorge Keishing 1425977d2aeSGeorge Keishing`tools/`: Contains various tools. 1435977d2aeSGeorge Keishing 1445977d2aeSGeorge Keishing`templates/`: Contains sample code examples and setup testing. 145acb3c1efSGeorge Keishing 146aba958dcSGeorge Keishing`test_list/`: Contains the argument files used for skipping test cases 147aba958dcSGeorge Keishing (e.g "skip_test", "skip_test_extended", etc.) or 148aba958dcSGeorge Keishing grouping them (e.g "HW_CI", "CT_basic_run", etc.). 149aba958dcSGeorge Keishing 150acb3c1efSGeorge Keishing 151acb3c1efSGeorge Keishing## Redfish Test Layout ## 152acb3c1efSGeorge Keishing 153acb3c1efSGeorge KeishingOpenBMC is moving steadily towards DTMF Redfish, which is an open industry standard 154acb3c1efSGeorge Keishingspecification and schema that meets the expectations of end users for simple, 155acb3c1efSGeorge Keishingmodern and secure management of scalable platform hardware. 156acb3c1efSGeorge Keishing 157f07f5b13SGeorge Keishing`redfish/`: Contains test cases for DMTF Redfish-related feature supported on OpenBMC. 1583292d4acSGeorge Keishing 159f32ee4cbSGeorge Keishing`redfish/extended/`: Contains test cases for combined legacy REST and DMTF Redfish-related 160f32ee4cbSGeorge Keishing feature supported on OpenBMC. 161f32ee4cbSGeorge Keishing 162*87a3ff53SGeorge KeishingNote: Work in progress test developement parameter `-v REDFISH_SUPPORT_TRANS_STATE:1` to force 163*87a3ff53SGeorge Keishing the test suites to execute in redfish mode only. 164acb3c1efSGeorge Keishing 1655977d2aeSGeorge Keishing 166a5c6e9d6SGeorge Keishing## Quickstart ## 1675977d2aeSGeorge KeishingTo run openbmc-automation first you need to install the prerequisite Python 1685977d2aeSGeorge Keishingpackages which will help to invoke tests through tox (Note that tox 1695977d2aeSGeorge Keishingversion 2.3.1 or greater is required) or via Robot CLI command. 1705977d2aeSGeorge Keishing 1715977d2aeSGeorge Keishing**Robot Command Line** 1725977d2aeSGeorge Keishing 1735977d2aeSGeorge Keishing* Execute all test suites for `tests/`: 1745977d2aeSGeorge Keishing ``` 1755977d2aeSGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx tests 1765977d2aeSGeorge Keishing ``` 177aba958dcSGeorge Keishing 1785977d2aeSGeorge Keishing* Execute a test suite: 1795977d2aeSGeorge Keishing ``` 180088da102SGeorge Keishing $ robot -v OPENBMC_HOST:xx.xx.xx.xx redfish/extended/test_basic_ci.robot 1815977d2aeSGeorge Keishing ``` 182b29d2e84SChris Austen 18349a80604SGeorge Keishing* Initialize the following test variables which will be used during test execution: 18449a80604SGeorge Keishing 18549a80604SGeorge Keishing User can forward declare as environment variables: 186a7c46e39SGeorge Keishing ``` 18749a80604SGeorge Keishing $ export OPENBMC_HOST=<openbmc machine IP address/hostname> 1887a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 18949a80604SGeorge Keishing $ export OPENBMC_PASSWORD=<openbmc password> 1907a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 19149a80604SGeorge Keishing ``` 19249a80604SGeorge Keishing 19349a80604SGeorge Keishing or 19449a80604SGeorge Keishing 19549a80604SGeorge Keishing User can input as robot variables as part of the CLI command: 19649a80604SGeorge Keishing ``` 19749a80604SGeorge Keishing -v OPENBMC_HOST:<openbmc machine IP address/hostname> 19849a80604SGeorge Keishing -v OPENBMC_USERNAME:<openbmc username> 19949a80604SGeorge Keishing -v OPENBMC_PASSWORD:<openbmc password> 200359f8039SGeorge Keishing ``` 201b29d2e84SChris Austen 202a7c46e39SGeorge Keishing* For QEMU tests, set the following environment variables as well: 203a7c46e39SGeorge Keishing ``` 2044d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 2054d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 206a7c46e39SGeorge Keishing ``` 207b29d2e84SChris Austen 208a7c46e39SGeorge Keishing* Run tests: 209a7c46e39SGeorge Keishing ``` 2105e021124SAlvin Wang $ tox tests 211b29d2e84SChris Austen ``` 212b29d2e84SChris Austen 213a7c46e39SGeorge Keishing* How to run individual test: 21488acd8daSGeorge Keishing 215a7c46e39SGeorge Keishing One specific test: 216a7c46e39SGeorge Keishing ``` 217a7c46e39SGeorge Keishing $ tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 218a7c46e39SGeorge Keishing ``` 219a7c46e39SGeorge Keishing No preset environment variables, default configuration for all supported 220a7c46e39SGeorge Keishing systems: 221a7c46e39SGeorge Keishing ``` 22288acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 223a7c46e39SGeorge Keishing ``` 224a7c46e39SGeorge Keishing No preset environment variables, one test case from a test suite: 225a7c46e39SGeorge Keishing ``` 226a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --include Power_On_Test tests/test_basic_poweron.robot 227a7c46e39SGeorge Keishing ``` 228a7c46e39SGeorge Keishing No preset environment variables, the entire test suite: 229a7c46e39SGeorge Keishing ``` 230a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 231a7c46e39SGeorge Keishing ``` 232a7c46e39SGeorge Keishing 23308fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 234a7c46e39SGeorge Keishing cases using argument file: 235a7c46e39SGeorge Keishing ``` 236a7c46e39SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/skip_test tests 237a7c46e39SGeorge Keishing ``` 238ddf0cc2aSGeorge Keishing 239ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 240a7c46e39SGeorge Keishing ``` 241ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 242ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 243b29d2e84SChris Austen ``` 244b29d2e84SChris Austen 2457eea8ed0SRahul Maheshwari 2467eea8ed0SRahul Maheshwari* Run IPMI tests: 2477eea8ed0SRahul Maheshwari 2487eea8ed0SRahul Maheshwari Running only out-of-band IPMI tests: 2497eea8ed0SRahul Maheshwari ``` 2507eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:External -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/witherspoon/skip_inband_ipmi tests/ipmi/ 2517eea8ed0SRahul Maheshwari ``` 2527eea8ed0SRahul Maheshwari 2537eea8ed0SRahul Maheshwari Running only inband IPMI tests: 2547eea8ed0SRahul Maheshwari ``` 2557eea8ed0SRahul Maheshwari $ robot -v IPMI_COMMAND:Inband -v OPENBMC_HOST:x.x.x.x -v OS_HOST:x.x.x.x -v OS_USERNAME:xxxx -v OS_PASSWORD:xxxx --argumentfile test_lists/witherspoon/skip_oob_ipmi tests/ipmi/ 2567eea8ed0SRahul Maheshwari ``` 2577eea8ed0SRahul Maheshwari 2587eea8ed0SRahul Maheshwari 259dd38aef3SRahul Maheshwari* Run GUI tests: 260dd38aef3SRahul Maheshwari 261dd38aef3SRahul Maheshwari By default, GUI runs with Firefox browser and headless mode. Example with chrome browser and header mode: 262dd38aef3SRahul Maheshwari ``` 263eeb93c31SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x -v GUI_BROWSER:gc -v GUI_MODE:header gui/test/ 264dd38aef3SRahul Maheshwari ``` 265dd38aef3SRahul Maheshwari 2662bd32ff1SRahul Maheshwari Run GUI default CI test bucket: 2672bd32ff1SRahul Maheshwari ``` 268eeb93c31SRahul Maheshwari $ robot -v OPENBMC_HOST:x.x.x.x --argumentfile test_lists/BMC_WEB_CI gui/test/ 2692bd32ff1SRahul Maheshwari ``` 2702bd32ff1SRahul Maheshwari 2715d00a3c8SSivas SRR* Run LDAP tests: 2725d00a3c8SSivas SRR 2735d00a3c8SSivas SRR Before using LDAP test functions, be sure appropriate LDAP user(s) and group(s) have been created on your LDAP server. 2745d00a3c8SSivas SRR Note: There are multiple ways to create LDAP users / groups and all depend on your LDAP server. 2755d00a3c8SSivas SRR One common way for openldap is ldapadd / ldapmodify refer https://linux.die.net/man/1/ldapadd 2765d00a3c8SSivas SRR For ldapsearch, refer to "https://linux.die.net/man/1/ldapsearch". 2775d00a3c8SSivas SRR Microsoft ADS: refer to https://searchwindowsserver.techtarget.com/definition/Microsoft-Active-Directory-Domain-Services-AD-DS 278962ff298SSivas SRR 279962ff298SSivas SRR Note: Currently, LDAP test automation for Redfish API is in progress. The format to invoke LDAP test is as follows: 2805d00a3c8SSivas SRR ``` 281962ff298SSivas SRR $ cd redfish/account_service/ 282962ff298SSivas SRR $ robot -v OPENBMC_HOST:x.x.x.x -v LDAP_SERVER_URI:<ldap(s)//LDAP Hostname / IP> -v LDAP_BIND_DN:<LDAP Bind DN> -v LDAP_BASE_DN:<LDAP Base DN> -v LDAP_BIND_DN_PASSWORD:<LDAP Bind password> -v LDAP_SEARCH_SCOPE:<LDAP search scope> -v LDAP_SERVER_TYPE:<LDAP server type> -v LDAP_USER:<LDAP user-id> -v LDAP_USER_PASSWORD:<LDAP PASSWORD> -v GROUP_NAME:<Group Name> -v GROUP_PRIVILEGE:<Privilege> ./test_ldap_configuration.robot 2835d00a3c8SSivas SRR ``` 2845d00a3c8SSivas SRR 285d7c26840SGeorge Keishing* How to run CI and CT bucket test: 286d7c26840SGeorge Keishing 287d7c26840SGeorge Keishing Default CI test bucket list: 288d7c26840SGeorge Keishing ``` 289d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 290d7c26840SGeorge Keishing ``` 291d7c26840SGeorge Keishing 292d7c26840SGeorge Keishing Default CI smoke test bucket list: 293d7c26840SGeorge Keishing ``` 294d7c26840SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/CT_basic_run tests 295d7c26840SGeorge Keishing ``` 296d7c26840SGeorge Keishing 297aba958dcSGeorge Keishing* Run extended tests: 298aba958dcSGeorge Keishing 299aba958dcSGeorge Keishing For-loop test (default iteration is 10): 300aba958dcSGeorge Keishing ``` 301aba958dcSGeorge Keishing $ robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 302aba958dcSGeorge Keishing ``` 303aba958dcSGeorge Keishing 304aba958dcSGeorge Keishing Example using tox testing a test suite for 5 iterations "witherspoon": 305aba958dcSGeorge Keishing ``` 306aba958dcSGeorge Keishing OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=witherspoon tox -e witherspoon -- ./extended/full_suite_regression.robot 307aba958dcSGeorge Keishing ``` 308aba958dcSGeorge Keishing 309d61a64baSKonstantin Aladyshev* Host CPU architecture 310d61a64baSKonstantin Aladyshev 311d61a64baSKonstantin Aladyshev By default openbmc-test-automation framework assumes that host CPU is based on the POWER architecture. 312c056c2bbSKonstantin Aladyshev If your host CPU is x86 add `-v PLATFORM_ARCH_TYPE:x86` variable setting to your CLI commands or set an environment variable: 313c056c2bbSKonstantin Aladyshev ``` 314c056c2bbSKonstantin Aladyshev $ export PLATFORM_ARCH_TYPE=x86 315c056c2bbSKonstantin Aladyshev ``` 316d61a64baSKonstantin Aladyshev 317aba958dcSGeorge Keishing**Jenkins jobs tox commands** 318aba958dcSGeorge Keishing* HW CI tox command: 319aba958dcSGeorge Keishing ``` 320aba958dcSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- --argumentfile test_lists/HW_CI tests 321aba958dcSGeorge Keishing ``` 322