1b29d2e84SChris Austen#openbmc-automation 2b29d2e84SChris Austen 3b29d2e84SChris AustenQuickstart 4b29d2e84SChris Austen---------- 5b29d2e84SChris Austen 6b29d2e84SChris AustenTo run openbmc-automation first you need to install the prerequisite python 74319d194SJay Azurinpackages which will help to invoke tests through tox. Note that tox 84319d194SJay Azurinversion 2.3.1 or greater is required. 9b29d2e84SChris Austen 10b29d2e84SChris AustenInstall the python dependencies for tox 11b29d2e84SChris Austen```shell 124319d194SJay Azurin $ easy_install tox 13b29d2e84SChris Austen $ easy_install pip 14b29d2e84SChris Austen``` 15b29d2e84SChris Austen 16ee1d38c3SGunnar MillsInitialize the following environment variable which will be used during testing 17b29d2e84SChris Austen```shell 18b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 197a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 207a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 21b29d2e84SChris Austen $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc] 227a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 237a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 24359f8039SGeorge Keishing``` 25b29d2e84SChris Austen 26359f8039SGeorge KeishingThere are two different set of test suite existing based on the usage. 27359f8039SGeorge KeishingThe test suites are distinctly separated by directory as under 28359f8039SGeorge Keishing tests/ 29359f8039SGeorge Keishing extended/ 30b29d2e84SChris Austen 31359f8039SGeorge Keishing`tests`: directory contains the general test cases 32359f8039SGeorge Keishing 33359f8039SGeorge Keishing`extended`: directory contains the use cases for new IP network testing, PDU, 34359f8039SGeorge KeishingBIOS and BMC code update. 35359f8039SGeorge Keishing 36359f8039SGeorge Keishing```shell 37b29d2e84SChris AustenUse Following Variables for networking test cases 38b29d2e84SChris Austen=========================================================== 39b29d2e84SChris Austen $export NEW_BMC_IP=<openbmc machine ip address> 40b29d2e84SChris Austen $export NEW_SUBNET_MASK=<openbmc new subnet mask> 41b29d2e84SChris Austen $export NEW_GATEWAY=<openbmc new gateway> 42b29d2e84SChris Austen========================================================== 43b29d2e84SChris Austen 44b29d2e84SChris Austen Use following parameters for PDU: 45b29d2e84SChris Austen $ export PDU_IP=<PDU IP address> 46b29d2e84SChris Austen $ export PDU_USERNAME=<PDU username> 47b29d2e84SChris Austen $ export PDU_PASSWORD=<PDU password> 48b29d2e84SChris Austen $ export PDU_TYPE=<PDU type> 49b29d2e84SChris Austen $ export PDU_SLOT_NO=<SLOT number> 50b29d2e84SChris Austen 51b29d2e84SChris Austen for PDU_TYPE we support only synaccess at the moment 52b29d2e84SChris Austen 53b29d2e84SChris AustenUse following variables for syslog test cases 54b29d2e84SChris Austen========================================================== 55b29d2e84SChris Austen $ export SYSLOG_IP_ADDRESS=<remote syslog system ip> 56b29d2e84SChris Austen $ export SYSLOG_PORT=<remote syslog system port> 57b29d2e84SChris Austen 584d6c1dabSGeorge KeishingUse the following variables for Qemu test run 594d6c1dabSGeorge Keishing========================================================== 604d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 614d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 624d6c1dabSGeorge Keishing 63e4c52eb2SJay AzurinUse the following variables for BIOS update testing 64e4c52eb2SJay Azurin========================================================== 65e4c52eb2SJay Azurin $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor 66e4c52eb2SJay Azurin 67b29d2e84SChris Austen``` 68b29d2e84SChris Austen 69b29d2e84SChris AustenRun tests 70b29d2e84SChris Austen```shell 71b29d2e84SChris Austen $ tox -e tests 72b29d2e84SChris Austen``` 73b29d2e84SChris Austen 74b29d2e84SChris AustenHow to test individual test 75b29d2e84SChris Austen```shell 76b29d2e84SChris Austen One specific test 77b29d2e84SChris Austen $ tox -e custom -- -t '"DIMM0 no fault"' tests/test_sensors.robot 78b29d2e84SChris Austen 79b29d2e84SChris Austen No preset environment variables, one test case from one test suite 80b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot 81b29d2e84SChris Austen 82b29d2e84SChris Austen No preset environment variables, one test suite for a palmetto system 83b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot 84b29d2e84SChris Austen 85b29d2e84SChris Austen No preset environment variables, the entire test suite for a barreleye system 86b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests 8708fd617cSRahul Maheshwari 88ddf0cc2aSGeorge Keishing Default CI test bucket list: 8908fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 90f8119103SRahul Maheshwari cases using argument file. 91f8119103SRahul Maheshwari $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/skip_test tests 92ddf0cc2aSGeorge Keishing 93ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 94ddf0cc2aSGeorge Keishing Barrleye: test_lists/skip_test_barreleye 95ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 96ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 97b29d2e84SChris Austen``` 98b29d2e84SChris Austen 99ee1d38c3SGunnar MillsIt can also be run by passing variables from the cli... 100b29d2e84SChris Austen```shell 1017a212a09SRahul Maheshwari Run one test suite using using pybot 1027a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot 1037a212a09SRahul Maheshwari 1047a212a09SRahul Maheshwari Run entire test suite using using pybot 1057a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests 1067a212a09SRahul Maheshwari 1077a212a09SRahul Maheshwari Run entire test suite using external ipmitool 1087a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests 109b29d2e84SChris Austen``` 110359f8039SGeorge Keishing 111359f8039SGeorge KeishingRun extended tests 112359f8039SGeorge Keishing```shell 113359f8039SGeorge Keishing Set the preset environment variables, run test suite for a barreleye system 114359f8039SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot 115359f8039SGeorge Keishing 116359f8039SGeorge Keishing Similarly for Network, PDU and update BIOS 117359f8039SGeorge Keishing 118359f8039SGeorge Keishing For BMC code update, download the system type *.all.tar image from https://openpower.xyz 119359f8039SGeorge Keishing and run as follows: 120359f8039SGeorge Keishing 121359f8039SGeorge Keishing For Barreleye system 122359f8039SGeorge Keishing python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar extended/code_update/update_bmc.robot 123594a6c9dSSivas SRR 124594a6c9dSSivas SRR For loop test (Default iteration is 10) 125594a6c9dSSivas SRR python -m robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 126594a6c9dSSivas SRR Below is sample command using tox to test only fw version using Barreleye system for 5 times 127594a6c9dSSivas SRR OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot 128359f8039SGeorge Keishing``` 12997651c75SGeorge Keishing 13097651c75SGeorge KeishingJenkins jobs tox commands 13197651c75SGeorge Keishing```shell 13297651c75SGeorge Keishing HW CI tox command 13397651c75SGeorge Keishing Set the preset environment variables, run HW CI test for a barreleye system 134f8119103SRahul Maheshwari $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/HW_CI tests 13597651c75SGeorge Keishing 13697651c75SGeorge Keishing``` 1374cae87daSMichael Walsh 1384cae87daSMichael WalshTemplate to be used to create new stand-alone python programs. 1394cae87daSMichael Walsh```shell 1404cae87daSMichael Walsh 1414cae87daSMichael WalshIf you wish to create a new python stand-alone program, please copy bin/python_pgm_template to your new name and then begin your work. Example: 1424cae87daSMichael Walsh 1434cae87daSMichael Walshcd bin 1444cae87daSMichael Walshcp python_pgm_template my_new_program 1454cae87daSMichael Walsh 1464cae87daSMichael WalshThis template has much of your preliminary work done for you and it will help us all follow a similar structure. 1474cae87daSMichael Walsh 1484cae87daSMichael WalshFeatures: 1494cae87daSMichael Walsh- Help text and argparsing started for you. 1504cae87daSMichael Walsh- Support for "stock" parameters like quiet, debug, test_mode. 1514cae87daSMichael Walsh- exit_function and signal_handler defined. 1524cae87daSMichael Walsh- validate_parms function pre-created. 1534cae87daSMichael Walsh- main function follows conventional startup: 1544cae87daSMichael Walsh 1554cae87daSMichael Walsh if not gen_get_options(parser, stock_list): 1564cae87daSMichael Walsh return False 1574cae87daSMichael Walsh 1584cae87daSMichael Walsh if not validate_parms(): 1594cae87daSMichael Walsh return False 1604cae87daSMichael Walsh 1614cae87daSMichael Walsh qprint_pgm_header() 1624cae87daSMichael Walsh 1634cae87daSMichael Walsh # Your code here. 1644cae87daSMichael Walsh 1654cae87daSMichael Walsh``` 166a464a7ceSSivas SRR 167a464a7ceSSivas SRRCommand to get GitHub issues (any GitHub repository) report in CSV format 1689fe1b12eSSivas SRRNote: On Prompt "Enter your GitHub Password:" enter your GitHub password. 169a464a7ceSSivas SRR```shell 1709fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> <github repo> 171a464a7ceSSivas SRR 172a464a7ceSSivas SRRExample for getting openbmc issues 1739fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> openbmc/openbmc 174a464a7ceSSivas SRR 175a464a7ceSSivas SRRExample for getting openbmc-test-automation issues 1769fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> openbmc/openbmc-test-automation 177a464a7ceSSivas SRR``` 178*4470957eSSivas SRR 179*4470957eSSivas SRR 180*4470957eSSivas SRRCommand to generate Robot test cases test documentations 181*4470957eSSivas SRR 182*4470957eSSivas SRR```shell 183*4470957eSSivas SRR./tools/generate_test_document <Robot test cases directory path> <test case document file path> 184*4470957eSSivas SRR 185*4470957eSSivas SRRExample for generating tests cases documentation for tests directory 186*4470957eSSivas SRR./tools/generate_test_document tests testsdirectoryTCdocs.html 187*4470957eSSivas SRR 188*4470957eSSivas SRRExample for generating tests cases documentation (tests,gui,extended TCs) 189*4470957eSSivas SRR# Note: Invoke the tool with out argument 190*4470957eSSivas SRR./tools/generate_test_document 191*4470957eSSivas SRR``` 192