1b29d2e84SChris Austen#openbmc-automation 2b29d2e84SChris Austen 3b29d2e84SChris AustenQuickstart 4b29d2e84SChris Austen---------- 5b29d2e84SChris Austen 6b29d2e84SChris AustenTo run openbmc-automation first you need to install the prerequisite python 74319d194SJay Azurinpackages which will help to invoke tests through tox. Note that tox 84319d194SJay Azurinversion 2.3.1 or greater is required. 9b29d2e84SChris Austen 10b29d2e84SChris AustenInstall the python dependencies for tox 11b29d2e84SChris Austen```shell 124319d194SJay Azurin $ easy_install tox 13b29d2e84SChris Austen $ easy_install pip 14b29d2e84SChris Austen``` 15b29d2e84SChris Austen 16ee1d38c3SGunnar MillsInitialize the following environment variable which will be used during testing 17b29d2e84SChris Austen```shell 18b29d2e84SChris Austen $ export OPENBMC_HOST=<openbmc machine ip address> 197a212a09SRahul Maheshwari $ export OPENBMC_PASSWORD=<openbmc password> 207a212a09SRahul Maheshwari $ export OPENBMC_USERNAME=<openbmc username> 21b29d2e84SChris Austen $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc] 227a212a09SRahul Maheshwari $ export IPMI_COMMAND=<Dbus/External> 237a212a09SRahul Maheshwari $ export IPMI_PASSWORD=<External IPMI password> 24359f8039SGeorge Keishing``` 25b29d2e84SChris Austen 26359f8039SGeorge KeishingThere are two different set of test suite existing based on the usage. 27359f8039SGeorge KeishingThe test suites are distinctly separated by directory as under 28359f8039SGeorge Keishing tests/ 29359f8039SGeorge Keishing extended/ 30b29d2e84SChris Austen 31359f8039SGeorge Keishing`tests`: directory contains the general test cases 32359f8039SGeorge Keishing 33359f8039SGeorge Keishing`extended`: directory contains the use cases for new IP network testing, PDU, 34359f8039SGeorge KeishingBIOS and BMC code update. 35359f8039SGeorge Keishing 36359f8039SGeorge Keishing```shell 37b29d2e84SChris AustenUse Following Variables for networking test cases 38b29d2e84SChris Austen=========================================================== 39b29d2e84SChris Austen $export NEW_BMC_IP=<openbmc machine ip address> 40b29d2e84SChris Austen $export NEW_SUBNET_MASK=<openbmc new subnet mask> 41b29d2e84SChris Austen $export NEW_GATEWAY=<openbmc new gateway> 42b29d2e84SChris Austen========================================================== 43b29d2e84SChris Austen 44b29d2e84SChris Austen Use following parameters for PDU: 45b29d2e84SChris Austen $ export PDU_IP=<PDU IP address> 46b29d2e84SChris Austen $ export PDU_USERNAME=<PDU username> 47b29d2e84SChris Austen $ export PDU_PASSWORD=<PDU password> 48b29d2e84SChris Austen $ export PDU_TYPE=<PDU type> 49b29d2e84SChris Austen $ export PDU_SLOT_NO=<SLOT number> 50b29d2e84SChris Austen 51b29d2e84SChris Austen for PDU_TYPE we support only synaccess at the moment 52b29d2e84SChris Austen 53b29d2e84SChris AustenUse following variables for syslog test cases 54b29d2e84SChris Austen========================================================== 55b29d2e84SChris Austen $ export SYSLOG_IP_ADDRESS=<remote syslog system ip> 56b29d2e84SChris Austen $ export SYSLOG_PORT=<remote syslog system port> 57b29d2e84SChris Austen 584d6c1dabSGeorge KeishingUse the following variables for Qemu test run 594d6c1dabSGeorge Keishing========================================================== 604d6c1dabSGeorge Keishing $ export SSH_PORT=<ssh port number> 614d6c1dabSGeorge Keishing $ export HTTPS_PORT=<https port number> 624d6c1dabSGeorge Keishing 63e4c52eb2SJay AzurinUse the following variables for BIOS update testing 64e4c52eb2SJay Azurin========================================================== 65e4c52eb2SJay Azurin $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor 66e4c52eb2SJay Azurin 67b29d2e84SChris Austen``` 68b29d2e84SChris Austen 69b29d2e84SChris AustenRun tests 70b29d2e84SChris Austen```shell 71b29d2e84SChris Austen $ tox -e tests 72b29d2e84SChris Austen``` 73b29d2e84SChris Austen 74b29d2e84SChris AustenHow to test individual test 75b29d2e84SChris Austen```shell 76b29d2e84SChris Austen One specific test 77*88acd8daSGeorge Keishing $ tox -e default -- -t '"DIMM0 no fault"' tests/test_sensors.robot 78*88acd8daSGeorge Keishing 79*88acd8daSGeorge Keishing No preset environment variables, default configuration for all supported systems. 80*88acd8daSGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e default -- tests 81b29d2e84SChris Austen 82b29d2e84SChris Austen No preset environment variables, one test case from one test suite 83b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot 84b29d2e84SChris Austen 85b29d2e84SChris Austen No preset environment variables, one test suite for a palmetto system 86b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot 87b29d2e84SChris Austen 88b29d2e84SChris Austen No preset environment variables, the entire test suite for a barreleye system 89b29d2e84SChris Austen $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests 9008fd617cSRahul Maheshwari 91ddf0cc2aSGeorge Keishing Default CI test bucket list: 9208fd617cSRahul Maheshwari No preset environment variables, the entire test suite excluding test 93f8119103SRahul Maheshwari cases using argument file. 94f8119103SRahul Maheshwari $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/skip_test tests 95ddf0cc2aSGeorge Keishing 96ddf0cc2aSGeorge Keishing Exclude test list for supported systems: 97ddf0cc2aSGeorge Keishing Barrleye: test_lists/skip_test_barreleye 98ddf0cc2aSGeorge Keishing Palmetto: test_lists/skip_test_palmetto 99ddf0cc2aSGeorge Keishing Witherspoon: test_lists/skip_test_witherspoon 100b29d2e84SChris Austen``` 101b29d2e84SChris Austen 102ee1d38c3SGunnar MillsIt can also be run by passing variables from the cli... 103b29d2e84SChris Austen```shell 1047a212a09SRahul Maheshwari Run one test suite using using pybot 1057a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot 1067a212a09SRahul Maheshwari 1077a212a09SRahul Maheshwari Run entire test suite using using pybot 1087a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests 1097a212a09SRahul Maheshwari 1107a212a09SRahul Maheshwari Run entire test suite using external ipmitool 1117a212a09SRahul Maheshwari $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests 112b29d2e84SChris Austen``` 113359f8039SGeorge Keishing 114359f8039SGeorge KeishingRun extended tests 115359f8039SGeorge Keishing```shell 116359f8039SGeorge Keishing Set the preset environment variables, run test suite for a barreleye system 117359f8039SGeorge Keishing $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot 118359f8039SGeorge Keishing 119359f8039SGeorge Keishing Similarly for Network, PDU and update BIOS 120359f8039SGeorge Keishing 121359f8039SGeorge Keishing For BMC code update, download the system type *.all.tar image from https://openpower.xyz 122359f8039SGeorge Keishing and run as follows: 123359f8039SGeorge Keishing 124359f8039SGeorge Keishing For Barreleye system 125359f8039SGeorge Keishing python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar extended/code_update/update_bmc.robot 126594a6c9dSSivas SRR 127594a6c9dSSivas SRR For loop test (Default iteration is 10) 128594a6c9dSSivas SRR python -m robot -v OPENBMC_HOST:x.x.x.x -v OPENBMC_SYSTEMMODEL:xxxxxx -v ITERATION:n -v LOOP_TEST_COMMAND:xxxxxx extended/full_suite_regression.robot 129594a6c9dSSivas SRR Below is sample command using tox to test only fw version using Barreleye system for 5 times 130594a6c9dSSivas SRR OPENBMC_HOST=x.x.x.x LOOP_TEST_COMMAND="--argumentfile test_lists/skip_test tests/test_fw_version.robot" ITERATION=5 OPENBMC_SYSTEMMODEL=barreleye tox -e barreleye -- ./extended/full_suite_regression.robot 131359f8039SGeorge Keishing``` 13297651c75SGeorge Keishing 13397651c75SGeorge KeishingJenkins jobs tox commands 13497651c75SGeorge Keishing```shell 13597651c75SGeorge Keishing HW CI tox command 13697651c75SGeorge Keishing Set the preset environment variables, run HW CI test for a barreleye system 137f8119103SRahul Maheshwari $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile test_lists/HW_CI tests 13897651c75SGeorge Keishing 13997651c75SGeorge Keishing``` 1404cae87daSMichael Walsh 1414cae87daSMichael WalshTemplate to be used to create new stand-alone python programs. 1424cae87daSMichael Walsh```shell 1434cae87daSMichael Walsh 1444cae87daSMichael WalshIf you wish to create a new python stand-alone program, please copy bin/python_pgm_template to your new name and then begin your work. Example: 1454cae87daSMichael Walsh 1464cae87daSMichael Walshcd bin 1474cae87daSMichael Walshcp python_pgm_template my_new_program 1484cae87daSMichael Walsh 1494cae87daSMichael WalshThis template has much of your preliminary work done for you and it will help us all follow a similar structure. 1504cae87daSMichael Walsh 1514cae87daSMichael WalshFeatures: 1524cae87daSMichael Walsh- Help text and argparsing started for you. 1534cae87daSMichael Walsh- Support for "stock" parameters like quiet, debug, test_mode. 1544cae87daSMichael Walsh- exit_function and signal_handler defined. 1554cae87daSMichael Walsh- validate_parms function pre-created. 1564cae87daSMichael Walsh- main function follows conventional startup: 1574cae87daSMichael Walsh 1584cae87daSMichael Walsh if not gen_get_options(parser, stock_list): 1594cae87daSMichael Walsh return False 1604cae87daSMichael Walsh 1614cae87daSMichael Walsh if not validate_parms(): 1624cae87daSMichael Walsh return False 1634cae87daSMichael Walsh 1644cae87daSMichael Walsh qprint_pgm_header() 1654cae87daSMichael Walsh 1664cae87daSMichael Walsh # Your code here. 1674cae87daSMichael Walsh 1684cae87daSMichael Walsh``` 169a464a7ceSSivas SRR 170a464a7ceSSivas SRRCommand to get GitHub issues (any GitHub repository) report in CSV format 1719fe1b12eSSivas SRRNote: On Prompt "Enter your GitHub Password:" enter your GitHub password. 172a464a7ceSSivas SRR```shell 1739fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> <github repo> 174a464a7ceSSivas SRR 175a464a7ceSSivas SRRExample for getting openbmc issues 1769fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> openbmc/openbmc 177a464a7ceSSivas SRR 178a464a7ceSSivas SRRExample for getting openbmc-test-automation issues 1799fe1b12eSSivas SRRpython ./tools/github_issues_to_csv <github user> openbmc/openbmc-test-automation 180a464a7ceSSivas SRR``` 1814470957eSSivas SRR 1824470957eSSivas SRR 1834470957eSSivas SRRCommand to generate Robot test cases test documentations 1844470957eSSivas SRR 1854470957eSSivas SRR```shell 1864470957eSSivas SRR./tools/generate_test_document <Robot test cases directory path> <test case document file path> 1874470957eSSivas SRR 1884470957eSSivas SRRExample for generating tests cases documentation for tests directory 1894470957eSSivas SRR./tools/generate_test_document tests testsdirectoryTCdocs.html 1904470957eSSivas SRR 1914470957eSSivas SRRExample for generating tests cases documentation (tests,gui,extended TCs) 1924470957eSSivas SRR# Note: Invoke the tool with out argument 1934470957eSSivas SRR./tools/generate_test_document 1944470957eSSivas SRR``` 195