1Contributing to OpenBMC Test Automation 2======================================= 3Guide to working on OpenBMC test automation. This document will always be a 4work-in-progress, feel free to propose changes. 5 6Submitting changes via Gerrit server 7------------------------------------ 8- Reference [OpenBMC CLA signers](https://github.com/openbmc/openbmc-tools/blob/master/emilyshaffer/cla-signers/cla-signers) 9- Reference [OpenBMC docs](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server) 10 11Robot Coding Guidelines 12----------------------- 13- For this project, we will write Robot keyword definitions in either Robot 14 or Python. Robot code should be quite simple. Therefore, if the algorithm 15 in question is the least bit complex, please write it in Python. 16- Observe a maximum line length of 110 characters. 17- Avoid trailing space at the end of any line of Robot code. 18- Avoid the use of tabs. 19- Robot supports delimiting cells with either two or more spaces or with a 20 pipe symbol (e.g. "\|"). Our team has chosen to use spaces rather than the 21 pipe character. Make sure all space delimiters in Robot code are the 22 **minimum** of two spaces. There may be some exceptions to this rule. 23 24 Exceptions to two-space delimiter rule: 25 - When you wish to line up resource, library or variable values: 26 ``` 27 Library Lib1 28 Resource Resource1 29 *** Variables *** 30 ${var1} ${EMPTY} 31 ``` 32 - When you wish to line up fields for test templates: 33 ``` 34 [Template] Set System LED State 35 # LED Name LED State 36 power On 37 power Off 38 ``` 39 - When you wish to indent if/else or loop bodies for visual effect: 40 ``` 41 Run Keyword If '${this}' == '${that}' 42 ... Log Bla, bla... 43 ... ELSE 44 ... Run Keywords Key1 parms 45 ... AND Key2 parms 46 ``` 47- Use spaces to make conditions more readable: 48 49 Correct example: 50 ``` 51 Run Keyword If '${var1}' == '${0}' My Keyword 52 ``` 53 Incorrect example: 54 ``` 55 Run Keyword If '${var1}'=='${0}' My Keyword 56 ``` 57- When you define or call a Robot keyword, Robot pays no attention to spaces, 58 underscores or case. However, our team will observe the following 59 conventions in both our definitions and our calls: 60 - Separate words with single spaces. 61 - Capitalize the first character of each word. 62 - Capitalize all characters in any word that is an acronym (e.g. JSON, BMC, 63 etc). 64 65 Examples: 66 ``` 67 *** Keywords *** 68 69 This Is Correct 70 71 # This keyword name is correct. 72 73 this_is_incorrect 74 75 # This keyword name is incorrect because of 1) the 76 # underscores instead of spaces and 2) the failure to 77 # capitalize each word in the keyword. 78 79 soisthis 80 81 # This keyword name is incorrect because of 1) a failure to 82 # separate words with spaces and 2) a failure to capitalize 83 # each word in the keyword. 84 85 BMC Is An Acronym 86 87 # This keyword name is correct. Note that "BMC" is an 88 # acronym and as such is entirely uppercase. 89 ``` 90- Documentation strings: 91 - Each documentation string should be phrased as an **English command**. 92 Punctuate it correctly with the first word capitalized and a period at 93 the end. 94 95 Correct example: 96 ``` 97 Boot BMC 98 [Documentation] Boot the BMC. 99 ``` 100 Incorrect example: 101 ``` 102 Boot BMC 103 [Documentation] This keyword boots the BMC. 104 105 # The doc string above is not phrased as a command. 106 ``` 107 - Doc strings should be just one terse, descriptive sentence. 108 Remember that this doc string shows up in the HTML log file. Put 109 additional commentary below in standard comment lines. 110 111 Correct example: 112 ``` 113 Stop SOL Console Logging 114 115 [Documentation] Stop system console logging and return log output. 116 ``` 117 Incorrect example: 118 ``` 119 Stop SOL Console Logging 120 121 [Documentation] Stop system console logging. If there are multiple 122 ... system console processes, they will all be 123 ... stopped. If there is no existing log file this 124 ... keyword will return an error message to that 125 ... effect (and write that message to targ_file_path, 126 ... if specified). NOTE: This keyword will not fail 127 ... if there is no running system console process. 128 129 # This doc string is way too long. 130 ``` 131- Tags: 132 - Create a tag for every test case with a tag name that mirrors the test case 133 name as follows: 134 ``` 135 Create Intermediate File 136 137 [Tags] Create_Intermediate_File 138 ``` 139- Description of argument(s): 140 - As shown in the following example, if your keyword has any arguments, include 141 a "**Description of argument(s)**" section. This effectively serves as the 142 help text for anyone wanting to use or understand your keyword. Include 143 real data examples wherever possible and applicable. Leave at least 2 spaces 144 between the argument name and the description. Align all description text as 145 shown in the example below. 146 147 Example: 148 ``` 149 Get URL List 150 [Documentation] Return list of URLs under given URL. 151 [Arguments] ${openbmc_url} ${policy} 152 153 # Description of argument(s): 154 # openbmc_url URL for list operation (e.g. 155 # "/xyz/openbmc_project/inventory"). 156 # policy Power restore policy (e.g "RESTORE_LAST_STATE", 157 # ${RESTORE_LAST_STATE}). 158 ``` 159- Variable assignments: 160 161 When assigning a variable as output from a keyword, do not precede the 162 equal sign with a space. 163 164 Correct examples: 165 ``` 166 ${var1}= Set Variable ${1} 167 ${var1}= My Keyword 168 ``` 169 Incorrect examples: 170 171 ``` 172 ${var1} = Set Variable ${1} 173 ${var1} = My Keyword 174 ``` 175- General variable naming conventions: 176 - Variable names should be lower case with few exceptions: 177 - Environment variables should be all upper case. 178 - Variables intended to be set by Robot -v parameters may be all 179 upper case. 180 - Words within a variable name should be separated by underscores: 181 182 Correct examples: 183 ``` 184 ${host_name} 185 ${program_pid} 186 ``` 187 Incorrect examples: 188 ``` 189 ${HostName} 190 ${ProgramPid} 191 ``` 192- Special variable naming conventions. 193 194 For certain very commonly used kinds of variables, please observe these 195 conventions in order to achieve consistency throughout the code. 196 197 - hosts 198 199 When a variable is intended to contain **either** an IP address **or** 200 a host name (either long or short), please give it a suffix of "_host". 201 202 Examples: 203 ``` 204 openbmc_host 205 os_host 206 pdu_host 207 openbmc_serial_host 208 ``` 209 - host names 210 211 For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use 212 a suffix of _host_name. 213 214 Examples: 215 ``` 216 openbmc_host_name 217 os_host_name 218 pdu_host_name 219 openbmc_serial_host_name 220 ``` 221 - Short host names 222 223 For short host names (e.g. "bmc1"), use a suffix of _host_short_name. 224 225 Examples: 226 ``` 227 openbmc_host_short_name 228 os_host_short_name 229 pdu_host_short_name 230 openbmc_serial_host_short_name 231 ``` 232 - IP addresses 233 234 For IP addresses, use a suffix of _ip. 235 236 Example: 237 ``` 238 openbmc_ip 239 os_ip 240 pdu_ip 241 openbmc_serial_ip 242 ``` 243 - Files and directories: 244 - Files: 245 - If your variable is to contain only the file's name, use a suffix 246 of _file_name. 247 248 Examples: 249 ``` 250 ffdc_file_name = "bmc1.170428.120200.ffdc" 251 ``` 252 - If your variable is to contain the path to a file, use a suffix of 253 _file_path. Bear in mind that a file path can be relative or 254 absolute so that should not be a consideration in whether to use 255 the "_file_path" suffix. 256 257 Examples: 258 ``` 259 status_file_path = "bmc1.170428.120200.status" 260 status_file_path = "subdir/bmc1.170428.120200.status" 261 status_file_path = "./bmc1.170428.120200.status" 262 status_file_path = "../bmc1.170428.120200.status" 263 status_file_path = "/home/user1/status/bmc1.170428.120200.status" 264 ``` 265 To re-iterate, it doesn't matter whether the contents of the 266 variable are a relative or absolute path (as shown in the 267 examples above). A file path is simply a value with enough 268 information in it for the program to find the file. 269 270 - If the variable **must** contain an absolute path (which should be 271 the rare case), use a suffix _abs_file_path. 272 273 - Directories: 274 - Directory variables should follow the same conventions as file 275 variables. 276 277 - If your variable is to contain only the directory's name, use a 278 suffix of _dir_name. 279 280 Example: 281 ``` 282 ffdc_dir_name = "ffdc" 283 ``` 284 - If your variable is to contain the path to a directory, use a 285 suffix of _dir_path. Bear in mind that a dir path can be 286 relative or absolute so that should not be a consideration in 287 whether to use _dir_path. 288 289 Examples: 290 ``` 291 status_dir_path = "status/" 292 status_dir_path = "subdir/status" 293 status_dir_path = "./status/" 294 status_dir_path = "../status/" 295 status_dir_path = "/home/user1/status/" 296 ``` 297 To re-iterate, it doesn't matter whether the contents of 298 the variable are a relative or absolute path (as shown in 299 the examples above). A dir path is simply a value with 300 enough information in it for the program to find the 301 directory. 302 303 - If the variable **must** contain an absolute path (which 304 should be the rare case), use a suffix _abs_dir_path. 305 - IMPORTANT: As a programming convention, do pre- 306 processing on all dir_path variables to ensure that they 307 contain a trailing slash. If we follow that convention 308 religiously, then when changes are made in other parts of 309 the program, the programmer can count on the value having 310 a trailing slash. Therefore, they can safely do this kind 311 of thing: 312 ``` 313 my_file_path = my_dir_path + my_file_name 314 ``` 315 - Setup/Teardown keywords 316 317 Use standardized names for setup and teardown keywords: 318 - Suite Setup Execution 319 - Suite Teardown Execution 320 - Test Setup Execution 321 - Test Teardown Execution 322- Traditional comments (i.e. using the hashtag style comments) 323 - Please leave one space following the hashtag. 324 ``` 325 #wrong 326 327 # Right 328 ``` 329 - Please use proper English punctuation: 330 - Capitalize the first word in the sentence or phrase. 331 - End sentences (or stand-alone phrases) with a period. 332 333 - Do not keep commented-out code in your program. Instead, remove it 334 entirely. 335- Robot Template Test Cases 336 - Follow this format for Robot template test cases: 337 338 Note: Documentation, Tags and Template lines are all required and should be coded in the order shown. 339 ``` 340 Test Case Name 341 [Documentation] 342 [Tags] 343 [Template] 344 # arg1 arg2 etc. 345 <arg1> <arg2> 346 347 Example: 348 349 Get Response Codes 350 [Documentation] REST "Get" response status test. 351 [Tags] Get_Response_Codes 352 [Template] Execute Get And Check Response 353 354 # expected_response_code url_path 355 ${HTTP_OK} /org/ 356 ${HTTP_OK} /xyz/ 357 ${HTTP_OK} /xyz/openbmc_project/ 358 ${HTTP_OK} /xyz/openbmc_project/state/enumerate 359 ${HTTP_NOT_FOUND} /xyz/i/dont/exist/ 360 ``` 361 362 Note: Normally, a template test case would have many rows of data arguments as in the example above. 363 However, contributors frequently define multiple template test cases that each have only 364 one row of data which may seem to defeat the value of using templates in the first place. However, 365 it is done for these reasons: 366 1) Template tests are counted as a single test. The user may wish to have separate results for 367 each call to the template function. 368 2) If any call to the template function fails, one would like FFDC data collected immediately 369 and would like one set of FFDC data for EACH such failure. 370 371 372Python Coding Guidelines 373----------------------- 374- The minimum required Python version is 2.7.x. 375- Run pycodestyle on all Python files and correct errors to follow the guidelines in 376 https://www.python.org/dev/peps/pep-0008/. Note that when python code is checked into gerrit, pycodestyle is run automatically on it. 377 378 Example as run from a Linux command line: 379 ``` 380 pycodestyle my_pgm.py 381 382 my_pgm.py:41:1: E302 expected 2 blank lines, found 1 383 my_pgm.py:58:52: W291 trailing whitespace 384 ``` 385- Include doc strings in every function and follow the guidelines in 386 https://www.python.org/dev/peps/pep-0257/. 387 388 Example: 389 ``` 390 r""" 391 Return the function name associated with the indicated stack frame. 392 393 Description of argument(s): 394 stack_frame_ix The index of the stack frame whose 395 function name should be returned. If 396 the caller does not specify a value, 397 this function will set the value to 1 398 which is the index of the caller's 399 stack frame. If the caller is the 400 wrapper function "print_func_name", 401 this function will bump it up by 1. 402 """ 403 ``` 404- As shown in the prior example, if your function has any arguments, include 405 a "Description of argument(s)" section. This effectively serves as the 406 help text for anyone wanting to use or understand your function. Include 407 real data examples wherever possible and applicable. 408- Function definitions: 409 - Put each function parameter on its own line: 410 ``` 411 def func1(parm1, 412 413 parm2): 414 ``` 415- Do not keep commented-out code in your program. Instead, remove it 416 entirely. 417- When you define a python function, observe the following 418 conventions: 419 - Separate words with single underscores. 420 - Use lower-case letters. 421 422 Examples: 423 ``` 424 425 def this_is_correct(): 426 427 # This function name is correct. 428 429 def This_Is_Incorrect(): 430 431 # This keyword name is incorrect because of the upper case letters used. 432 433 def soisthis(): 434 435 # This keyword name is incorrect because of 1) a failure to 436 # separate words with underscores. 437 438 ``` 439- Documentation strings: 440 - Each documentation string should be phrased as an **English command**. 441 Punctuate it correctly with the first word capitalized and a period at 442 the end. 443 444 Correct example: 445 ``` 446 def boot_bmc(): 447 r""" 448 Boot the BMC. 449 """ 450 ``` 451 Incorrect example: 452 ``` 453 def boot_bmc(): 454 r""" 455 This function boots the BMC. 456 """ 457 458 # The doc string above is not phrased as a command. 459 ``` 460 - Doc strings should begin with a summary line which one terse, descriptive sentence. 461 Put additional commentary below. 462 463 Correct example: 464 ``` 465 def stop_sol_console_logging(): 466 r""" 467 Stop system console logging and return log output. 468 469 Additional comments... 470 """ 471 ``` 472 Incorrect example: 473 ``` 474 def stop_sol_console_logging(): 475 r""" 476 Stop system console logging. If there are multiple system console 477 processes, they will all be stopped. If there is no existing log file 478 this keyword will return an error message to that effect (and write that 479 message to targ_file_path, if specified). NOTE: This keyword will not 480 fail if there is no running system console process. 481 """ 482 483 # This doc string is way too long. 484 ``` 485- General variable naming conventions: 486 - Variable names should be lower case with few exceptions: 487 - Environment variables should be all upper case. 488 - Words within a variable name should be separated by underscores: 489 490 Correct examples: 491 ``` 492 ${host_name} 493 ${program_pid} 494 ``` 495 Incorrect examples: 496 ``` 497 ${HostName} 498 ${ProgramPid} 499 ``` 500- Special variable naming conventions. 501 502 For certain very commonly used kinds of variables, please observe these 503 conventions in order to achieve consistency throughout the code. 504 505 - hosts 506 507 When a variable is intended to contain **either** an IP address **or** 508 a host name (either long or short), please give it a suffix of "_host". 509 510 Examples: 511 ``` 512 openbmc_host 513 os_host 514 pdu_host 515 openbmc_serial_host 516 ``` 517 - host names 518 519 For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use 520 a suffix of _host_name. 521 522 Examples: 523 ``` 524 openbmc_host_name 525 os_host_name 526 pdu_host_name 527 openbmc_serial_host_name 528 ``` 529 - Short host names 530 531 For short host names (e.g. "bmc1"), use a suffix of _host_short_name. 532 533 Examples: 534 ``` 535 openbmc_host_short_name 536 os_host_short_name 537 pdu_host_short_name 538 openbmc_serial_host_short_name 539 ``` 540 - IP addresses 541 542 For IP addresses, use a suffix of _ip. 543 544 Example: 545 ``` 546 openbmc_ip 547 os_ip 548 pdu_ip 549 openbmc_serial_ip 550 ``` 551- Files and directories: 552 - Files: 553 - If your variable is to contain only the file's name, use a suffix 554 of _file_name. 555 556 Examples: 557 ``` 558 ffdc_file_name = "bmc1.170428.120200.ffdc" 559 ``` 560 - If your variable is to contain the path to a file, use a suffix of 561 _file_path. Bear in mind that a file path can be relative or 562 absolute so that should not be a consideration in whether to use 563 the "_file_path" suffix. 564 565 Examples: 566 ``` 567 status_file_path = "bmc1.170428.120200.status" 568 status_file_path = "subdir/bmc1.170428.120200.status" 569 status_file_path = "./bmc1.170428.120200.status" 570 status_file_path = "../bmc1.170428.120200.status" 571 status_file_path = "/home/user1/status/bmc1.170428.120200.status" 572 ``` 573 To re-iterate, it doesn't matter whether the contents of the 574 variable are a relative or absolute path (as shown in the 575 examples above). A file path is simply a value with enough 576 information in it for the program to find the file. 577 578 - If the variable **must** contain an absolute path (which should be 579 the rare case), use a suffix _abs_file_path. 580 581 - Directories: 582 - Directory variables should follow the same conventions as file 583 variables. 584 585 - If your variable is to contain only the directory's name, use a 586 suffix of _dir_name. 587 588 Example: 589 ``` 590 ffdc_dir_name = "ffdc" 591 ``` 592 - If your variable is to contain the path to a directory, use a 593 suffix of _dir_path. Bear in mind that a dir path can be 594 relative or absolute so that should not be a consideration in 595 whether to use _dir_path. 596 597 Examples: 598 ``` 599 status_dir_path = "status/" 600 status_dir_path = "subdir/status" 601 status_dir_path = "./status/" 602 status_dir_path = "../status/" 603 status_dir_path = "/home/user1/status/" 604 ``` 605 To re-iterate, it doesn't matter whether the contents of 606 the variable are a relative or absolute path (as shown in 607 the examples above). A dir path is simply a value with 608 enough information in it for the program to find the 609 directory. 610 611 - If the variable **must** contain an absolute path (which 612 should be the rare case), use a suffix _abs_dir_path. 613 - IMPORTANT: As a programming convention, do pre- 614 processing on all dir_path variables to ensure that they 615 contain a trailing slash. If we follow that convention 616 religiously, that when changes are made in other parts of 617 the program, the programmer can count on the value having 618 a trailing slash. Therefore they can safely do this kind 619 of thing: 620 ``` 621 my_file_path = my_dir_path + my_file_name 622 ``` 623- Traditional comments (i.e. using the hashtag style comments) 624 - Please leave one space following the hashtag. 625 ``` 626 #wrong 627 628 # Right 629 ``` 630 - Please use proper English punction: 631 - Capitalize the first word in the sentence or phrase. 632 - End sentences (or stand-alone phrases) with a period. 633 634 - Do not keep commented-out code in your program. Instead, remove it 635 entirely. 636 637Template Usage Guidelines 638------------------------- 639We have several templates in the templates/ sub-directory. If there is a 640template that applies to your programming situation (Python, bash, etc.), 641it should be used to create new programs as in the following example 642 643- Example: 644 645 ``` 646 $ cd templates 647 $ cp python_pgm_template ../bin/my_new_program.py 648 ``` 649 650These templates have much of your preliminary work done for you and will help 651us all follow a similar structure. 652 653- Features: 654 - Help text and arg parsing started for you. 655 - Support for "stock" parameters like "quiet", "debug", "test_mode". 656 - "exit_function" and "signal_handler" defined. 657 - "validate_parms" function pre-created. 658 - "main" function follows conventional startup sequence: 659 660 ``` 661 if not gen_get_options(parser, stock_list): 662 return False 663 664 if not validate_parms(): 665 return False 666 667 qprint_pgm_header() 668 669 # Your code here. 670 ``` 671