1Contributing to OpenBMC Test Automation 2======================================= 3Guide to working on OpenBMC test automation. This document will always be a 4work-in-progress, feel free to propose changes. 5 6Submitting changes via Gerrit server 7------------------------------------ 8- Reference [OpenBMC CLA signers](https://github.com/openbmc/openbmc-tools/blob/master/emilyshaffer/cla-signers/cla-signers) 9- Reference [OpenBMC docs](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server) 10 11Robot Coding Guidelines 12----------------------- 13- For this project, we will write Robot keyword definitions in either Robot 14 or Python. Robot code should be quite simple. Therefore, if the algorithm 15 in question is the least bit complex, please write it in Python. 16- Observe a maximum line length of 79 characters. 17- Avoid trailing space at the end of any line of Robot code. 18- Avoid the use of tabs. 19- Robot supports delimiting cells with either two or more spaces or with a 20 pipe symbol (e.g. "\|"). Our team has chosen to use spaces rather than the 21 pipe character. Make sure all space delimiters in Robot code are the 22 **minimum** of two spaces. There may be some exceptions to this rule. 23 24 Exceptions to two-space delimiter rule: 25 - When you wish to line up resource, library or variable values: 26 ``` 27 Library Lib1 28 Resource Resource1 29 *** Variables *** 30 ${var1} ${EMPTY} 31 ``` 32 - When you wish to line up fields for test templates: 33 ``` 34 [Template] Set System LED State 35 # LED Name LED State 36 power On 37 power Off 38 ``` 39 - When you wish to indent if/else or loop bodies for visual effect: 40 ``` 41 Run Keyword If '${this}' == '${that}' 42 ... Log Bla, bla... 43 ... ELSE 44 ... Run Keywords Key1 parms 45 ... AND Key2 parms 46 ``` 47- Use spaces to make conditions more readable: 48 49 Correct example: 50 ``` 51 Run Keyword If '${var1}' == '${0}' My Keyword 52 ``` 53 Incorrect example: 54 ``` 55 Run Keyword If '${var1}'=='${0}' My Keyword 56 ``` 57- When you define or call a Robot keyword, Robot pays no attention to spaces, 58 underscores or case. However, our team will observe the following 59 conventions in both our definitions and our calls: 60 - Separate words with single spaces. 61 - Capitalize the first character of each word. 62 - Capitalize all characters in any word that is an acronym (e.g. JSON, BMC, 63 etc). 64 65 Examples: 66 ``` 67 *** Keywords *** 68 69 This Is Correct 70 71 # This keyword name is correct. 72 73 this_is_incorrect 74 75 # This keyword name is incorrect because of 1) the 76 # underscores instead of spaces and 2) the failure to 77 # capitalize each word in the keyword. 78 79 soisthis 80 81 # This keyword name is incorrect because of 1) a failure to 82 # separate words with spaces and 2) a failure to capitalize 83 # each word in the keyword. 84 85 BMC Is An Acronym 86 87 # This keyword name is correct. Note that "BMC" is an 88 # acronym and as such is entirely uppercase. 89 ``` 90- Documentation strings: 91 - Each documentation string should be phrased as an **English command**. 92 Punctuate it correctly with the first word capitalized and a period at 93 the end. 94 95 Correct example: 96 ``` 97 Boot BMC 98 [Documentation] Boot the BMC. 99 ``` 100 Incorrect example: 101 ``` 102 Boot BMC 103 [Documentation] This keyword boots the BMC. 104 105 # The doc string above is not phrased as a command. 106 ``` 107 - Doc strings should be just one terse, descriptive sentence. 108 Remember that this doc string shows up in the HTML log file. Put 109 additional commentary below in standard comment lines. 110 111 Correct example: 112 ``` 113 Stop SOL Console Logging 114 115 [Documentation] Stop system console logging and return log output. 116 ``` 117 Incorrect example: 118 ``` 119 Stop SOL Console Logging 120 121 [Documentation] Stop system console logging. If there are multiple 122 ... system console processes, they will all be 123 ... stopped. If there is no existing log file this 124 ... keyword will return an error message to that 125 ... effect (and write that message to targ_file_path, 126 ... if specified). NOTE: This keyword will not fail 127 ... if there is no running system console process. 128 129 # This doc string is way too long. 130 ``` 131- Tags: 132 - Create a tag for every test case with a tag name that mirrors the test case 133 name as follows: 134 ``` 135 Create Intermediate File 136 137 [Tags] Create_Intermediate_File 138 ``` 139- Description of argument(s): 140 - As shown in the following example, if your keyword has any arguments, include 141 a "**Description of argument(s)**" section. This effectively serves as the 142 help text for anyone wanting to use or understand your keyword. Include 143 real data examples wherever possible and applicable. Leave at least 2 spaces 144 between the argument name and the description. Align all description text as 145 shown in the example below. 146 147 Example: 148 ``` 149 Get URL List 150 [Documentation] Return list of URLs under given URL. 151 [Arguments] ${openbmc_url} ${policy} 152 153 # Description of argument(s): 154 # openbmc_url URL for list operation (e.g. 155 # "/xyz/openbmc_project/inventory"). 156 # policy Power restore policy (e.g "RESTORE_LAST_STATE", 157 # ${RESTORE_LAST_STATE}). 158 ``` 159- Variable assignments: 160 161 When assigning a variable as output from a keyword, do not precede the 162 equal sign with a space. 163 164 Correct examples: 165 ``` 166 ${var1}= Set Variable ${1} 167 ${var1}= My Keyword 168 ``` 169 Incorrect examples: 170 171 ``` 172 ${var1}= Set Variable ${1} 173 ${var1}= My Keyword 174 ``` 175- General variable naming conventions: 176 - Variable names should be lower case with few exceptions: 177 - Environment variables should be all upper case. 178 - Variables intended to be set by Robot -v parameters may be all 179 upper case. 180 - Words within a variable name should be separated by underscores: 181 182 Correct examples: 183 ``` 184 ${host_name} 185 ${program_pid} 186 ``` 187 Incorrect examples: 188 ``` 189 ${HostName} 190 ${ProgramPid} 191 ``` 192- Special variable naming conventions. 193 194 For certain very commonly used kinds of variables, please observe these 195 conventions in order to achieve consistency throughout the code. 196 197 - hosts 198 199 When a variable is intended to contain **either** an IP address **or** 200 a host name (either long or short), please give it a suffix of "_host". 201 202 Examples: 203 ``` 204 openbmc_host 205 os_host 206 pdu_host 207 openbmc_serial_host 208 ``` 209 - host names 210 211 For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use 212 a suffix of _host_name. 213 214 Examples: 215 ``` 216 openbmc_host_name 217 os_host_name 218 pdu_host_name 219 openbmc_serial_host_name 220 ``` 221 - Short host names 222 223 For short host names (e.g. "bmc1"), use a suffix of _host_short_name. 224 225 Examples: 226 ``` 227 openbmc_host_short_name 228 os_host_short_name 229 pdu_host_short_name 230 openbmc_serial_host_short_name 231 ``` 232 - IP addresses 233 234 For IP addresses, use a suffix of _ip. 235 236 Example: 237 ``` 238 openbmc_ip 239 os_ip 240 pdu_ip 241 openbmc_serial_ip 242 ``` 243 - Files and directories: 244 - Files: 245 - If your variable is to contain only the file's name, use a suffix 246 of _file_name. 247 248 Examples: 249 ``` 250 ffdc_file_name = "bmc1.170428.120200.ffdc" 251 ``` 252 - If your variable is to contain the path to a file, use a suffix of 253 _file_path. Bear in mind that a file path can be relative or 254 absolute so that should not be a consideration in whether to use 255 the "_file_path" suffix. 256 257 Examples: 258 ``` 259 status_file_path = "bmc1.170428.120200.status" 260 status_file_path = "subdir/bmc1.170428.120200.status" 261 status_file_path = "./bmc1.170428.120200.status" 262 status_file_path = "../bmc1.170428.120200.status" 263 status_file_path = "/home/user1/status/bmc1.170428.120200.status" 264 ``` 265 To re-iterate, it doesn't matter whether the contents of the 266 variable are a relative or absolute path (as shown in the 267 examples above). A file path is simply a value with enough 268 information in it for the program to find the file. 269 270 - If the variable **must** contain an absolute path (which should be 271 the rare case), use a suffix _abs_file_path. 272 273 - Directories: 274 - Directory variables should follow the same conventions as file 275 variables. 276 277 - If your variable is to contain only the directory's name, use a 278 suffix of _dir_name. 279 280 Example: 281 ``` 282 ffdc_dir_name = "ffdc" 283 ``` 284 - If your variable is to contain the path to a directory, use a 285 suffix of _dir_path. Bear in mind that a dir path can be 286 relative or absolute so that should not be a consideration in 287 whether to use _dir_path. 288 289 Examples: 290 ``` 291 status_dir_path = "status/" 292 status_dir_path = "subdir/status" 293 status_dir_path = "./status/" 294 status_dir_path = "../status/" 295 status_dir_path = "/home/user1/status/" 296 ``` 297 To re-iterate, it doesn't matter whether the contents of 298 the variable are a relative or absolute path (as shown in 299 the examples above). A dir path is simply a value with 300 enough information in it for the program to find the 301 directory. 302 303 - If the variable **must** contain an absolute path (which 304 should be the rare case), use a suffix _abs_dir_path. 305 - IMPORTANT: As a programming convention, do pre- 306 processing on all dir_path variables to ensure that they 307 contain a trailing slash. If we follow that convention 308 religiously, that when changes are made in other parts of 309 the program, the programmer can count on the value having 310 a trailing slash. Therefore they can safely do this kind 311 of thing: 312 ``` 313 my_file_path = my_dir_path + my_file_name 314 ``` 315 - Setup/Teardown keywords 316 317 Use standardized names for setup and teardown keywords: 318 - Suite Setup Execution 319 - Suite Teardown Execution 320 - Test Setup Execution 321 - Test Teardown Execution 322- Traditional comments (i.e. using the hashtag style comments) 323 - Please leave one space following the hashtag. 324 ``` 325 #wrong 326 327 # Right 328 ``` 329 - Please use proper English punction: 330 - Capitalize the first word in the sentence or phrase. 331 - End sentences (or stand-alone phrases) with a period. 332 333 - Do not keep commented out code in your program. Instead, remove it 334 entirely. 335- Robot Template Test Cases 336 - Follow this format for Robot template test cases: 337 Note: Documentation, Tags and Template lines are all required and should be coded in the order shown. 338 ``` 339 Test Case Name 340 [Documentation] 341 [Tags] 342 [Template] 343 # arg1 arg2 etc. 344 <arg1> <arg2> 345 346 Example: 347 348 Get Response Codes 349 [Documentation] REST "Get" response status test. 350 [Tags] Get_Response_Codes 351 [Template] Execute Get And Check Response 352 353 # Expect status URL Path 354 ${HTTP_OK} /org/ 355 ${HTTP_OK} /xyz/ 356 ${HTTP_OK} /xyz/openbmc_project/ 357 ${HTTP_OK} /xyz/openbmc_project/state/enumerate 358 ${HTTP_NOT_FOUND} /xyz/i/dont/exist/ 359 ``` 360 361Python Coding Guidelines 362----------------------- 363- The minimum required Python version is 2.7.x. 364- Run pycodestyle on all Python files and correct errors to follow the guidelines in 365 https://www.python.org/dev/peps/pep-0008/. 366 367 Example as run from a Linux command line: 368 ``` 369 pycodestyle my_pgm.py 370 371 my_pgm.py:41:1: E302 expected 2 blank lines, found 1 372 my_pgm.py:58:52: W291 trailing whitespace 373 ``` 374- Include doc strings in every function and follow the guidelines in 375 https://www.python.org/dev/peps/pep-0257/. 376 377 Example: 378 ``` 379 r""" 380 Return the function name associated with the indicated stack frame. 381 382 Description of argument(s): 383 stack_frame_ix The index of the stack frame whose 384 function name should be returned. If 385 the caller does not specify a value, 386 this function will set the value to 1 387 which is the index of the caller's 388 stack frame. If the caller is the 389 wrapper function "print_func_name", 390 this function will bump it up by 1. 391 """ 392 ``` 393- As shown in the prior example, if your function has any arguments, include 394 a "Description of argument(s)" section. This effectively serves as the 395 help text for anyone wanting to use or understand your function. Include 396 real data examples wherever possible and applicable. 397- Function definitions: 398 - Put each function parameter on its own line: 399 ``` 400 def func1(parm1, 401 402 parm2): 403 ``` 404- Do not keep commented out code in your program. Instead, remove it 405 entirely. 406- When you define or call a Robot keyword, Robot pays no attention to spaces, 407 underscores or case. However, our team will observe the following 408 conventions in both our definitions and our calls: 409 - Separate words with single spaces. 410 - Capitalize the first character of each word. 411 - Capitalize all characters in any word that is an acronym (e.g. JSON, BMC, 412 etc). 413 414 Examples: 415 ``` 416 *** Keywords *** 417 418 This Is Correct 419 420 # This keyword name is correct. 421 422 this_is_incorrect 423 424 # This keyword name is incorrect because of 1) the 425 # underscores instead of spaces and 2) the failure to 426 # capitalize each word in the keyword. 427 428 soisthis 429 430 # This keyword name is incorrect because of 1) a failure to 431 # separate words with spaces and 2) a failure to capitalize 432 # each word in the keyword. 433 434 BMC Is An Acronym 435 436 # This keyword name is correct. Note that "BMC" is an 437 # acronym and as such is entirely uppercase. 438 ``` 439- Documentation strings: 440 - Each documentation string should be phrased as an **English command**. 441 Punctuate it correctly with the first word capitalized and a period at 442 the end. 443 444 Correct example: 445 ``` 446 Boot BMC 447 [Documentation] Boot the BMC. 448 ``` 449 Incorrect example: 450 ``` 451 Boot BMC 452 [Documentation] This keyword boots the BMC. 453 454 # The doc string above is not phrased as a command. 455 ``` 456 - Doc strings should be just one terse, descriptive sentence. 457 Remember that this doc string shows up in the HTML log file. Put 458 additional commentary below in standard comment lines. 459 460 Correct example: 461 ``` 462 Stop SOL Console Logging 463 464 [Documentation] Stop system console logging and return log output. 465 ``` 466 Incorrect example: 467 ``` 468 Stop SOL Console Logging 469 470 [Documentation] Stop system console logging. If there are multiple 471 ... system console processes, they will all be 472 ... stopped. If there is no existing log file this 473 ... keyword will return an error message to that 474 ... effect (and write that message to targ_file_path, 475 ... if specified). NOTE: This keyword will not fail 476 ... if there is no running system console process. 477 478 # This doc string is way too long. 479 ``` 480- Tags: 481 - Create a tag for every test case with a tag name that mirrors the test case 482 name as follows: 483 ``` 484 Create Intermediate File 485 486 [Tags] Create_Intermediate_File 487 ``` 488- General variable naming conventions: 489 - Variable names should be lower case with few exceptions: 490 - Environment variables should be all upper case. 491 - Variables intended to be set by Robot -v parameters may be all 492 upper case. 493 - Words within a variable name should be separated by underscores: 494 495 Correct examples: 496 ``` 497 ${host_name} 498 ${program_pid} 499 ``` 500 Incorrect examples: 501 ``` 502 ${HostName} 503 ${ProgramPid} 504 ``` 505- Special variable naming conventions. 506 507 For certain very commonly used kinds of variables, please observe these 508 conventions in order to achieve consistency throughout the code. 509 510 - hosts 511 512 When a variable is intended to contain **either** an IP address **or** 513 a host name (either long or short), please give it a suffix of "_host". 514 515 Examples: 516 ``` 517 openbmc_host 518 os_host 519 pdu_host 520 openbmc_serial_host 521 ``` 522 - host names 523 524 For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use 525 a suffix of _host_name. 526 527 Examples: 528 ``` 529 openbmc_host_name 530 os_host_name 531 pdu_host_name 532 openbmc_serial_host_name 533 ``` 534 - Short host names 535 536 For short host names (e.g. "bmc1"), use a suffix of _host_short_name. 537 538 Examples: 539 ``` 540 openbmc_host_short_name 541 os_host_short_name 542 pdu_host_short_name 543 openbmc_serial_host_short_name 544 ``` 545 - IP addresses 546 547 For IP addresses, use a suffix of _ip. 548 549 Example: 550 ``` 551 openbmc_ip 552 os_ip 553 pdu_ip 554 openbmc_serial_ip 555 ``` 556- Files and directories: 557 - Files: 558 - If your variable is to contain only the file's name, use a suffix 559 of _file_name. 560 561 Examples: 562 ``` 563 ffdc_file_name = "bmc1.170428.120200.ffdc" 564 ``` 565 - If your variable is to contain the path to a file, use a suffix of 566 _file_path. Bear in mind that a file path can be relative or 567 absolute so that should not be a consideration in whether to use 568 the "_file_path" suffix. 569 570 Examples: 571 ``` 572 status_file_path = "bmc1.170428.120200.status" 573 status_file_path = "subdir/bmc1.170428.120200.status" 574 status_file_path = "./bmc1.170428.120200.status" 575 status_file_path = "../bmc1.170428.120200.status" 576 status_file_path = "/home/user1/status/bmc1.170428.120200.status" 577 ``` 578 To re-iterate, it doesn't matter whether the contents of the 579 variable are a relative or absolute path (as shown in the 580 examples above). A file path is simply a value with enough 581 information in it for the program to find the file. 582 583 - If the variable **must** contain an absolute path (which should be 584 the rare case), use a suffix _abs_file_path. 585 586 - Directories: 587 - Directory variables should follow the same conventions as file 588 variables. 589 590 - If your variable is to contain only the directory's name, use a 591 suffix of _dir_name. 592 593 Example: 594 ``` 595 ffdc_dir_name = "ffdc" 596 ``` 597 - If your variable is to contain the path to a directory, use a 598 suffix of _dir_path. Bear in mind that a dir path can be 599 relative or absolute so that should not be a consideration in 600 whether to use _dir_path. 601 602 Examples: 603 ``` 604 status_dir_path = "status/" 605 status_dir_path = "subdir/status" 606 status_dir_path = "./status/" 607 status_dir_path = "../status/" 608 status_dir_path = "/home/user1/status/" 609 ``` 610 To re-iterate, it doesn't matter whether the contents of 611 the variable are a relative or absolute path (as shown in 612 the examples above). A dir path is simply a value with 613 enough information in it for the program to find the 614 directory. 615 616 - If the variable **must** contain an absolute path (which 617 should be the rare case), use a suffix _abs_dir_path. 618 - IMPORTANT: As a programming convention, do pre- 619 processing on all dir_path variables to ensure that they 620 contain a trailing slash. If we follow that convention 621 religiously, that when changes are made in other parts of 622 the program, the programmer can count on the value having 623 a trailing slash. Therefore they can safely do this kind 624 of thing: 625 ``` 626 my_file_path = my_dir_path + my_file_name 627 ``` 628- Traditional comments (i.e. using the hashtag style comments) 629 - Please leave one space following the hashtag. 630 ``` 631 #wrong 632 633 # Right 634 ``` 635 - Please use proper English punction: 636 - Capitalize the first word in the sentence or phrase. 637 - End sentences (or stand-alone phrases) with a period. 638 639 - Do not keep commented out code in your program. Instead, remove it 640 entirely. 641 642Template Usage Guidelines 643------------------------- 644We have several templates in the templates/ sub-directory. If there is a 645template that applies to your programming situation (Python, bash, etc.), 646it should be used to create new programs as in the following example 647 648- Example: 649 650 ``` 651 $ cd templates 652 $ cp python_pgm_template ../bin/my_new_program 653 ``` 654 655These templates have much of your preliminary work done for you and will help 656us all follow a similar structure. 657 658- Features: 659 - Help text and arg parsing started for you. 660 - Support for "stock" parameters like "quiet", "debug", "test_mode". 661 - "exit_function" and "signal_handler" defined. 662 - "validate_parms" function pre-created. 663 - "main" function follows conventional startup sequence: 664 665 ``` 666 if not gen_get_options(parser, stock_list): 667 return False 668 669 if not validate_parms(): 670 return False 671 672 qprint_pgm_header() 673 674 # Your code here. 675 ``` 676