1Contributing to OpenBMC Test Automation
2=======================================
3Guide to working on OpenBMC test automation. This document will always be a
4work-in-progress, feel free to propose changes.
5
6Submitting changes via Gerrit server
7------------------------------------
8-   Reference [OpenBMC CLA](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server-to-openbmc)
9-   Reference [OpenBMC docs](https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md#submitting-changes-via-gerrit-server)
10
11Links
12-----
13
14- [Redfish Coding Guidelines](docs/redfish_coding_guidelines.md)
15
16Robot Coding Guidelines
17-----------------------
18-   For this project, we will write Robot keyword definitions in either Robot
19    or Python.  Robot code should be quite simple.  Therefore, if the algorithm
20    in question is the least bit complex, please write it in Python.
21
22    See the following for support on writing most keywords in python.
23      - [Quit Writing Ugly Robot Code](https://esalagea.wordpress.com/2014/11/24/robot-framework-quit-writing-ugly-robot-code-just-write-proper-python/)
24      - [Robot Dos and Don'ts](https://www.slideshare.net/pekkaklarck/robot-framework-dos-and-donts)
25
26-   Observe a maximum line length of 110 characters.
27-   Avoid trailing space at the end of any line of Robot code.
28-   Avoid the use of tabs.
29-   Robot supports delimiting cells with either two or more spaces or with a
30    pipe symbol (e.g. "\|"). Our team has chosen to use spaces rather than the
31    pipe character. Make sure all space delimiters in Robot code are the
32    **minimum** of two spaces. There may be some exceptions to this rule.
33
34    Exceptions to two-space delimiter rule:
35    - When you wish to line up resource, library or variable values:
36      ```
37      Library         Lib1
38      Resource        Resource1
39      *** Variables ***
40      ${var1}         ${EMPTY}
41      ```
42    - When you wish to line up fields for test templates:
43      ```
44      [Template]  Set System LED State
45      # LED Name  LED State
46      power       On
47      power       Off
48      ```
49    - When you wish to indent if/else or loop bodies for visual effect:
50      ```
51      Run Keyword If  '${this}' == '${that}'
52      ...    Log  Bla, bla...
53      ...  ELSE
54      ...    Run Keywords  Key1  parms
55      ...    AND  Key2  parms
56      ```
57-   Use single spaces to make conditions more readable:
58
59    Correct example:
60    ```
61    Run Keyword If  '${var1}' == '${0}'  My Keyword
62    ```
63    Incorrect example:
64    ```
65    Run Keyword If  '${var1}'=='${0}'  My Keyword
66    ```
67-   When you define or call a Robot keyword, Robot pays no attention to spaces,
68    underscores or case.  However, our team will observe the following
69    conventions in both our definitions and our calls:
70    - Separate words with single spaces.
71    - Capitalize the first character of each word.
72    - Capitalize all characters in any word that is an acronym (e.g. JSON, BMC,
73      etc).
74
75    Examples:
76    ```
77    *** Keywords ***
78
79    This Is Correct
80
81        # This keyword name is correct.
82
83    this_is_incorrect
84
85        # This keyword name is incorrect because of 1) the
86        # underscores instead of spaces and 2) the failure to
87        # capitalize each word in the keyword.
88
89    soisthis
90
91        # This keyword name is incorrect because of 1) a failure to
92        # separate words with single spaces and 2) a failure to capitalize
93        # each word in the keyword.
94
95    BMC Is An Acronym
96
97        # This keyword name is correct.  Note that "BMC" is an
98        # acronym and as such is entirely uppercase.
99    ```
100-   Documentation strings:
101    -  Each documentation string should be phrased as an **English command**.
102       Punctuate it correctly with the first word capitalized and a period at
103       the end.
104
105       Correct example:
106        ```
107        Boot BMC
108            [Documentation]  Boot the BMC.
109        ```
110        Incorrect example:
111        ```
112        Boot BMC
113            [Documentation]  This keyword boots the BMC.
114
115            # The doc string above is not phrased as a command.
116        ```
117    -   Doc strings should be just one terse, descriptive sentence.
118        Remember that this doc string shows up in the HTML log file.  Put
119        additional commentary below in standard comment lines.
120
121        Correct example:
122        ```
123        Stop SOL Console Logging
124
125            [Documentation]  Stop system console logging and return log output.
126        ```
127        Incorrect example:
128        ```
129        Stop SOL Console Logging
130
131            [Documentation]  Stop system console logging.  If there are multiple
132            ...              system console processes, they will all be
133            ...              stopped.  If there is no existing log file this
134            ...              keyword will return an error message to that
135            ...              effect (and write that message to targ_file_path,
136            ...              if specified).  NOTE: This keyword will not fail
137            ...              if there is no running system console process.
138
139            # This doc string is way too long.
140        ```
141-   Tags:
142    -   Create a tag for every test case with a tag name that mirrors the test case
143        name as follows:
144        ```
145        Create Intermediate File
146
147            [Tags]  Create_Intermediate_File
148        ```
149-   Description of argument(s):
150    -   As shown in the following example, if your keyword has any arguments, include
151        a "**Description of argument(s)**" section.  This effectively serves as the
152        help text for anyone wanting to use or understand your keyword.  Include
153        real data examples wherever possible and applicable.  Leave at least 2 spaces
154        between the argument name and the description.  Align all description text as
155        shown in the example below.
156
157        Example:
158        ```
159        Get URL List
160            [Documentation]  Return list of URLs under given URL.
161            [Arguments]  ${openbmc_url}  ${policy}   ${verify}
162
163            # Description of argument(s):
164            # openbmc_url  URL for list operation (e.g.
165            #              "/xyz/openbmc_project/inventory").
166            # policy       Power restore policy (e.g "RESTORE_LAST_STATE",
167            #              ${RESTORE_LAST_STATE}, etc.).
168            # verify       Verify the results (${TRUE}/${FALSE}).
169        ```
170        Additional rules for example text in descriptions:
171        - Put parentheses around your examples.  Leave one space prior to the
172          left parenthesis.
173        - Use "e.g." (which effectively means "for example") to set introduce
174          your examples.
175        - Quote string examples.
176        - Include ", etc." after multiple examples.
177        - For cases where you're providing an complete list of possible values
178          (${TRUE}/${FALSE} or "PASS"/"FAIL"), do NOT use "e.g." and do NOT
179          use "etc.".  Separate such values with a slash.
180-   Variable assignments:
181
182    When assigning a variable as output from a keyword, do not precede the
183    equal sign with a space.
184
185    Correct examples:
186    ```
187    ${var1}=  Set Variable  ${1}
188    ${var1}=  My Keyword
189    ```
190    Incorrect examples:
191
192    ```
193    ${var1} =  Set Variable  ${1}
194    ${var1} =  My Keyword
195    ```
196-   General variable naming conventions:
197    -   Variable names should be lower case with few exceptions (listed here):
198        -   Environment variables should be all upper case.
199        -   Variables intended to be set by Robot -v parameters may be all
200            upper case.
201    -   Words within a variable name should be separated by underscores:
202
203        Correct examples:
204        ```
205        ${host_name}
206        ${program_pid}
207        ```
208        Incorrect examples:
209        ```
210        ${HostName}
211        ${ProgramPid}
212        ```
213    -   Use good judgement in choosing variable names that are neither way too
214        short nor way too long.
215
216        Correct examples:
217        ```
218        ${host_name}
219        ${program_pid}
220        ```
221        Incorrect examples:
222        ```
223        ${HostName}
224        ${ProgramPid}
225        ```
226    -   Avoid having the variable's type as a suffix portion of the name:
227
228        Incorrect examples:
229        ```
230        ${inventory_dict}
231        ${led_list}
232        ```
233
234        Incorrect examples:
235        ```
236        ${inventory}
237        # Use plural name to indicate that it is a list.
238        ${leds}
239        ```
240
241        A possible exception to this rule is when your keyword or function has
242        an ongoing need to refer to one piece of data in more than one format
243        (e.g. date_str, date_epoch, etc.).
244
245    -   Consistency of variable names:
246
247        Try to avoid referring to the same data entity by multiple different
248        names.  It creates a lot of confusion for the reader.
249
250        Incorrect example:
251        ```
252        # Why call the receiving variable rc (return code) when the keyword is
253        # supposed to return status.
254        ${rc}=  Run Keyword And Return Status  Bla Bla
255        ```
256
257        Correct example:
258        ```
259        ${status}=  Run Keyword And Return Status  Bla Bla
260        ```
261
262-   Special variable naming conventions.
263
264    For certain very commonly used kinds of variables, please observe these
265    conventions in order to achieve consistency throughout the code.
266
267    -   hosts
268
269        When a variable is intended to contain **either** an IP address **or**
270        a host name (whether long or short), please give it a suffix of "_host".
271
272        Examples:
273        ```
274        openbmc_host
275        os_host
276        pdu_host
277        openbmc_serial_host
278        ```
279    -   host names
280
281        For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use
282        a suffix of _host_name.
283
284        Examples:
285        ```
286        openbmc_host_name
287        os_host_name
288        pdu_host_name
289        openbmc_serial_host_name
290        ```
291    -   Short host names
292
293        For short host names (e.g. "bmc1"), use a suffix of _host_short_name.
294
295        Examples:
296        ```
297        openbmc_host_short_name
298        os_host_short_name
299        pdu_host_short_name
300        openbmc_serial_host_short_name
301        ```
302    -   IP addresses
303
304        For IP addresses, use a suffix of _ip.
305
306        Example:
307        ```
308        openbmc_ip
309        os_ip
310        pdu_ip
311        openbmc_serial_ip
312        ```
313    -   Files and directories:
314        -   Files:
315            -   If your variable is to contain only the file's name, use a suffix
316                of _file_name.
317
318                Examples:
319                ```
320                ffdc_file_name = "bmc1.170428.120200.ffdc"
321                ```
322            -   If your variable is to contain the path to a file, use a suffix of
323                _file_path.  Bear in mind that a file path can be relative or
324                absolute, so that should not be a consideration in whether to use
325                the "_file_path" suffix.
326
327                Examples:
328                ```
329                status_file_path = "bmc1.170428.120200.status"
330                status_file_path = "subdir/bmc1.170428.120200.status"
331                status_file_path = "./bmc1.170428.120200.status"
332                status_file_path = "../bmc1.170428.120200.status"
333                status_file_path = "/home/user1/status/bmc1.170428.120200.status"
334                ```
335                To re-iterate, it doesn't matter whether the contents of the
336                variable are a relative or absolute path (as shown in the
337                examples above).  A file path is simply a value with enough
338                information in it for the program to find the file.
339
340            -   If the variable **must** contain an absolute path (which should be
341                the rare case), use a suffix _abs_file_path.
342
343        -   Directories:
344            -   Directory variables should follow the same conventions as file
345                variables.
346
347            -   If your variable is to contain only the directory's name, use a
348                suffix of _dir_name.
349
350                Example:
351                ```
352                ffdc_dir_name = "ffdc"
353                ```
354            -   If your variable is to contain the path to a directory, use a
355                suffix of _dir_path.  Bear in mind that a dir path can be
356                relative or absolute, so that should not be a consideration in
357                whether to use _dir_path.
358
359                Examples:
360                ```
361                status_dir_path = "status/"
362                status_dir_path = "subdir/status"
363                status_dir_path = "./status/"
364                status_dir_path = "../status/"
365                status_dir_path = "/home/user1/status/"
366                ```
367                To re-iterate, it doesn't matter whether the contents of
368                the variable are a relative or absolute path (as shown in
369                the examples above).  A dir path is simply a value with
370                enough information in it for the program to find the
371                directory.
372
373            -   If the variable **must** contain an absolute path (which
374                should be the rare case), use a suffix _abs_dir_path.
375            -   IMPORTANT:  As a programming convention, do pre-
376                processing on all dir_path variables to ensure that they
377                contain a trailing slash.  If we follow that convention
378                religiously, then when changes are made in other parts of
379                the program, the programmer can count on the value having
380                a trailing slash.  Therefore, they can safely do this kind
381                of thing:
382                ```
383                my_file_path = my_dir_path + my_file_name
384                ```
385    -   Setup/Teardown keywords
386
387        Use standardized names for setup and teardown keywords:
388        - Suite Setup Execution
389        - Suite Teardown Execution
390        - Test Setup Execution
391        - Test Teardown Execution
392-   Traditional comments (i.e. using the hashtag style comments)
393    -   Please leave one space following the hashtag.
394        ```
395        #wrong
396
397        # Right
398        ```
399    -   Please use proper English punctuation:
400        -   Capitalize the first word in the sentence or phrase.
401        -   End sentences (or stand-alone phrases) with a period.
402
403    -   Do not keep commented-out code in your program.  Instead, remove it
404        entirely.
405-   Robot Template Test Cases
406    -   Follow this format for Robot template test cases:
407
408        Note: Documentation, Tags and Template lines are all required and should be coded in the order shown.
409        ```
410        Test Case Name
411            [Documentation]
412            [Tags]
413            [Template]
414            # arg1  arg2  etc.
415            <arg1>  <arg2>
416
417        Example:
418
419        Get Response Codes
420            [Documentation]  REST "Get" response status test.
421            [Tags]  Get_Response_Codes
422            [Template]  Execute Get And Check Response
423
424            # expected_response_code  url_path
425            ${HTTP_OK}                /org/
426            ${HTTP_OK}                /xyz/
427            ${HTTP_OK}                /xyz/openbmc_project/
428            ${HTTP_OK}                /xyz/openbmc_project/state/enumerate
429            ${HTTP_NOT_FOUND}         /xyz/i/dont/exist/
430        ```
431
432        Note: Normally, a template test case would have many rows of data arguments as in the example above.
433              However, contributors frequently define multiple template test cases that each have only
434              one row of data which may seem to defeat the value of using templates in the first place. However,
435              it is done for these reasons:
436              1) Template tests are counted as a single test. The user may wish to have separate results for
437                 each call to the template function.
438              2) If any call to the template function fails, one would like FFDC data collected immediately
439                 and would like one set of FFDC data for EACH such failure.
440
441
442Python Coding Guidelines
443-----------------------
444-   The minimum required Python version is 2.7.x.  In the very near future, we
445    will stop supporting python 2 and will require python 3.
446-   Run pycodestyle on all Python files and correct errors to follow the
447    guidelines in https://www.python.org/dev/peps/pep-0008/.  Note that when
448    python code is checked into gerrit, pycodestyle is run automatically on it.
449
450    Example as run from a Linux command line:
451    ```
452    pycodestyle my_pgm.py
453
454    my_pgm.py:41:1: E302 expected 2 blank lines, found 1
455    my_pgm.py:58:52: W291 trailing whitespace
456    ```
457-   Include doc strings in every function and follow the guidelines in
458    https://www.python.org/dev/peps/pep-0257/.
459
460    Example:
461    ```
462        r"""
463        Return the function name associated with the indicated stack frame.
464
465        Description of argument(s):
466        stack_frame_ix                  The index of the stack frame whose
467                                        function name should be returned.  If
468                                        the caller does not specify a value,
469                                        this function will set the value to 1
470                                        which is the index of the caller's
471                                        stack frame.  If the caller is the
472                                        wrapper function "print_func_name",
473                                        this function will bump it up by 1.
474        """
475    ```
476-   As shown in the prior example, if your function has any arguments, include
477    a "Description of argument(s)" section.  This effectively serves as the
478    help text for anyone wanting to use or understand your function.  Include
479    real data examples wherever possible and applicable.
480-   Function definitions:
481    -   Put each function parameter on its own line:
482        ```
483        def func1(parm1,
484
485                  parm2):
486        ```
487-   Do not keep commented-out code in your program.  Instead, remove it
488    entirely.
489-   When you define a python function, observe the following
490    conventions:
491    - Separate words with single underscores.
492    - Use lower-case letters.
493
494    Examples:
495    ```
496
497    def this_is_correct():
498
499        # This function name is correct.
500
501    def This_Is_Incorrect():
502
503        # This keyword name is incorrect because of the upper case letters used.
504
505    def soisthis():
506
507        # This keyword name is incorrect because of 1) a failure to
508        # separate words with underscores.
509
510    ```
511-   Documentation strings:
512    -  Each documentation string should be phrased as an **English command**.
513       Punctuate it correctly with the first word capitalized and a period at
514       the end.
515
516       Correct example:
517        ```
518        def boot_bmc():
519            r"""
520            Boot the BMC.
521            """
522        ```
523        Incorrect example:
524        ```
525        def boot_bmc():
526            r"""
527            This function boots the BMC.
528            """
529
530            # The doc string above is not phrased as a command.
531        ```
532    -   Doc strings should begin with a summary line which is one terse,
533        descriptive sentence.  Put additional commentary below.
534
535        Correct example:
536        ```
537        def stop_sol_console_logging():
538            r"""
539            Stop system console logging and return log output.
540
541            Additional comments...
542            """
543        ```
544        Incorrect example:
545        ```
546        def stop_sol_console_logging():
547            r"""
548            Stop system console logging.  If there are multiple system console
549            processes, they will all be stopped.  If there is no existing log file
550            this keyword will return an error message to that effect (and write that
551            message to targ_file_path, if specified).  NOTE: This keyword will not
552            fail if there is no running system console process.
553            """
554
555            # This summary is way too long.
556        ```
557-   General variable naming conventions:
558    -   Variable names should be lower case with few exceptions (listed here):
559        -   Environment variables should be all upper case.
560    -   Words within a variable name should be separated by underscores:
561
562        Correct examples:
563        ```
564        ${host_name}
565        ${program_pid}
566        ```
567        Incorrect examples:
568        ```
569        ${HostName}
570        ${ProgramPid}
571        ```
572-   Special variable naming conventions.
573
574    For certain very commonly used kinds of variables, please observe these
575    conventions in order to achieve consistency throughout the code.
576
577    -   hosts
578
579        When a variable is intended to contain **either** an IP address **or**
580        a host name (either long or short), please give it a suffix of "_host".
581
582        Examples:
583        ```
584        openbmc_host
585        os_host
586        pdu_host
587        openbmc_serial_host
588        ```
589    -   host names
590
591        For host names (long or short, e.g. "bmc1" or "bmc1.example.com"), use
592        a suffix of _host_name.
593
594        Examples:
595        ```
596        openbmc_host_name
597        os_host_name
598        pdu_host_name
599        openbmc_serial_host_name
600        ```
601    -   Short host names
602
603        For short host names (e.g. "bmc1"), use a suffix of _host_short_name.
604
605        Examples:
606        ```
607        openbmc_host_short_name
608        os_host_short_name
609        pdu_host_short_name
610        openbmc_serial_host_short_name
611        ```
612    -   IP addresses
613
614        For IP addresses, use a suffix of _ip.
615
616        Example:
617        ```
618        openbmc_ip
619        os_ip
620        pdu_ip
621        openbmc_serial_ip
622        ```
623-   Files and directories:
624    -   Files:
625        -   If your variable is to contain only the file's name, use a suffix
626            of _file_name.
627
628            Examples:
629            ```
630            ffdc_file_name = "bmc1.170428.120200.ffdc"
631            ```
632        -   If your variable is to contain the path to a file, use a suffix of
633            _file_path.  Bear in mind that a file path can be relative or
634            absolute, so that should not be a consideration in whether to use
635            the "_file_path" suffix.
636
637            Examples:
638            ```
639            status_file_path = "bmc1.170428.120200.status"
640            status_file_path = "subdir/bmc1.170428.120200.status"
641            status_file_path = "./bmc1.170428.120200.status"
642            status_file_path = "../bmc1.170428.120200.status"
643            status_file_path = "/home/user1/status/bmc1.170428.120200.status"
644            ```
645            To re-iterate, it doesn't matter whether the contents of the
646            variable are a relative or absolute path (as shown in the
647            examples above).  A file path is simply a value with enough
648            information in it for the program to find the file.
649
650        -   If the variable **must** contain an absolute path (which should be
651            the rare case), use a suffix _abs_file_path.
652
653    -   Directories:
654        -   Directory variables should follow the same conventions as file
655            variables.
656
657        -   If your variable is to contain only the directory's name, use a
658            suffix of _dir_name.
659
660            Example:
661            ```
662            ffdc_dir_name = "ffdc"
663            ```
664        -   If your variable is to contain the path to a directory, use a
665            suffix of _dir_path.  Bear in mind that a dir path can be
666            relative or absolute so, that should not be a consideration in
667            whether to use _dir_path.
668
669            Examples:
670            ```
671            status_dir_path = "status/"
672            status_dir_path = "subdir/status"
673            status_dir_path = "./status/"
674            status_dir_path = "../status/"
675            status_dir_path = "/home/user1/status/"
676            ```
677            To re-iterate, it doesn't matter whether the contents of
678            the variable are a relative or absolute path (as shown in
679            the examples above).  A dir path is simply a value with
680            enough information in it for the program to find the
681            directory.
682
683        -   If the variable **must** contain an absolute path (which
684            should be the rare case), use a suffix _abs_dir_path.
685        -   IMPORTANT:  As a programming convention, do pre-
686            processing on all dir_path variables to ensure that they
687            contain a trailing slash.  If we follow that convention
688            religiously, that when changes are made in other parts of
689            the program, the programmer can count on the value having
690            a trailing slash.  Therefore they can safely do this kind
691            of thing:
692            ```
693            my_file_path = my_dir_path + my_file_name
694            ```
695-   Traditional comments (i.e. using the hashtag style comments)
696    -   Please leave one space following the hashtag.
697        ```
698        #wrong
699
700        # Right
701        ```
702    -   Please use proper English punction:
703        -   Capitalize the first word in the sentence or phrase.
704        -   End sentences (or stand-alone phrases) with a period.
705
706    -   Do not keep commented-out code in your program.  Instead, remove it
707        entirely.
708
709Template Usage Requirements
710---------------------------
711We have several templates in the templates/ sub-directory. If there is a
712template that applies to your programming situation (Python, bash, etc.),
713it should be used to create new programs as in the following example
714
715- Example:
716
717    ```
718    $ cd templates
719    $ cp python_pgm_template] ../bin/my_new_program.py
720    ```
721
722These templates have much of your preliminary work done for you and will help
723us all follow a similar structure.
724
725See [python_pgm_template](templates/python_pgm_template) as an example.
726
727- Features:
728    - Help text and arg parsing started for you.
729    - Support for "stock" parameters like "quiet", "debug", "test_mode".
730    - "exit_function" pre-defined.
731    - "validate_parms" function pre-defined.
732    - "main" function follows conventional startup sequence:
733
734        ```
735            gen_get_options(parser, stock_list)
736
737            validate_parms()
738
739            qprint_pgm_header()
740
741            # Your code here.
742        ```
743