1tdc - Linux Traffic Control (tc) unit testing suite
2
3Author: Lucas Bates - lucasb@mojatatu.com
4
5tdc is a Python script to load tc unit tests from a separate JSON file and
6execute them inside a network namespace dedicated to the task.
7
8
9REQUIREMENTS
10------------
11
12*  Minimum Python version of 3.4. Earlier 3.X versions may work but are not
13   guaranteed.
14
15*  The kernel must have network namespace support
16
17*  The kernel must have veth support available, as a veth pair is created
18   prior to running the tests.
19
20*  All tc-related features being tested must be built in or available as
21   modules.  To check what is required in current setup run:
22   ./tdc.py -c
23
24   Note:
25   In the current release, tdc run will abort due to a failure in setup or
26   teardown commands - which includes not being able to run a test simply
27   because the kernel did not support a specific feature. (This will be
28   handled in a future version - the current workaround is to run the tests
29   on specific test categories that your kernel supports)
30
31
32BEFORE YOU RUN
33--------------
34
35The path to the tc executable that will be most commonly tested can be defined
36in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
37define the path.
38
39If you need to test a different tc executable on the fly, you can do so by
40using the -p option when running tdc:
41	./tdc.py -p /path/to/tc
42
43
44RUNNING TDC
45-----------
46
47To use tdc, root privileges are required.  This is because the
48commands being tested must be run as root.  The code that enforces
49execution by root uid has been moved into a plugin (see PLUGIN
50ARCHITECTURE, below).
51
52If nsPlugin is linked, all tests are executed inside a network
53namespace to prevent conflicts within the host.
54
55Running tdc without any arguments will run all tests. Refer to the section
56on command line arguments for more information, or run:
57	./tdc.py -h
58
59tdc will list the test names as they are being run, and print a summary in
60TAP (Test Anything Protocol) format when they are done. If tests fail,
61output captured from the failing test will be printed immediately following
62the failed test in the TAP output.
63
64
65OVERVIEW OF TDC EXECUTION
66-------------------------
67
68One run of tests is considered a "test suite" (this will be refined in the
69future).  A test suite has one or more test cases in it.
70
71A test case has four stages:
72
73  - setup
74  - execute
75  - verify
76  - teardown
77
78The setup and teardown stages can run zero or more commands.  The setup
79stage does some setup if the test needs it.  The teardown stage undoes
80the setup and returns the system to a "neutral" state so any other test
81can be run next.  These two stages require any commands run to return
82success, but do not otherwise verify the results.
83
84The execute and verify stages each run one command.  The execute stage
85tests the return code against one or more acceptable values.  The
86verify stage checks the return code for success, and also compares
87the stdout with a regular expression.
88
89Each of the commands in any stage will run in a shell instance.
90
91
92USER-DEFINED CONSTANTS
93----------------------
94
95The tdc_config.py file contains multiple values that can be altered to suit
96your needs. Any value in the NAMES dictionary can be altered without affecting
97the tests to be run. These values are used in the tc commands that will be
98executed as part of the test. More will be added as test cases require.
99
100Example:
101	$TC qdisc add dev $DEV1 ingress
102
103The NAMES values are used to substitute into the commands in the test cases.
104
105
106COMMAND LINE ARGUMENTS
107----------------------
108
109Run tdc.py -h to see the full list of available arguments.
110
111usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
112              [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]
113              [-d DEVICE] [-n NS] [-V]
114
115Linux TC unit tests
116
117optional arguments:
118  -h, --help            show this help message and exit
119  -p PATH, --path PATH  The full path to the tc executable to use
120  -v, --verbose         Show the commands that are being run
121  -d DEVICE, --device DEVICE
122                        Execute the test case in flower category
123
124selection:
125  select which test cases: files plus directories; filtered by categories
126  plus testids
127
128  -D DIR [DIR ...], --directory DIR [DIR ...]
129                        Collect tests from the specified directory(ies)
130                        (default [tc-tests])
131  -f FILE [FILE ...], --file FILE [FILE ...]
132                        Run tests from the specified file(s)
133  -c [CATG [CATG ...]], --category [CATG [CATG ...]]
134                        Run tests only from the specified category/ies, or if
135                        no category/ies is/are specified, list known
136                        categories.
137  -e ID [ID ...], --execute ID [ID ...]
138                        Execute the specified test cases with specified IDs
139
140action:
141  select action to perform on selected test cases
142
143  -l, --list            List all test cases, or those only within the
144                        specified category
145  -s, --show            Display the selected test cases
146  -i, --id              Generate ID numbers for new test cases
147
148netns:
149  options for nsPlugin(run commands in net namespace)
150
151  -n NS, --namespace NS
152                        Run commands in namespace NS
153
154valgrind:
155  options for valgrindPlugin (run command under test under Valgrind)
156
157  -V, --valgrind        Run commands under valgrind
158
159
160PLUGIN ARCHITECTURE
161-------------------
162
163There is now a plugin architecture, and some of the functionality that
164was in the tdc.py script has been moved into the plugins.
165
166The plugins are in the directory plugin-lib.  The are executed from
167directory plugins.  Put symbolic links from plugins to plugin-lib,
168and name them according to the order you want them to run.
169
170Example:
171
172bjb@bee:~/work/tc-testing$ ls -l plugins
173total 4
174lrwxrwxrwx  1 bjb  bjb    27 Oct  4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
175lrwxrwxrwx  1 bjb  bjb    25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
176-rwxr-xr-x  1 bjb  bjb     0 Sep 29 15:56 __init__.py
177
178The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
179must be called "SubPlugin" so tdc can find them.  They are
180distinguished from each other in the python program by their module
181name.
182
183This base class supplies "hooks" to run extra functions.  These hooks are as follows:
184
185pre- and post-suite
186pre- and post-case
187pre- and post-execute stage
188adjust-command (runs in all stages and receives the stage name)
189
190The pre-suite hook receives the number of tests and an array of test ids.
191This allows you to dump out the list of skipped tests in the event of a
192failure during setup or teardown stage.
193
194The pre-case hook receives the ordinal number and test id of the current test.
195
196The adjust-command hook receives the stage id (see list below) and the
197full command to be executed.  This allows for last-minute adjustment
198of the command.
199
200The stages are identified by the following strings:
201
202  - pre  (pre-suite)
203  - setup
204  - command
205  - verify
206  - teardown
207  - post (post-suite)
208
209
210To write a plugin, you need to inherit from TdcPlugin in
211TdcPlugin.py.  To use the plugin, you have to put the
212implementation file in plugin-lib, and add a symbolic link to it from
213plugins.  It will be detected at run time and invoked at the
214appropriate times.  There are a few examples in the plugin-lib
215directory:
216
217  - rootPlugin.py:
218      implements the enforcement of running as root
219  - nsPlugin.py:
220      sets up a network namespace and runs all commands in that namespace
221  - valgrindPlugin.py
222      runs each command in the execute stage under valgrind,
223      and checks for leaks.
224      This plugin will output an extra test for each test in the test file,
225      one is the existing output as to whether the test passed or failed,
226      and the other is a test whether the command leaked memory or not.
227      (This one is a preliminary version, it may not work quite right yet,
228      but the overall template is there and it should only need tweaks.)
229
230
231ACKNOWLEDGEMENTS
232----------------
233
234Thanks to:
235
236Jamal Hadi Salim, for providing valuable test cases
237Keara Leibovitz, who wrote the CLI test driver that I used as a base for the
238   first version of the tc testing suite. This work was presented at
239   Netdev 1.2 Tokyo in October 2016.
240Samir Hussain, for providing help while I dove into Python for the first time
241    and being a second eye for this code.
242