1.. SPDX-License-Identifier: GPL-2.0
2
3===========
4Using KUnit
5===========
6
7The purpose of this document is to describe what KUnit is, how it works, how it
8is intended to be used, and all the concepts and terminology that are needed to
9understand it. This guide assumes a working knowledge of the Linux kernel and
10some basic knowledge of testing.
11
12For a high level introduction to KUnit, including setting up KUnit for your
13project, see :doc:`start`.
14
15Organization of this document
16=============================
17
18This document is organized into two main sections: Testing and Isolating
19Behavior. The first covers what unit tests are and how to use KUnit to write
20them. The second covers how to use KUnit to isolate code and make it possible
21to unit test code that was otherwise un-unit-testable.
22
23Testing
24=======
25
26What is KUnit?
27--------------
28
29"K" is short for "kernel" so "KUnit" is the "(Linux) Kernel Unit Testing
30Framework." KUnit is intended first and foremost for writing unit tests; it is
31general enough that it can be used to write integration tests; however, this is
32a secondary goal. KUnit has no ambition of being the only testing framework for
33the kernel; for example, it does not intend to be an end-to-end testing
34framework.
35
36What is Unit Testing?
37---------------------
38
39A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is a test that
40tests code at the smallest possible scope, a *unit* of code. In the C
41programming language that's a function.
42
43Unit tests should be written for all the publicly exposed functions in a
44compilation unit; so that is all the functions that are exported in either a
45*class* (defined below) or all functions which are **not** static.
46
47Writing Tests
48-------------
49
50Test Cases
51~~~~~~~~~~
52
53The fundamental unit in KUnit is the test case. A test case is a function with
54the signature ``void (*)(struct kunit *test)``. It calls a function to be tested
55and then sets *expectations* for what should happen. For example:
56
57.. code-block:: c
58
59	void example_test_success(struct kunit *test)
60	{
61	}
62
63	void example_test_failure(struct kunit *test)
64	{
65		KUNIT_FAIL(test, "This test never passes.");
66	}
67
68In the above example ``example_test_success`` always passes because it does
69nothing; no expectations are set, so all expectations pass. On the other hand
70``example_test_failure`` always fails because it calls ``KUNIT_FAIL``, which is
71a special expectation that logs a message and causes the test case to fail.
72
73Expectations
74~~~~~~~~~~~~
75An *expectation* is a way to specify that you expect a piece of code to do
76something in a test. An expectation is called like a function. A test is made
77by setting expectations about the behavior of a piece of code under test; when
78one or more of the expectations fail, the test case fails and information about
79the failure is logged. For example:
80
81.. code-block:: c
82
83	void add_test_basic(struct kunit *test)
84	{
85		KUNIT_EXPECT_EQ(test, 1, add(1, 0));
86		KUNIT_EXPECT_EQ(test, 2, add(1, 1));
87	}
88
89In the above example ``add_test_basic`` makes a number of assertions about the
90behavior of a function called ``add``; the first parameter is always of type
91``struct kunit *``, which contains information about the current test context;
92the second parameter, in this case, is what the value is expected to be; the
93last value is what the value actually is. If ``add`` passes all of these
94expectations, the test case, ``add_test_basic`` will pass; if any one of these
95expectations fail, the test case will fail.
96
97It is important to understand that a test case *fails* when any expectation is
98violated; however, the test will continue running, potentially trying other
99expectations until the test case ends or is otherwise terminated. This is as
100opposed to *assertions* which are discussed later.
101
102To learn about more expectations supported by KUnit, see :doc:`api/test`.
103
104.. note::
105   A single test case should be pretty short, pretty easy to understand,
106   focused on a single behavior.
107
108For example, if we wanted to properly test the add function above, we would
109create additional tests cases which would each test a different property that an
110add function should have like this:
111
112.. code-block:: c
113
114	void add_test_basic(struct kunit *test)
115	{
116		KUNIT_EXPECT_EQ(test, 1, add(1, 0));
117		KUNIT_EXPECT_EQ(test, 2, add(1, 1));
118	}
119
120	void add_test_negative(struct kunit *test)
121	{
122		KUNIT_EXPECT_EQ(test, 0, add(-1, 1));
123	}
124
125	void add_test_max(struct kunit *test)
126	{
127		KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX));
128		KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN));
129	}
130
131	void add_test_overflow(struct kunit *test)
132	{
133		KUNIT_EXPECT_EQ(test, INT_MIN, add(INT_MAX, 1));
134	}
135
136Notice how it is immediately obvious what all the properties that we are testing
137for are.
138
139Assertions
140~~~~~~~~~~
141
142KUnit also has the concept of an *assertion*. An assertion is just like an
143expectation except the assertion immediately terminates the test case if it is
144not satisfied.
145
146For example:
147
148.. code-block:: c
149
150	static void mock_test_do_expect_default_return(struct kunit *test)
151	{
152		struct mock_test_context *ctx = test->priv;
153		struct mock *mock = ctx->mock;
154		int param0 = 5, param1 = -5;
155		const char *two_param_types[] = {"int", "int"};
156		const void *two_params[] = {&param0, &param1};
157		const void *ret;
158
159		ret = mock->do_expect(mock,
160				      "test_printk", test_printk,
161				      two_param_types, two_params,
162				      ARRAY_SIZE(two_params));
163		KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ret);
164		KUNIT_EXPECT_EQ(test, -4, *((int *) ret));
165	}
166
167In this example, the method under test should return a pointer to a value, so
168if the pointer returned by the method is null or an errno, we don't want to
169bother continuing the test since the following expectation could crash the test
170case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us to bail out of the test case if
171the appropriate conditions have not been satisfied to complete the test.
172
173Test Suites
174~~~~~~~~~~~
175
176Now obviously one unit test isn't very helpful; the power comes from having
177many test cases covering all of a unit's behaviors. Consequently it is common
178to have many *similar* tests; in order to reduce duplication in these closely
179related tests most unit testing frameworks - including KUnit - provide the
180concept of a *test suite*. A *test suite* is just a collection of test cases
181for a unit of code with a set up function that gets invoked before every test
182case and then a tear down function that gets invoked after every test case
183completes.
184
185Example:
186
187.. code-block:: c
188
189	static struct kunit_case example_test_cases[] = {
190		KUNIT_CASE(example_test_foo),
191		KUNIT_CASE(example_test_bar),
192		KUNIT_CASE(example_test_baz),
193		{}
194	};
195
196	static struct kunit_suite example_test_suite = {
197		.name = "example",
198		.init = example_test_init,
199		.exit = example_test_exit,
200		.test_cases = example_test_cases,
201	};
202	kunit_test_suite(example_test_suite);
203
204In the above example the test suite, ``example_test_suite``, would run the test
205cases ``example_test_foo``, ``example_test_bar``, and ``example_test_baz``,
206each would have ``example_test_init`` called immediately before it and would
207have ``example_test_exit`` called immediately after it.
208``kunit_test_suite(example_test_suite)`` registers the test suite with the
209KUnit test framework.
210
211.. note::
212   A test case will only be run if it is associated with a test suite.
213
214For more information on these types of things see the :doc:`api/test`.
215
216Isolating Behavior
217==================
218
219The most important aspect of unit testing that other forms of testing do not
220provide is the ability to limit the amount of code under test to a single unit.
221In practice, this is only possible by being able to control what code gets run
222when the unit under test calls a function and this is usually accomplished
223through some sort of indirection where a function is exposed as part of an API
224such that the definition of that function can be changed without affecting the
225rest of the code base. In the kernel this primarily comes from two constructs,
226classes, structs that contain function pointers that are provided by the
227implementer, and architecture specific functions which have definitions selected
228at compile time.
229
230Classes
231-------
232
233Classes are not a construct that is built into the C programming language;
234however, it is an easily derived concept. Accordingly, pretty much every project
235that does not use a standardized object oriented library (like GNOME's GObject)
236has their own slightly different way of doing object oriented programming; the
237Linux kernel is no exception.
238
239The central concept in kernel object oriented programming is the class. In the
240kernel, a *class* is a struct that contains function pointers. This creates a
241contract between *implementers* and *users* since it forces them to use the
242same function signature without having to call the function directly. In order
243for it to truly be a class, the function pointers must specify that a pointer
244to the class, known as a *class handle*, be one of the parameters; this makes
245it possible for the member functions (also known as *methods*) to have access
246to member variables (more commonly known as *fields*) allowing the same
247implementation to have multiple *instances*.
248
249Typically a class can be *overridden* by *child classes* by embedding the
250*parent class* in the child class. Then when a method provided by the child
251class is called, the child implementation knows that the pointer passed to it is
252of a parent contained within the child; because of this, the child can compute
253the pointer to itself because the pointer to the parent is always a fixed offset
254from the pointer to the child; this offset is the offset of the parent contained
255in the child struct. For example:
256
257.. code-block:: c
258
259	struct shape {
260		int (*area)(struct shape *this);
261	};
262
263	struct rectangle {
264		struct shape parent;
265		int length;
266		int width;
267	};
268
269	int rectangle_area(struct shape *this)
270	{
271		struct rectangle *self = container_of(this, struct shape, parent);
272
273		return self->length * self->width;
274	};
275
276	void rectangle_new(struct rectangle *self, int length, int width)
277	{
278		self->parent.area = rectangle_area;
279		self->length = length;
280		self->width = width;
281	}
282
283In this example (as in most kernel code) the operation of computing the pointer
284to the child from the pointer to the parent is done by ``container_of``.
285
286Faking Classes
287~~~~~~~~~~~~~~
288
289In order to unit test a piece of code that calls a method in a class, the
290behavior of the method must be controllable, otherwise the test ceases to be a
291unit test and becomes an integration test.
292
293A fake just provides an implementation of a piece of code that is different than
294what runs in a production instance, but behaves identically from the standpoint
295of the callers; this is usually done to replace a dependency that is hard to
296deal with, or is slow.
297
298A good example for this might be implementing a fake EEPROM that just stores the
299"contents" in an internal buffer. For example, let's assume we have a class that
300represents an EEPROM:
301
302.. code-block:: c
303
304	struct eeprom {
305		ssize_t (*read)(struct eeprom *this, size_t offset, char *buffer, size_t count);
306		ssize_t (*write)(struct eeprom *this, size_t offset, const char *buffer, size_t count);
307	};
308
309And we want to test some code that buffers writes to the EEPROM:
310
311.. code-block:: c
312
313	struct eeprom_buffer {
314		ssize_t (*write)(struct eeprom_buffer *this, const char *buffer, size_t count);
315		int flush(struct eeprom_buffer *this);
316		size_t flush_count; /* Flushes when buffer exceeds flush_count. */
317	};
318
319	struct eeprom_buffer *new_eeprom_buffer(struct eeprom *eeprom);
320	void destroy_eeprom_buffer(struct eeprom *eeprom);
321
322We can easily test this code by *faking out* the underlying EEPROM:
323
324.. code-block:: c
325
326	struct fake_eeprom {
327		struct eeprom parent;
328		char contents[FAKE_EEPROM_CONTENTS_SIZE];
329	};
330
331	ssize_t fake_eeprom_read(struct eeprom *parent, size_t offset, char *buffer, size_t count)
332	{
333		struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent);
334
335		count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset);
336		memcpy(buffer, this->contents + offset, count);
337
338		return count;
339	}
340
341	ssize_t fake_eeprom_write(struct eeprom *parent, size_t offset, const char *buffer, size_t count)
342	{
343		struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent);
344
345		count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset);
346		memcpy(this->contents + offset, buffer, count);
347
348		return count;
349	}
350
351	void fake_eeprom_init(struct fake_eeprom *this)
352	{
353		this->parent.read = fake_eeprom_read;
354		this->parent.write = fake_eeprom_write;
355		memset(this->contents, 0, FAKE_EEPROM_CONTENTS_SIZE);
356	}
357
358We can now use it to test ``struct eeprom_buffer``:
359
360.. code-block:: c
361
362	struct eeprom_buffer_test {
363		struct fake_eeprom *fake_eeprom;
364		struct eeprom_buffer *eeprom_buffer;
365	};
366
367	static void eeprom_buffer_test_does_not_write_until_flush(struct kunit *test)
368	{
369		struct eeprom_buffer_test *ctx = test->priv;
370		struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
371		struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
372		char buffer[] = {0xff};
373
374		eeprom_buffer->flush_count = SIZE_MAX;
375
376		eeprom_buffer->write(eeprom_buffer, buffer, 1);
377		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
378
379		eeprom_buffer->write(eeprom_buffer, buffer, 1);
380		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0);
381
382		eeprom_buffer->flush(eeprom_buffer);
383		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
384		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
385	}
386
387	static void eeprom_buffer_test_flushes_after_flush_count_met(struct kunit *test)
388	{
389		struct eeprom_buffer_test *ctx = test->priv;
390		struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
391		struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
392		char buffer[] = {0xff};
393
394		eeprom_buffer->flush_count = 2;
395
396		eeprom_buffer->write(eeprom_buffer, buffer, 1);
397		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
398
399		eeprom_buffer->write(eeprom_buffer, buffer, 1);
400		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
401		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
402	}
403
404	static void eeprom_buffer_test_flushes_increments_of_flush_count(struct kunit *test)
405	{
406		struct eeprom_buffer_test *ctx = test->priv;
407		struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer;
408		struct fake_eeprom *fake_eeprom = ctx->fake_eeprom;
409		char buffer[] = {0xff, 0xff};
410
411		eeprom_buffer->flush_count = 2;
412
413		eeprom_buffer->write(eeprom_buffer, buffer, 1);
414		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0);
415
416		eeprom_buffer->write(eeprom_buffer, buffer, 2);
417		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff);
418		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff);
419		/* Should have only flushed the first two bytes. */
420		KUNIT_EXPECT_EQ(test, fake_eeprom->contents[2], 0);
421	}
422
423	static int eeprom_buffer_test_init(struct kunit *test)
424	{
425		struct eeprom_buffer_test *ctx;
426
427		ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
428		KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx);
429
430		ctx->fake_eeprom = kunit_kzalloc(test, sizeof(*ctx->fake_eeprom), GFP_KERNEL);
431		KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->fake_eeprom);
432		fake_eeprom_init(ctx->fake_eeprom);
433
434		ctx->eeprom_buffer = new_eeprom_buffer(&ctx->fake_eeprom->parent);
435		KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->eeprom_buffer);
436
437		test->priv = ctx;
438
439		return 0;
440	}
441
442	static void eeprom_buffer_test_exit(struct kunit *test)
443	{
444		struct eeprom_buffer_test *ctx = test->priv;
445
446		destroy_eeprom_buffer(ctx->eeprom_buffer);
447	}
448
449.. _kunit-on-non-uml:
450
451KUnit on non-UML architectures
452==============================
453
454By default KUnit uses UML as a way to provide dependencies for code under test.
455Under most circumstances KUnit's usage of UML should be treated as an
456implementation detail of how KUnit works under the hood. Nevertheless, there
457are instances where being able to run architecture specific code or test
458against real hardware is desirable. For these reasons KUnit supports running on
459other architectures.
460
461Running existing KUnit tests on non-UML architectures
462-----------------------------------------------------
463
464There are some special considerations when running existing KUnit tests on
465non-UML architectures:
466
467*   Hardware may not be deterministic, so a test that always passes or fails
468    when run under UML may not always do so on real hardware.
469*   Hardware and VM environments may not be hermetic. KUnit tries its best to
470    provide a hermetic environment to run tests; however, it cannot manage state
471    that it doesn't know about outside of the kernel. Consequently, tests that
472    may be hermetic on UML may not be hermetic on other architectures.
473*   Some features and tooling may not be supported outside of UML.
474*   Hardware and VMs are slower than UML.
475
476None of these are reasons not to run your KUnit tests on real hardware; they are
477only things to be aware of when doing so.
478
479The biggest impediment will likely be that certain KUnit features and
480infrastructure may not support your target environment. For example, at this
481time the KUnit Wrapper (``tools/testing/kunit/kunit.py``) does not work outside
482of UML. Unfortunately, there is no way around this. Using UML (or even just a
483particular architecture) allows us to make a lot of assumptions that make it
484possible to do things which might otherwise be impossible.
485
486Nevertheless, all core KUnit framework features are fully supported on all
487architectures, and using them is straightforward: all you need to do is to take
488your kunitconfig, your Kconfig options for the tests you would like to run, and
489merge them into whatever config your are using for your platform. That's it!
490
491For example, let's say you have the following kunitconfig:
492
493.. code-block:: none
494
495	CONFIG_KUNIT=y
496	CONFIG_KUNIT_EXAMPLE_TEST=y
497
498If you wanted to run this test on an x86 VM, you might add the following config
499options to your ``.config``:
500
501.. code-block:: none
502
503	CONFIG_KUNIT=y
504	CONFIG_KUNIT_EXAMPLE_TEST=y
505	CONFIG_SERIAL_8250=y
506	CONFIG_SERIAL_8250_CONSOLE=y
507
508All these new options do is enable support for a common serial console needed
509for logging.
510
511Next, you could build a kernel with these tests as follows:
512
513
514.. code-block:: bash
515
516	make ARCH=x86 olddefconfig
517	make ARCH=x86
518
519Once you have built a kernel, you could run it on QEMU as follows:
520
521.. code-block:: bash
522
523	qemu-system-x86_64 -enable-kvm \
524			   -m 1024 \
525			   -kernel arch/x86_64/boot/bzImage \
526			   -append 'console=ttyS0' \
527			   --nographic
528
529Interspersed in the kernel logs you might see the following:
530
531.. code-block:: none
532
533	TAP version 14
534		# Subtest: example
535		1..1
536		# example_simple_test: initializing
537		ok 1 - example_simple_test
538	ok 1 - example
539
540Congratulations, you just ran a KUnit test on the x86 architecture!
541
542In a similar manner, kunit and kunit tests can also be built as modules,
543so if you wanted to run tests in this way you might add the following config
544options to your ``.config``:
545
546.. code-block:: none
547
548	CONFIG_KUNIT=m
549	CONFIG_KUNIT_EXAMPLE_TEST=m
550
551Once the kernel is built and installed, a simple
552
553.. code-block:: bash
554
555	modprobe example-test
556
557...will run the tests.
558
559Writing new tests for other architectures
560-----------------------------------------
561
562The first thing you must do is ask yourself whether it is necessary to write a
563KUnit test for a specific architecture, and then whether it is necessary to
564write that test for a particular piece of hardware. In general, writing a test
565that depends on having access to a particular piece of hardware or software (not
566included in the Linux source repo) should be avoided at all costs.
567
568Even if you only ever plan on running your KUnit test on your hardware
569configuration, other people may want to run your tests and may not have access
570to your hardware. If you write your test to run on UML, then anyone can run your
571tests without knowing anything about your particular setup, and you can still
572run your tests on your hardware setup just by compiling for your architecture.
573
574.. important::
575   Always prefer tests that run on UML to tests that only run under a particular
576   architecture, and always prefer tests that run under QEMU or another easy
577   (and monetarily free) to obtain software environment to a specific piece of
578   hardware.
579
580Nevertheless, there are still valid reasons to write an architecture or hardware
581specific test: for example, you might want to test some code that really belongs
582in ``arch/some-arch/*``. Even so, try your best to write the test so that it
583does not depend on physical hardware: if some of your test cases don't need the
584hardware, only require the hardware for tests that actually need it.
585
586Now that you have narrowed down exactly what bits are hardware specific, the
587actual procedure for writing and running the tests is pretty much the same as
588writing normal KUnit tests. One special caveat is that you have to reset
589hardware state in between test cases; if this is not possible, you may only be
590able to run one test case per invocation.
591
592.. TODO(brendanhiggins@google.com): Add an actual example of an architecture
593   dependent KUnit test.
594
595KUnit debugfs representation
596============================
597When kunit test suites are initialized, they create an associated directory
598in /sys/kernel/debug/kunit/<test-suite>.  The directory contains one file
599
600- results: "cat results" displays results of each test case and the results
601  of the entire suite for the last test run.
602
603The debugfs representation is primarily of use when kunit test suites are
604run in a native environment, either as modules or builtin.  Having a way
605to display results like this is valuable as otherwise results can be
606intermixed with other events in dmesg output.  The maximum size of each
607results file is KUNIT_LOG_SIZE bytes (defined in include/kunit/test.h).
608