1================== 2BPF Selftest Notes 3================== 4General instructions on running selftests can be found in 5`Documentation/bpf/bpf_devel_QA.rst`__. 6 7__ /Documentation/bpf/bpf_devel_QA.rst#q-how-to-run-bpf-selftests 8 9============= 10BPF CI System 11============= 12 13BPF employs a continuous integration (CI) system to check patch submission in an 14automated fashion. The system runs selftests for each patch in a series. Results 15are propagated to patchwork, where failures are highlighted similar to 16violations of other checks (such as additional warnings being emitted or a 17``scripts/checkpatch.pl`` reported deficiency): 18 19 https://patchwork.kernel.org/project/netdevbpf/list/?delegate=121173 20 21The CI system executes tests on multiple architectures. It uses a kernel 22configuration derived from both the generic and architecture specific config 23file fragments below ``tools/testing/selftests/bpf/`` (e.g., ``config`` and 24``config.x86_64``). 25 26Denylisting Tests 27================= 28 29It is possible for some architectures to not have support for all BPF features. 30In such a case tests in CI may fail. An example of such a shortcoming is BPF 31trampoline support on IBM's s390x architecture. For cases like this, an in-tree 32deny list file, located at ``tools/testing/selftests/bpf/DENYLIST.<arch>``, can 33be used to prevent the test from running on such an architecture. 34 35In addition to that, the generic ``tools/testing/selftests/bpf/DENYLIST`` is 36honored on every architecture running tests. 37 38These files are organized in three columns. The first column lists the test in 39question. This can be the name of a test suite or of an individual test. The 40remaining two columns provide additional meta data that helps identify and 41classify the entry: column two is a copy and paste of the error being reported 42when running the test in the setting in question. The third column, if 43available, summarizes the underlying problem. A value of ``trampoline``, for 44example, indicates that lack of trampoline support is causing the test to fail. 45This last entry helps identify tests that can be re-enabled once such support is 46added. 47 48========================= 49Running Selftests in a VM 50========================= 51 52It's now possible to run the selftests using ``tools/testing/selftests/bpf/vmtest.sh``. 53The script tries to ensure that the tests are run with the same environment as they 54would be run post-submit in the CI used by the Maintainers, with the exception 55that deny lists are not automatically honored. 56 57This script uses the in-tree kernel configuration and downloads a VM userspace 58image from the system used by the CI. It builds the kernel (without overwriting 59your existing Kconfig), recompiles the bpf selftests, runs them (by default 60``tools/testing/selftests/bpf/test_progs``) and saves the resulting output (by 61default in ``~/.bpf_selftests``). 62 63Script dependencies: 64- clang (preferably built from sources, https://github.com/llvm/llvm-project); 65- pahole (preferably built from sources, https://git.kernel.org/pub/scm/devel/pahole/pahole.git/); 66- qemu; 67- docutils (for ``rst2man``); 68- libcap-devel. 69 70For more information about using the script, run: 71 72.. code-block:: console 73 74 $ tools/testing/selftests/bpf/vmtest.sh -h 75 76In case of linker errors when running selftests, try using static linking: 77 78.. code-block:: console 79 80 $ LDLIBS=-static vmtest.sh 81 82.. note:: Some distros may not support static linking. 83 84.. note:: The script uses pahole and clang based on host environment setting. 85 If you want to change pahole and llvm, you can change `PATH` environment 86 variable in the beginning of script. 87 88.. note:: The script currently only supports x86_64 and s390x architectures. 89 90Additional information about selftest failures are 91documented here. 92 93profiler[23] test failures with clang/llvm <12.0.0 94================================================== 95 96With clang/llvm <12.0.0, the profiler[23] test may fail. 97The symptom looks like 98 99.. code-block:: c 100 101 // r9 is a pointer to map_value 102 // r7 is a scalar 103 17: bf 96 00 00 00 00 00 00 r6 = r9 104 18: 0f 76 00 00 00 00 00 00 r6 += r7 105 math between map_value pointer and register with unbounded min value is not allowed 106 107 // the instructions below will not be seen in the verifier log 108 19: a5 07 01 00 01 01 00 00 if r7 < 257 goto +1 109 20: bf 96 00 00 00 00 00 00 r6 = r9 110 // r6 is used here 111 112The verifier will reject such code with above error. 113At insn 18 the r7 is indeed unbounded. The later insn 19 checks the bounds and 114the insn 20 undoes map_value addition. It is currently impossible for the 115verifier to understand such speculative pointer arithmetic. 116Hence `this patch`__ addresses it on the compiler side. It was committed on llvm 12. 117 118__ https://reviews.llvm.org/D85570 119 120The corresponding C code 121 122.. code-block:: c 123 124 for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; i++) { 125 filepart_length = bpf_probe_read_str(payload, ...); 126 if (filepart_length <= MAX_PATH) { 127 barrier_var(filepart_length); // workaround 128 payload += filepart_length; 129 } 130 } 131 132bpf_iter test failures with clang/llvm 10.0.0 133============================================= 134 135With clang/llvm 10.0.0, the following two bpf_iter tests failed: 136 * ``bpf_iter/ipv6_route`` 137 * ``bpf_iter/netlink`` 138 139The symptom for ``bpf_iter/ipv6_route`` looks like 140 141.. code-block:: c 142 143 2: (79) r8 = *(u64 *)(r1 +8) 144 ... 145 14: (bf) r2 = r8 146 15: (0f) r2 += r1 147 ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen); 148 16: (7b) *(u64 *)(r8 +64) = r2 149 only read is supported 150 151The symptom for ``bpf_iter/netlink`` looks like 152 153.. code-block:: c 154 155 ; struct netlink_sock *nlk = ctx->sk; 156 2: (79) r7 = *(u64 *)(r1 +8) 157 ... 158 15: (bf) r2 = r7 159 16: (0f) r2 += r1 160 ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_protocol); 161 17: (7b) *(u64 *)(r7 +0) = r2 162 only read is supported 163 164This is due to a llvm BPF backend bug. `The fix`__ 165has been pushed to llvm 10.x release branch and will be 166available in 10.0.1. The patch is available in llvm 11.0.0 trunk. 167 168__ https://reviews.llvm.org/D78466 169 170bpf_verif_scale/loop6.bpf.o test failure with Clang 12 171====================================================== 172 173With Clang 12, the following bpf_verif_scale test failed: 174 * ``bpf_verif_scale/loop6.bpf.o`` 175 176The verifier output looks like 177 178.. code-block:: c 179 180 R1 type=ctx expected=fp 181 The sequence of 8193 jumps is too complex. 182 183The reason is compiler generating the following code 184 185.. code-block:: c 186 187 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) { 188 14: 16 05 40 00 00 00 00 00 if w5 == 0 goto +64 <LBB0_6> 189 15: bc 51 00 00 00 00 00 00 w1 = w5 190 16: 04 01 00 00 ff ff ff ff w1 += -1 191 17: 67 05 00 00 20 00 00 00 r5 <<= 32 192 18: 77 05 00 00 20 00 00 00 r5 >>= 32 193 19: a6 01 01 00 05 00 00 00 if w1 < 5 goto +1 <LBB0_4> 194 20: b7 05 00 00 06 00 00 00 r5 = 6 195 00000000000000a8 <LBB0_4>: 196 21: b7 02 00 00 00 00 00 00 r2 = 0 197 22: b7 01 00 00 00 00 00 00 r1 = 0 198 ; for (i = 0; (i < VIRTIO_MAX_SGS) && (i < num); i++) { 199 23: 7b 1a e0 ff 00 00 00 00 *(u64 *)(r10 - 32) = r1 200 24: 7b 5a c0 ff 00 00 00 00 *(u64 *)(r10 - 64) = r5 201 202Note that insn #15 has w1 = w5 and w1 is refined later but 203r5(w5) is eventually saved on stack at insn #24 for later use. 204This cause later verifier failure. The bug has been `fixed`__ in 205Clang 13. 206 207__ https://reviews.llvm.org/D97479 208 209BPF CO-RE-based tests and Clang version 210======================================= 211 212A set of selftests use BPF target-specific built-ins, which might require 213bleeding-edge Clang versions (Clang 12 nightly at this time). 214 215Few sub-tests of core_reloc test suit (part of test_progs test runner) require 216the following built-ins, listed with corresponding Clang diffs introducing 217them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too 218old to support them, they shouldn't cause build failures or runtime test 219failures: 220 221- __builtin_btf_type_id() [0_, 1_, 2_]; 222- __builtin_preserve_type_info(), __builtin_preserve_enum_value() [3_, 4_]. 223 224.. _0: https://reviews.llvm.org/D74572 225.. _1: https://reviews.llvm.org/D74668 226.. _2: https://reviews.llvm.org/D85174 227.. _3: https://reviews.llvm.org/D83878 228.. _4: https://reviews.llvm.org/D83242 229 230Floating-point tests and Clang version 231====================================== 232 233Certain selftests, e.g. core_reloc, require support for the floating-point 234types, which was introduced in `Clang 13`__. The older Clang versions will 235either crash when compiling these tests, or generate an incorrect BTF. 236 237__ https://reviews.llvm.org/D83289 238 239Kernel function call test and Clang version 240=========================================== 241 242Some selftests (e.g. kfunc_call and bpf_tcp_ca) require a LLVM support 243to generate extern function in BTF. It was introduced in `Clang 13`__. 244 245Without it, the error from compiling bpf selftests looks like: 246 247.. code-block:: console 248 249 libbpf: failed to find BTF for extern 'tcp_slow_start' [25] section: -2 250 251__ https://reviews.llvm.org/D93563 252 253btf_tag test and Clang version 254============================== 255 256The btf_tag selftest requires LLVM support to recognize the btf_decl_tag and 257btf_type_tag attributes. They are introduced in `Clang 14` [0_, 1_]. 258The subtests ``btf_type_tag_user_{mod1, mod2, vmlinux}`` also requires 259pahole version ``1.23``. 260 261Without them, the btf_tag selftest will be skipped and you will observe: 262 263.. code-block:: console 264 265 #<test_num> btf_tag:SKIP 266 267.. _0: https://reviews.llvm.org/D111588 268.. _1: https://reviews.llvm.org/D111199 269 270Clang dependencies for static linking tests 271=========================================== 272 273linked_vars, linked_maps, and linked_funcs tests depend on `Clang fix`__ to 274generate valid BTF information for weak variables. Please make sure you use 275Clang that contains the fix. 276 277__ https://reviews.llvm.org/D100362 278 279Clang relocation changes 280======================== 281 282Clang 13 patch `clang reloc patch`_ made some changes on relocations such 283that existing relocation types are broken into more types and 284each new type corresponds to only one way to resolve relocation. 285See `kernel llvm reloc`_ for more explanation and some examples. 286Using clang 13 to compile old libbpf which has static linker support, 287there will be a compilation failure:: 288 289 libbpf: ELF relo #0 in section #6 has unexpected type 2 in .../bpf_tcp_nogpl.bpf.o 290 291Here, ``type 2`` refers to new relocation type ``R_BPF_64_ABS64``. 292To fix this issue, user newer libbpf. 293 294.. Links 295.. _clang reloc patch: https://reviews.llvm.org/D102712 296.. _kernel llvm reloc: /Documentation/bpf/llvm_reloc.rst 297 298Clang dependencies for the u32 spill test (xdpwall) 299=================================================== 300The xdpwall selftest requires a change in `Clang 14`__. 301 302Without it, the xdpwall selftest will fail and the error message 303from running test_progs will look like: 304 305.. code-block:: console 306 307 test_xdpwall:FAIL:Does LLVM have https://reviews.llvm.org/D109073? unexpected error: -4007 308 309__ https://reviews.llvm.org/D109073 310