1================================= 2HOWTO interact with BPF subsystem 3================================= 4 5This document provides information for the BPF subsystem about various 6workflows related to reporting bugs, submitting patches, and queueing 7patches for stable kernels. 8 9For general information about submitting patches, please refer to 10`Documentation/process/`_. This document only describes additional specifics 11related to BPF. 12 13.. contents:: 14 :local: 15 :depth: 2 16 17Reporting bugs 18============== 19 20Q: How do I report bugs for BPF kernel code? 21-------------------------------------------- 22A: Since all BPF kernel development as well as bpftool and iproute2 BPF 23loader development happens through the netdev kernel mailing list, 24please report any found issues around BPF to the following mailing 25list: 26 27 netdev@vger.kernel.org 28 29This may also include issues related to XDP, BPF tracing, etc. 30 31Given netdev has a high volume of traffic, please also add the BPF 32maintainers to Cc (from kernel MAINTAINERS_ file): 33 34* Alexei Starovoitov <ast@kernel.org> 35* Daniel Borkmann <daniel@iogearbox.net> 36 37In case a buggy commit has already been identified, make sure to keep 38the actual commit authors in Cc as well for the report. They can 39typically be identified through the kernel's git tree. 40 41**Please do NOT report BPF issues to bugzilla.kernel.org since it 42is a guarantee that the reported issue will be overlooked.** 43 44Submitting patches 45================== 46 47Q: To which mailing list do I need to submit my BPF patches? 48------------------------------------------------------------ 49A: Please submit your BPF patches to the netdev kernel mailing list: 50 51 netdev@vger.kernel.org 52 53Historically, BPF came out of networking and has always been maintained 54by the kernel networking community. Although these days BPF touches 55many other subsystems as well, the patches are still routed mainly 56through the networking community. 57 58In case your patch has changes in various different subsystems (e.g. 59tracing, security, etc), make sure to Cc the related kernel mailing 60lists and maintainers from there as well, so they are able to review 61the changes and provide their Acked-by's to the patches. 62 63Q: Where can I find patches currently under discussion for BPF subsystem? 64------------------------------------------------------------------------- 65A: All patches that are Cc'ed to netdev are queued for review under netdev 66patchwork project: 67 68 http://patchwork.ozlabs.org/project/netdev/list/ 69 70Those patches which target BPF, are assigned to a 'bpf' delegate for 71further processing from BPF maintainers. The current queue with 72patches under review can be found at: 73 74 https://patchwork.ozlabs.org/project/netdev/list/?delegate=77147 75 76Once the patches have been reviewed by the BPF community as a whole 77and approved by the BPF maintainers, their status in patchwork will be 78changed to 'Accepted' and the submitter will be notified by mail. This 79means that the patches look good from a BPF perspective and have been 80applied to one of the two BPF kernel trees. 81 82In case feedback from the community requires a respin of the patches, 83their status in patchwork will be set to 'Changes Requested', and purged 84from the current review queue. Likewise for cases where patches would 85get rejected or are not applicable to the BPF trees (but assigned to 86the 'bpf' delegate). 87 88Q: How do the changes make their way into Linux? 89------------------------------------------------ 90A: There are two BPF kernel trees (git repositories). Once patches have 91been accepted by the BPF maintainers, they will be applied to one 92of the two BPF trees: 93 94 * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/ 95 * https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/ 96 97The bpf tree itself is for fixes only, whereas bpf-next for features, 98cleanups or other kind of improvements ("next-like" content). This is 99analogous to net and net-next trees for networking. Both bpf and 100bpf-next will only have a master branch in order to simplify against 101which branch patches should get rebased to. 102 103Accumulated BPF patches in the bpf tree will regularly get pulled 104into the net kernel tree. Likewise, accumulated BPF patches accepted 105into the bpf-next tree will make their way into net-next tree. net and 106net-next are both run by David S. Miller. From there, they will go 107into the kernel mainline tree run by Linus Torvalds. To read up on the 108process of net and net-next being merged into the mainline tree, see 109the `netdev FAQ`_ under: 110 111 `Documentation/networking/netdev-FAQ.txt`_ 112 113Occasionally, to prevent merge conflicts, we might send pull requests 114to other trees (e.g. tracing) with a small subset of the patches, but 115net and net-next are always the main trees targeted for integration. 116 117The pull requests will contain a high-level summary of the accumulated 118patches and can be searched on netdev kernel mailing list through the 119following subject lines (``yyyy-mm-dd`` is the date of the pull 120request):: 121 122 pull-request: bpf yyyy-mm-dd 123 pull-request: bpf-next yyyy-mm-dd 124 125Q: How do I indicate which tree (bpf vs. bpf-next) my patch should be applied to? 126--------------------------------------------------------------------------------- 127 128A: The process is the very same as described in the `netdev FAQ`_, so 129please read up on it. The subject line must indicate whether the 130patch is a fix or rather "next-like" content in order to let the 131maintainers know whether it is targeted at bpf or bpf-next. 132 133For fixes eventually landing in bpf -> net tree, the subject must 134look like:: 135 136 git format-patch --subject-prefix='PATCH bpf' start..finish 137 138For features/improvements/etc that should eventually land in 139bpf-next -> net-next, the subject must look like:: 140 141 git format-patch --subject-prefix='PATCH bpf-next' start..finish 142 143If unsure whether the patch or patch series should go into bpf 144or net directly, or bpf-next or net-next directly, it is not a 145problem either if the subject line says net or net-next as target. 146It is eventually up to the maintainers to do the delegation of 147the patches. 148 149If it is clear that patches should go into bpf or bpf-next tree, 150please make sure to rebase the patches against those trees in 151order to reduce potential conflicts. 152 153In case the patch or patch series has to be reworked and sent out 154again in a second or later revision, it is also required to add a 155version number (``v2``, ``v3``, ...) into the subject prefix:: 156 157 git format-patch --subject-prefix='PATCH net-next v2' start..finish 158 159When changes have been requested to the patch series, always send the 160whole patch series again with the feedback incorporated (never send 161individual diffs on top of the old series). 162 163Q: What does it mean when a patch gets applied to bpf or bpf-next tree? 164----------------------------------------------------------------------- 165A: It means that the patch looks good for mainline inclusion from 166a BPF point of view. 167 168Be aware that this is not a final verdict that the patch will 169automatically get accepted into net or net-next trees eventually: 170 171On the netdev kernel mailing list reviews can come in at any point 172in time. If discussions around a patch conclude that they cannot 173get included as-is, we will either apply a follow-up fix or drop 174them from the trees entirely. Therefore, we also reserve to rebase 175the trees when deemed necessary. After all, the purpose of the tree 176is to: 177 178i) accumulate and stage BPF patches for integration into trees 179 like net and net-next, and 180 181ii) run extensive BPF test suite and 182 workloads on the patches before they make their way any further. 183 184Once the BPF pull request was accepted by David S. Miller, then 185the patches end up in net or net-next tree, respectively, and 186make their way from there further into mainline. Again, see the 187`netdev FAQ`_ for additional information e.g. on how often they are 188merged to mainline. 189 190Q: How long do I need to wait for feedback on my BPF patches? 191------------------------------------------------------------- 192A: We try to keep the latency low. The usual time to feedback will 193be around 2 or 3 business days. It may vary depending on the 194complexity of changes and current patch load. 195 196Q: How often do you send pull requests to major kernel trees like net or net-next? 197---------------------------------------------------------------------------------- 198 199A: Pull requests will be sent out rather often in order to not 200accumulate too many patches in bpf or bpf-next. 201 202As a rule of thumb, expect pull requests for each tree regularly 203at the end of the week. In some cases pull requests could additionally 204come also in the middle of the week depending on the current patch 205load or urgency. 206 207Q: Are patches applied to bpf-next when the merge window is open? 208----------------------------------------------------------------- 209A: For the time when the merge window is open, bpf-next will not be 210processed. This is roughly analogous to net-next patch processing, 211so feel free to read up on the `netdev FAQ`_ about further details. 212 213During those two weeks of merge window, we might ask you to resend 214your patch series once bpf-next is open again. Once Linus released 215a ``v*-rc1`` after the merge window, we continue processing of bpf-next. 216 217For non-subscribers to kernel mailing lists, there is also a status 218page run by David S. Miller on net-next that provides guidance: 219 220 http://vger.kernel.org/~davem/net-next.html 221 222Q: Verifier changes and test cases 223---------------------------------- 224Q: I made a BPF verifier change, do I need to add test cases for 225BPF kernel selftests_? 226 227A: If the patch has changes to the behavior of the verifier, then yes, 228it is absolutely necessary to add test cases to the BPF kernel 229selftests_ suite. If they are not present and we think they are 230needed, then we might ask for them before accepting any changes. 231 232In particular, test_verifier.c is tracking a high number of BPF test 233cases, including a lot of corner cases that LLVM BPF back end may 234generate out of the restricted C code. Thus, adding test cases is 235absolutely crucial to make sure future changes do not accidentally 236affect prior use-cases. Thus, treat those test cases as: verifier 237behavior that is not tracked in test_verifier.c could potentially 238be subject to change. 239 240Q: samples/bpf preference vs selftests? 241--------------------------------------- 242Q: When should I add code to `samples/bpf/`_ and when to BPF kernel 243selftests_ ? 244 245A: In general, we prefer additions to BPF kernel selftests_ rather than 246`samples/bpf/`_. The rationale is very simple: kernel selftests are 247regularly run by various bots to test for kernel regressions. 248 249The more test cases we add to BPF selftests, the better the coverage 250and the less likely it is that those could accidentally break. It is 251not that BPF kernel selftests cannot demo how a specific feature can 252be used. 253 254That said, `samples/bpf/`_ may be a good place for people to get started, 255so it might be advisable that simple demos of features could go into 256`samples/bpf/`_, but advanced functional and corner-case testing rather 257into kernel selftests. 258 259If your sample looks like a test case, then go for BPF kernel selftests 260instead! 261 262Q: When should I add code to the bpftool? 263----------------------------------------- 264A: The main purpose of bpftool (under tools/bpf/bpftool/) is to provide 265a central user space tool for debugging and introspection of BPF programs 266and maps that are active in the kernel. If UAPI changes related to BPF 267enable for dumping additional information of programs or maps, then 268bpftool should be extended as well to support dumping them. 269 270Q: When should I add code to iproute2's BPF loader? 271--------------------------------------------------- 272A: For UAPI changes related to the XDP or tc layer (e.g. ``cls_bpf``), 273the convention is that those control-path related changes are added to 274iproute2's BPF loader as well from user space side. This is not only 275useful to have UAPI changes properly designed to be usable, but also 276to make those changes available to a wider user base of major 277downstream distributions. 278 279Q: Do you accept patches as well for iproute2's BPF loader? 280----------------------------------------------------------- 281A: Patches for the iproute2's BPF loader have to be sent to: 282 283 netdev@vger.kernel.org 284 285While those patches are not processed by the BPF kernel maintainers, 286please keep them in Cc as well, so they can be reviewed. 287 288The official git repository for iproute2 is run by Stephen Hemminger 289and can be found at: 290 291 https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/ 292 293The patches need to have a subject prefix of '``[PATCH iproute2 294master]``' or '``[PATCH iproute2 net-next]``'. '``master``' or 295'``net-next``' describes the target branch where the patch should be 296applied to. Meaning, if kernel changes went into the net-next kernel 297tree, then the related iproute2 changes need to go into the iproute2 298net-next branch, otherwise they can be targeted at master branch. The 299iproute2 net-next branch will get merged into the master branch after 300the current iproute2 version from master has been released. 301 302Like BPF, the patches end up in patchwork under the netdev project and 303are delegated to 'shemminger' for further processing: 304 305 http://patchwork.ozlabs.org/project/netdev/list/?delegate=389 306 307Q: What is the minimum requirement before I submit my BPF patches? 308------------------------------------------------------------------ 309A: When submitting patches, always take the time and properly test your 310patches *prior* to submission. Never rush them! If maintainers find 311that your patches have not been properly tested, it is a good way to 312get them grumpy. Testing patch submissions is a hard requirement! 313 314Note, fixes that go to bpf tree *must* have a ``Fixes:`` tag included. 315The same applies to fixes that target bpf-next, where the affected 316commit is in net-next (or in some cases bpf-next). The ``Fixes:`` tag is 317crucial in order to identify follow-up commits and tremendously helps 318for people having to do backporting, so it is a must have! 319 320We also don't accept patches with an empty commit message. Take your 321time and properly write up a high quality commit message, it is 322essential! 323 324Think about it this way: other developers looking at your code a month 325from now need to understand *why* a certain change has been done that 326way, and whether there have been flaws in the analysis or assumptions 327that the original author did. Thus providing a proper rationale and 328describing the use-case for the changes is a must. 329 330Patch submissions with >1 patch must have a cover letter which includes 331a high level description of the series. This high level summary will 332then be placed into the merge commit by the BPF maintainers such that 333it is also accessible from the git log for future reference. 334 335Q: Features changing BPF JIT and/or LLVM 336---------------------------------------- 337Q: What do I need to consider when adding a new instruction or feature 338that would require BPF JIT and/or LLVM integration as well? 339 340A: We try hard to keep all BPF JITs up to date such that the same user 341experience can be guaranteed when running BPF programs on different 342architectures without having the program punt to the less efficient 343interpreter in case the in-kernel BPF JIT is enabled. 344 345If you are unable to implement or test the required JIT changes for 346certain architectures, please work together with the related BPF JIT 347developers in order to get the feature implemented in a timely manner. 348Please refer to the git log (``arch/*/net/``) to locate the necessary 349people for helping out. 350 351Also always make sure to add BPF test cases (e.g. test_bpf.c and 352test_verifier.c) for new instructions, so that they can receive 353broad test coverage and help run-time testing the various BPF JITs. 354 355In case of new BPF instructions, once the changes have been accepted 356into the Linux kernel, please implement support into LLVM's BPF back 357end. See LLVM_ section below for further information. 358 359Stable submission 360================= 361 362Q: I need a specific BPF commit in stable kernels. What should I do? 363-------------------------------------------------------------------- 364A: In case you need a specific fix in stable kernels, first check whether 365the commit has already been applied in the related ``linux-*.y`` branches: 366 367 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/ 368 369If not the case, then drop an email to the BPF maintainers with the 370netdev kernel mailing list in Cc and ask for the fix to be queued up: 371 372 netdev@vger.kernel.org 373 374The process in general is the same as on netdev itself, see also the 375`netdev FAQ`_ document. 376 377Q: Do you also backport to kernels not currently maintained as stable? 378---------------------------------------------------------------------- 379A: No. If you need a specific BPF commit in kernels that are currently not 380maintained by the stable maintainers, then you are on your own. 381 382The current stable and longterm stable kernels are all listed here: 383 384 https://www.kernel.org/ 385 386Q: The BPF patch I am about to submit needs to go to stable as well 387------------------------------------------------------------------- 388What should I do? 389 390A: The same rules apply as with netdev patch submissions in general, see 391`netdev FAQ`_ under: 392 393 `Documentation/networking/netdev-FAQ.txt`_ 394 395Never add "``Cc: stable@vger.kernel.org``" to the patch description, but 396ask the BPF maintainers to queue the patches instead. This can be done 397with a note, for example, under the ``---`` part of the patch which does 398not go into the git log. Alternatively, this can be done as a simple 399request by mail instead. 400 401Q: Queue stable patches 402----------------------- 403Q: Where do I find currently queued BPF patches that will be submitted 404to stable? 405 406A: Once patches that fix critical bugs got applied into the bpf tree, they 407are queued up for stable submission under: 408 409 http://patchwork.ozlabs.org/bundle/bpf/stable/?state=* 410 411They will be on hold there at minimum until the related commit made its 412way into the mainline kernel tree. 413 414After having been under broader exposure, the queued patches will be 415submitted by the BPF maintainers to the stable maintainers. 416 417Testing patches 418=============== 419 420Q: How to run BPF selftests 421--------------------------- 422A: After you have booted into the newly compiled kernel, navigate to 423the BPF selftests_ suite in order to test BPF functionality (current 424working directory points to the root of the cloned git tree):: 425 426 $ cd tools/testing/selftests/bpf/ 427 $ make 428 429To run the verifier tests:: 430 431 $ sudo ./test_verifier 432 433The verifier tests print out all the current checks being 434performed. The summary at the end of running all tests will dump 435information of test successes and failures:: 436 437 Summary: 418 PASSED, 0 FAILED 438 439In order to run through all BPF selftests, the following command is 440needed:: 441 442 $ sudo make run_tests 443 444See the kernels selftest `Documentation/dev-tools/kselftest.rst`_ 445document for further documentation. 446 447Q: Which BPF kernel selftests version should I run my kernel against? 448--------------------------------------------------------------------- 449A: If you run a kernel ``xyz``, then always run the BPF kernel selftests 450from that kernel ``xyz`` as well. Do not expect that the BPF selftest 451from the latest mainline tree will pass all the time. 452 453In particular, test_bpf.c and test_verifier.c have a large number of 454test cases and are constantly updated with new BPF test sequences, or 455existing ones are adapted to verifier changes e.g. due to verifier 456becoming smarter and being able to better track certain things. 457 458LLVM 459==== 460 461Q: Where do I find LLVM with BPF support? 462----------------------------------------- 463A: The BPF back end for LLVM is upstream in LLVM since version 3.7.1. 464 465All major distributions these days ship LLVM with BPF back end enabled, 466so for the majority of use-cases it is not required to compile LLVM by 467hand anymore, just install the distribution provided package. 468 469LLVM's static compiler lists the supported targets through 470``llc --version``, make sure BPF targets are listed. Example:: 471 472 $ llc --version 473 LLVM (http://llvm.org/): 474 LLVM version 6.0.0svn 475 Optimized build. 476 Default target: x86_64-unknown-linux-gnu 477 Host CPU: skylake 478 479 Registered Targets: 480 bpf - BPF (host endian) 481 bpfeb - BPF (big endian) 482 bpfel - BPF (little endian) 483 x86 - 32-bit X86: Pentium-Pro and above 484 x86-64 - 64-bit X86: EM64T and AMD64 485 486For developers in order to utilize the latest features added to LLVM's 487BPF back end, it is advisable to run the latest LLVM releases. Support 488for new BPF kernel features such as additions to the BPF instruction 489set are often developed together. 490 491All LLVM releases can be found at: http://releases.llvm.org/ 492 493Q: Got it, so how do I build LLVM manually anyway? 494-------------------------------------------------- 495A: You need cmake and gcc-c++ as build requisites for LLVM. Once you have 496that set up, proceed with building the latest LLVM and clang version 497from the git repositories:: 498 499 $ git clone http://llvm.org/git/llvm.git 500 $ cd llvm/tools 501 $ git clone --depth 1 http://llvm.org/git/clang.git 502 $ cd ..; mkdir build; cd build 503 $ cmake .. -DLLVM_TARGETS_TO_BUILD="BPF;X86" \ 504 -DBUILD_SHARED_LIBS=OFF \ 505 -DCMAKE_BUILD_TYPE=Release \ 506 -DLLVM_BUILD_RUNTIME=OFF 507 $ make -j $(getconf _NPROCESSORS_ONLN) 508 509The built binaries can then be found in the build/bin/ directory, where 510you can point the PATH variable to. 511 512Q: Reporting LLVM BPF issues 513---------------------------- 514Q: Should I notify BPF kernel maintainers about issues in LLVM's BPF code 515generation back end or about LLVM generated code that the verifier 516refuses to accept? 517 518A: Yes, please do! 519 520LLVM's BPF back end is a key piece of the whole BPF 521infrastructure and it ties deeply into verification of programs from the 522kernel side. Therefore, any issues on either side need to be investigated 523and fixed whenever necessary. 524 525Therefore, please make sure to bring them up at netdev kernel mailing 526list and Cc BPF maintainers for LLVM and kernel bits: 527 528* Yonghong Song <yhs@fb.com> 529* Alexei Starovoitov <ast@kernel.org> 530* Daniel Borkmann <daniel@iogearbox.net> 531 532LLVM also has an issue tracker where BPF related bugs can be found: 533 534 https://bugs.llvm.org/buglist.cgi?quicksearch=bpf 535 536However, it is better to reach out through mailing lists with having 537maintainers in Cc. 538 539Q: New BPF instruction for kernel and LLVM 540------------------------------------------ 541Q: I have added a new BPF instruction to the kernel, how can I integrate 542it into LLVM? 543 544A: LLVM has a ``-mcpu`` selector for the BPF back end in order to allow 545the selection of BPF instruction set extensions. By default the 546``generic`` processor target is used, which is the base instruction set 547(v1) of BPF. 548 549LLVM has an option to select ``-mcpu=probe`` where it will probe the host 550kernel for supported BPF instruction set extensions and selects the 551optimal set automatically. 552 553For cross-compilation, a specific version can be select manually as well :: 554 555 $ llc -march bpf -mcpu=help 556 Available CPUs for this target: 557 558 generic - Select the generic processor. 559 probe - Select the probe processor. 560 v1 - Select the v1 processor. 561 v2 - Select the v2 processor. 562 [...] 563 564Newly added BPF instructions to the Linux kernel need to follow the same 565scheme, bump the instruction set version and implement probing for the 566extensions such that ``-mcpu=probe`` users can benefit from the 567optimization transparently when upgrading their kernels. 568 569If you are unable to implement support for the newly added BPF instruction 570please reach out to BPF developers for help. 571 572By the way, the BPF kernel selftests run with ``-mcpu=probe`` for better 573test coverage. 574 575Q: clang flag for target bpf? 576----------------------------- 577Q: In some cases clang flag ``-target bpf`` is used but in other cases the 578default clang target, which matches the underlying architecture, is used. 579What is the difference and when I should use which? 580 581A: Although LLVM IR generation and optimization try to stay architecture 582independent, ``-target <arch>`` still has some impact on generated code: 583 584- BPF program may recursively include header file(s) with file scope 585 inline assembly codes. The default target can handle this well, 586 while ``bpf`` target may fail if bpf backend assembler does not 587 understand these assembly codes, which is true in most cases. 588 589- When compiled without ``-g``, additional elf sections, e.g., 590 .eh_frame and .rela.eh_frame, may be present in the object file 591 with default target, but not with ``bpf`` target. 592 593- The default target may turn a C switch statement into a switch table 594 lookup and jump operation. Since the switch table is placed 595 in the global readonly section, the bpf program will fail to load. 596 The bpf target does not support switch table optimization. 597 The clang option ``-fno-jump-tables`` can be used to disable 598 switch table generation. 599 600- For clang ``-target bpf``, it is guaranteed that pointer or long / 601 unsigned long types will always have a width of 64 bit, no matter 602 whether underlying clang binary or default target (or kernel) is 603 32 bit. However, when native clang target is used, then it will 604 compile these types based on the underlying architecture's conventions, 605 meaning in case of 32 bit architecture, pointer or long / unsigned 606 long types e.g. in BPF context structure will have width of 32 bit 607 while the BPF LLVM back end still operates in 64 bit. The native 608 target is mostly needed in tracing for the case of walking ``pt_regs`` 609 or other kernel structures where CPU's register width matters. 610 Otherwise, ``clang -target bpf`` is generally recommended. 611 612You should use default target when: 613 614- Your program includes a header file, e.g., ptrace.h, which eventually 615 pulls in some header files containing file scope host assembly codes. 616 617- You can add ``-fno-jump-tables`` to work around the switch table issue. 618 619Otherwise, you can use ``bpf`` target. Additionally, you *must* use bpf target 620when: 621 622- Your program uses data structures with pointer or long / unsigned long 623 types that interface with BPF helpers or context data structures. Access 624 into these structures is verified by the BPF verifier and may result 625 in verification failures if the native architecture is not aligned with 626 the BPF architecture, e.g. 64-bit. An example of this is 627 BPF_PROG_TYPE_SK_MSG require ``-target bpf`` 628 629 630.. Links 631.. _Documentation/process/: https://www.kernel.org/doc/html/latest/process/ 632.. _MAINTAINERS: ../../MAINTAINERS 633.. _Documentation/networking/netdev-FAQ.txt: ../networking/netdev-FAQ.txt 634.. _netdev FAQ: ../networking/netdev-FAQ.txt 635.. _samples/bpf/: ../../samples/bpf/ 636.. _selftests: ../../tools/testing/selftests/bpf/ 637.. _Documentation/dev-tools/kselftest.rst: 638 https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html 639 640Happy BPF hacking! 641