17e07e7aeSIlya Leoshkevich=================== 27e07e7aeSIlya LeoshkevichTesting BPF on s390 37e07e7aeSIlya Leoshkevich=================== 47e07e7aeSIlya Leoshkevich 57e07e7aeSIlya Leoshkevich1. Introduction 67e07e7aeSIlya Leoshkevich*************** 77e07e7aeSIlya Leoshkevich 87e07e7aeSIlya LeoshkevichIBM Z are mainframe computers, which are descendants of IBM System/360 from 97e07e7aeSIlya Leoshkevichyear 1964. They are supported by the Linux kernel under the name "s390". This 107e07e7aeSIlya Leoshkevichdocument describes how to test BPF in an s390 QEMU guest. 117e07e7aeSIlya Leoshkevich 127e07e7aeSIlya Leoshkevich2. One-time setup 137e07e7aeSIlya Leoshkevich***************** 147e07e7aeSIlya Leoshkevich 157e07e7aeSIlya LeoshkevichThe following is required to build and run the test suite: 167e07e7aeSIlya Leoshkevich 177e07e7aeSIlya Leoshkevich * s390 GCC 187e07e7aeSIlya Leoshkevich * s390 development headers and libraries 197e07e7aeSIlya Leoshkevich * Clang with BPF support 207e07e7aeSIlya Leoshkevich * QEMU with s390 support 217e07e7aeSIlya Leoshkevich * Disk image with s390 rootfs 227e07e7aeSIlya Leoshkevich 237e07e7aeSIlya LeoshkevichDebian supports installing compiler and libraries for s390 out of the box. 247e07e7aeSIlya LeoshkevichUsers of other distros may use debootstrap in order to set up a Debian chroot:: 257e07e7aeSIlya Leoshkevich 267e07e7aeSIlya Leoshkevich sudo debootstrap \ 277e07e7aeSIlya Leoshkevich --variant=minbase \ 287e07e7aeSIlya Leoshkevich --include=sudo \ 297e07e7aeSIlya Leoshkevich testing \ 307e07e7aeSIlya Leoshkevich ./s390-toolchain 317e07e7aeSIlya Leoshkevich sudo mount --rbind /dev ./s390-toolchain/dev 327e07e7aeSIlya Leoshkevich sudo mount --rbind /proc ./s390-toolchain/proc 337e07e7aeSIlya Leoshkevich sudo mount --rbind /sys ./s390-toolchain/sys 347e07e7aeSIlya Leoshkevich sudo chroot ./s390-toolchain 357e07e7aeSIlya Leoshkevich 367e07e7aeSIlya LeoshkevichOnce on Debian, the build prerequisites can be installed as follows:: 377e07e7aeSIlya Leoshkevich 387e07e7aeSIlya Leoshkevich sudo dpkg --add-architecture s390x 397e07e7aeSIlya Leoshkevich sudo apt-get update 407e07e7aeSIlya Leoshkevich sudo apt-get install \ 417e07e7aeSIlya Leoshkevich bc \ 427e07e7aeSIlya Leoshkevich bison \ 437e07e7aeSIlya Leoshkevich cmake \ 447e07e7aeSIlya Leoshkevich debootstrap \ 457e07e7aeSIlya Leoshkevich dwarves \ 467e07e7aeSIlya Leoshkevich flex \ 477e07e7aeSIlya Leoshkevich g++ \ 487e07e7aeSIlya Leoshkevich gcc \ 497e07e7aeSIlya Leoshkevich g++-s390x-linux-gnu \ 507e07e7aeSIlya Leoshkevich gcc-s390x-linux-gnu \ 517e07e7aeSIlya Leoshkevich gdb-multiarch \ 527e07e7aeSIlya Leoshkevich git \ 537e07e7aeSIlya Leoshkevich make \ 547e07e7aeSIlya Leoshkevich python3 \ 557e07e7aeSIlya Leoshkevich qemu-system-misc \ 567e07e7aeSIlya Leoshkevich qemu-utils \ 577e07e7aeSIlya Leoshkevich rsync \ 587e07e7aeSIlya Leoshkevich libcap-dev:s390x \ 597e07e7aeSIlya Leoshkevich libelf-dev:s390x \ 607e07e7aeSIlya Leoshkevich libncurses-dev 617e07e7aeSIlya Leoshkevich 627e07e7aeSIlya LeoshkevichLatest Clang targeting BPF can be installed as follows:: 637e07e7aeSIlya Leoshkevich 647e07e7aeSIlya Leoshkevich git clone https://github.com/llvm/llvm-project.git 657e07e7aeSIlya Leoshkevich ln -s ../../clang llvm-project/llvm/tools/ 667e07e7aeSIlya Leoshkevich mkdir llvm-project-build 677e07e7aeSIlya Leoshkevich cd llvm-project-build 687e07e7aeSIlya Leoshkevich cmake \ 697e07e7aeSIlya Leoshkevich -DLLVM_TARGETS_TO_BUILD=BPF \ 707e07e7aeSIlya Leoshkevich -DCMAKE_BUILD_TYPE=Release \ 717e07e7aeSIlya Leoshkevich -DCMAKE_INSTALL_PREFIX=/opt/clang-bpf \ 727e07e7aeSIlya Leoshkevich ../llvm-project/llvm 737e07e7aeSIlya Leoshkevich make 747e07e7aeSIlya Leoshkevich sudo make install 757e07e7aeSIlya Leoshkevich export PATH=/opt/clang-bpf/bin:$PATH 767e07e7aeSIlya Leoshkevich 777e07e7aeSIlya LeoshkevichThe disk image can be prepared using a loopback mount and debootstrap:: 787e07e7aeSIlya Leoshkevich 797e07e7aeSIlya Leoshkevich qemu-img create -f raw ./s390.img 1G 807e07e7aeSIlya Leoshkevich sudo losetup -f ./s390.img 817e07e7aeSIlya Leoshkevich sudo mkfs.ext4 /dev/loopX 827e07e7aeSIlya Leoshkevich mkdir ./s390.rootfs 837e07e7aeSIlya Leoshkevich sudo mount /dev/loopX ./s390.rootfs 847e07e7aeSIlya Leoshkevich sudo debootstrap \ 857e07e7aeSIlya Leoshkevich --foreign \ 867e07e7aeSIlya Leoshkevich --arch=s390x \ 877e07e7aeSIlya Leoshkevich --variant=minbase \ 887e07e7aeSIlya Leoshkevich --include=" \ 897e07e7aeSIlya Leoshkevich iproute2, \ 907e07e7aeSIlya Leoshkevich iputils-ping, \ 917e07e7aeSIlya Leoshkevich isc-dhcp-client, \ 927e07e7aeSIlya Leoshkevich kmod, \ 937e07e7aeSIlya Leoshkevich libcap2, \ 947e07e7aeSIlya Leoshkevich libelf1, \ 957e07e7aeSIlya Leoshkevich netcat, \ 967e07e7aeSIlya Leoshkevich procps" \ 977e07e7aeSIlya Leoshkevich testing \ 987e07e7aeSIlya Leoshkevich ./s390.rootfs 997e07e7aeSIlya Leoshkevich sudo umount ./s390.rootfs 1007e07e7aeSIlya Leoshkevich sudo losetup -d /dev/loopX 1017e07e7aeSIlya Leoshkevich 1027e07e7aeSIlya Leoshkevich3. Compilation 1037e07e7aeSIlya Leoshkevich************** 1047e07e7aeSIlya Leoshkevich 1057e07e7aeSIlya LeoshkevichIn addition to the usual Kconfig options required to run the BPF test suite, it 1067e07e7aeSIlya Leoshkevichis also helpful to select:: 1077e07e7aeSIlya Leoshkevich 1087e07e7aeSIlya Leoshkevich CONFIG_NET_9P=y 1097e07e7aeSIlya Leoshkevich CONFIG_9P_FS=y 1107e07e7aeSIlya Leoshkevich CONFIG_NET_9P_VIRTIO=y 1117e07e7aeSIlya Leoshkevich CONFIG_VIRTIO_PCI=y 1127e07e7aeSIlya Leoshkevich 1137e07e7aeSIlya Leoshkevichas that would enable a very easy way to share files with the s390 virtual 1147e07e7aeSIlya Leoshkevichmachine. 1157e07e7aeSIlya Leoshkevich 1167e07e7aeSIlya LeoshkevichCompiling kernel, modules and testsuite, as well as preparing gdb scripts to 1177e07e7aeSIlya Leoshkevichsimplify debugging, can be done using the following commands:: 1187e07e7aeSIlya Leoshkevich 1197e07e7aeSIlya Leoshkevich make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- menuconfig 1207e07e7aeSIlya Leoshkevich make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- bzImage modules scripts_gdb 1217e07e7aeSIlya Leoshkevich make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- \ 1227e07e7aeSIlya Leoshkevich -C tools/testing/selftests \ 1237e07e7aeSIlya Leoshkevich TARGETS=bpf \ 1247e07e7aeSIlya Leoshkevich INSTALL_PATH=$PWD/tools/testing/selftests/kselftest_install \ 1257e07e7aeSIlya Leoshkevich install 1267e07e7aeSIlya Leoshkevich 1277e07e7aeSIlya Leoshkevich4. Running the test suite 1287e07e7aeSIlya Leoshkevich************************* 1297e07e7aeSIlya Leoshkevich 1307e07e7aeSIlya LeoshkevichThe virtual machine can be started as follows:: 1317e07e7aeSIlya Leoshkevich 1327e07e7aeSIlya Leoshkevich qemu-system-s390x \ 1337e07e7aeSIlya Leoshkevich -cpu max,zpci=on \ 1347e07e7aeSIlya Leoshkevich -smp 2 \ 1357e07e7aeSIlya Leoshkevich -m 4G \ 1367e07e7aeSIlya Leoshkevich -kernel linux/arch/s390/boot/compressed/vmlinux \ 1377e07e7aeSIlya Leoshkevich -drive file=./s390.img,if=virtio,format=raw \ 1387e07e7aeSIlya Leoshkevich -nographic \ 1397e07e7aeSIlya Leoshkevich -append 'root=/dev/vda rw console=ttyS1' \ 1407e07e7aeSIlya Leoshkevich -virtfs local,path=./linux,security_model=none,mount_tag=linux \ 1417e07e7aeSIlya Leoshkevich -object rng-random,filename=/dev/urandom,id=rng0 \ 1427e07e7aeSIlya Leoshkevich -device virtio-rng-ccw,rng=rng0 \ 1437e07e7aeSIlya Leoshkevich -netdev user,id=net0 \ 1447e07e7aeSIlya Leoshkevich -device virtio-net-ccw,netdev=net0 1457e07e7aeSIlya Leoshkevich 1467e07e7aeSIlya LeoshkevichWhen using this on a real IBM Z, ``-enable-kvm`` may be added for better 1477e07e7aeSIlya Leoshkevichperformance. When starting the virtual machine for the first time, disk image 1487e07e7aeSIlya Leoshkevichsetup must be finalized using the following command:: 1497e07e7aeSIlya Leoshkevich 1507e07e7aeSIlya Leoshkevich /debootstrap/debootstrap --second-stage 1517e07e7aeSIlya Leoshkevich 1527e07e7aeSIlya LeoshkevichDirectory with the code built on the host as well as ``/proc`` and ``/sys`` 1537e07e7aeSIlya Leoshkevichneed to be mounted as follows:: 1547e07e7aeSIlya Leoshkevich 1557e07e7aeSIlya Leoshkevich mkdir -p /linux 1567e07e7aeSIlya Leoshkevich mount -t 9p linux /linux 1577e07e7aeSIlya Leoshkevich mount -t proc proc /proc 1587e07e7aeSIlya Leoshkevich mount -t sysfs sys /sys 1597e07e7aeSIlya Leoshkevich 1607e07e7aeSIlya LeoshkevichAfter that, the test suite can be run using the following commands:: 1617e07e7aeSIlya Leoshkevich 1627e07e7aeSIlya Leoshkevich cd /linux/tools/testing/selftests/kselftest_install 1637e07e7aeSIlya Leoshkevich ./run_kselftest.sh 1647e07e7aeSIlya Leoshkevich 1657e07e7aeSIlya LeoshkevichAs usual, tests can be also run individually:: 1667e07e7aeSIlya Leoshkevich 1677e07e7aeSIlya Leoshkevich cd /linux/tools/testing/selftests/bpf 1687e07e7aeSIlya Leoshkevich ./test_verifier 1697e07e7aeSIlya Leoshkevich 1707e07e7aeSIlya Leoshkevich5. Debugging 1717e07e7aeSIlya Leoshkevich************ 1727e07e7aeSIlya Leoshkevich 1737e07e7aeSIlya LeoshkevichIt is possible to debug the s390 kernel using QEMU GDB stub, which is activated 1747e07e7aeSIlya Leoshkevichby passing ``-s`` to QEMU. 1757e07e7aeSIlya Leoshkevich 1767e07e7aeSIlya LeoshkevichIt is preferable to turn KASLR off, so that gdb would know where to find the 1777e07e7aeSIlya Leoshkevichkernel image in memory, by building the kernel with:: 1787e07e7aeSIlya Leoshkevich 1797e07e7aeSIlya Leoshkevich RANDOMIZE_BASE=n 1807e07e7aeSIlya Leoshkevich 1817e07e7aeSIlya LeoshkevichGDB can then be attached using the following command:: 1827e07e7aeSIlya Leoshkevich 1837e07e7aeSIlya Leoshkevich gdb-multiarch -ex 'target remote localhost:1234' ./vmlinux 1847e07e7aeSIlya Leoshkevich 1857e07e7aeSIlya Leoshkevich6. Network 1867e07e7aeSIlya Leoshkevich********** 1877e07e7aeSIlya Leoshkevich 1887e07e7aeSIlya LeoshkevichIn case one needs to use the network in the virtual machine in order to e.g. 1897e07e7aeSIlya Leoshkevichinstall additional packages, it can be configured using:: 1907e07e7aeSIlya Leoshkevich 1917e07e7aeSIlya Leoshkevich dhclient eth0 1927e07e7aeSIlya Leoshkevich 1937e07e7aeSIlya Leoshkevich7. Links 1947e07e7aeSIlya Leoshkevich******** 1957e07e7aeSIlya Leoshkevich 1967e07e7aeSIlya LeoshkevichThis document is a compilation of techniques, whose more comprehensive 1977e07e7aeSIlya Leoshkevichdescriptions can be found by following these links: 1987e07e7aeSIlya Leoshkevich 1997e07e7aeSIlya Leoshkevich- `Debootstrap <https://wiki.debian.org/EmDebian/CrossDebootstrap>`_ 2007e07e7aeSIlya Leoshkevich- `Multiarch <https://wiki.debian.org/Multiarch/HOWTO>`_ 2017e07e7aeSIlya Leoshkevich- `Building LLVM <https://llvm.org/docs/CMake.html>`_ 2027e07e7aeSIlya Leoshkevich- `Cross-compiling the kernel <https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Cross-compiling_the_kernel>`_ 2037e07e7aeSIlya Leoshkevich- `QEMU s390x Guest Support <https://wiki.qemu.org/Documentation/Platforms/S390X>`_ 2047e07e7aeSIlya Leoshkevich- `Plan 9 folder sharing over Virtio <https://wiki.qemu.org/Documentation/9psetup>`_ 2057e07e7aeSIlya Leoshkevich- `Using GDB with QEMU <https://wiki.osdev.org/Kernel_Debugging#Use_GDB_with_QEMU>`_ 206