1ac13d5f3SPatrick Williams# Patchtest 2ac13d5f3SPatrick Williams 3ac13d5f3SPatrick Williams## Introduction 4ac13d5f3SPatrick Williams 5ac13d5f3SPatrick WilliamsPatchtest is a test framework for community patches based on the standard 6ac13d5f3SPatrick Williamsunittest python module. As input, it needs tree elements to work properly: 7ac13d5f3SPatrick Williamsa patch in mbox format (either created with `git format-patch` or fetched 8ac13d5f3SPatrick Williamsfrom 'patchwork'), a test suite and a target repository. 9ac13d5f3SPatrick Williams 10ac13d5f3SPatrick WilliamsThe first test suite intended to be used with patchtest is found in the 11ac13d5f3SPatrick Williamsopenembedded-core repository [1] targeted for patches that get into the 12ac13d5f3SPatrick Williamsopenembedded-core mailing list [2]. This suite is also intended as a 13ac13d5f3SPatrick Williamsbaseline for development of similar suites for other layers as needed. 14ac13d5f3SPatrick Williams 15ac13d5f3SPatrick WilliamsPatchtest can either run on a host or a guest machine, depending on which 16ac13d5f3SPatrick Williamsenvironment the execution needs to be done. If you plan to test your own patches 17ac13d5f3SPatrick Williams(a good practice before these are sent to the mailing list), the easiest way is 18ac13d5f3SPatrick Williamsto install and execute on your local host; in the other hand, if automatic 19ac13d5f3SPatrick Williamstesting is intended, the guest method is strongly recommended. The guest 20ac13d5f3SPatrick Williamsmethod requires the use of the patchtest layer, in addition to the tools 21ac13d5f3SPatrick Williamsavailable in oe-core: https://git.yoctoproject.org/patchtest/ 22ac13d5f3SPatrick Williams 23ac13d5f3SPatrick Williams## Installation 24ac13d5f3SPatrick Williams 25ac13d5f3SPatrick WilliamsAs a tool for use with the Yocto Project, the [quick start guide](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html) 26ac13d5f3SPatrick Williamscontains the necessary prerequisites for a basic project. In addition, 27ac13d5f3SPatrick Williamspatchtest relies on the following Python modules: 28ac13d5f3SPatrick Williams 29ac13d5f3SPatrick Williams- boto3 (for sending automated results emails only) 30ac13d5f3SPatrick Williams- git-pw>=2.5.0 31ac13d5f3SPatrick Williams- jinja2 32ac13d5f3SPatrick Williams- pylint 33ac13d5f3SPatrick Williams- pyparsing>=3.0.9 34ac13d5f3SPatrick Williams- unidiff 35ac13d5f3SPatrick Williams 36ac13d5f3SPatrick WilliamsThese can be installed by running `pip install -r 37ac13d5f3SPatrick Williamsmeta/lib/patchtest/requirements.txt`. Note that git-pw is not 38ac13d5f3SPatrick Williamsautomatically added to the user's PATH; by default, it is installed at 39ac13d5f3SPatrick Williams~/.local/bin/git-pw. 40ac13d5f3SPatrick Williams 41ac13d5f3SPatrick WilliamsFor git-pw (and therefore scripts such as patchtest-get--series) to work, you need 42ac13d5f3SPatrick Williamsto provide a Patchwork instance in your user's .gitconfig, like so (the project 43ac13d5f3SPatrick Williamscan be specified using the --project argument): 44ac13d5f3SPatrick Williams 45ac13d5f3SPatrick Williams git config --global pw.server "https://patchwork.yoctoproject.org/api/1.2/" 46ac13d5f3SPatrick Williams 47ac13d5f3SPatrick WilliamsTo work with patchtest, you should have the following repositories cloned: 48ac13d5f3SPatrick Williams 49ac13d5f3SPatrick Williams1. https://git.openembedded.org/openembedded-core/ (or https://git.yoctoproject.org/poky/) 50ac13d5f3SPatrick Williams2. https://git.openembedded.org/bitbake/ (if not using poky) 51ac13d5f3SPatrick Williams3. https://git.yoctoproject.org/patchtest (if using guest mode) 52ac13d5f3SPatrick Williams 53ac13d5f3SPatrick Williams## Usage 54ac13d5f3SPatrick Williams 55ac13d5f3SPatrick Williams### Obtaining Patches 56ac13d5f3SPatrick Williams 57ac13d5f3SPatrick WilliamsPatch files can be obtained directly from cloned repositories using `git 58ac13d5f3SPatrick Williamsformat-patch -N` (where N is the number of patches starting from HEAD to 59ac13d5f3SPatrick Williamsgenerate). git-pw can also be used with filters for users, patch/series IDs, 60ac13d5f3SPatrick Williamsand timeboxes if specific patches are desired. For more information, see the 61ac13d5f3SPatrick Williamsgit-pw [documentation](https://patchwork.readthedocs.io/projects/git-pw/en/latest/). 62ac13d5f3SPatrick Williams 63ac13d5f3SPatrick WilliamsAlternatively, `scripts/patchtest-get-series` can be used to pull mbox files from 64ac13d5f3SPatrick Williamsthe Patchwork instance configured previously in .gitconfig. It uses a log file 65ac13d5f3SPatrick Williamscalled ".series_test.log" to store and compare series IDs so that the same 66ac13d5f3SPatrick Williamsversions of a patch are not tested multiple times unintentionally. By default, 67ac13d5f3SPatrick Williamsit will pull up to five patch series from the last 30 minutes using oe-core as 68ac13d5f3SPatrick Williamsthe target project, but these parameters can be configured using the `--limit`, 69ac13d5f3SPatrick Williams`--interval`, and `--project` arguments respectively. For more information, run 70ac13d5f3SPatrick Williams`patchtest-get-series -h`. 71ac13d5f3SPatrick Williams 72ac13d5f3SPatrick Williams### Host Mode 73ac13d5f3SPatrick Williams 74ac13d5f3SPatrick WilliamsTo run patchtest on the host, do the following: 75ac13d5f3SPatrick Williams 76ac13d5f3SPatrick Williams1. In openembedded-core/poky, do `source oe-init-build-env` 77ac13d5f3SPatrick Williams2. Generate patch files from the target repository by doing `git-format patch -N`, 78ac13d5f3SPatrick Williams where N is the number of patches starting at HEAD, or by using git-pw 79ac13d5f3SPatrick Williams or patchtest-get-series 80ac13d5f3SPatrick Williams3. Run patchtest on a patch file by doing the following: 81ac13d5f3SPatrick Williams 82ac13d5f3SPatrick Williams patchtest --patch /path/to/patch/file 83ac13d5f3SPatrick Williams 84ac13d5f3SPatrick Williams or, if you have stored the patch files in a directory, do: 85ac13d5f3SPatrick Williams 86ac13d5f3SPatrick Williams patchtest --directory /path/to/patch/directory 87ac13d5f3SPatrick Williams 88ac13d5f3SPatrick Williams For example, to test `master-gcc-Fix--fstack-protector-issue-on-aarch64.patch` against the oe-core test suite: 89ac13d5f3SPatrick Williams 90ac13d5f3SPatrick Williams patchtest --patch master-gcc-Fix--fstack-protector-issue-on-aarch64.patch 91ac13d5f3SPatrick Williams 92ac13d5f3SPatrick Williams If you want to use a different test suite or target repository, you can use the --testdir and --repodir flags: 93ac13d5f3SPatrick Williams 94ac13d5f3SPatrick Williams patchtest --patch /path/to/patch/file --repodir /path/to/repo --testdir /path/to/test/dir 95ac13d5f3SPatrick Williams 96ac13d5f3SPatrick Williams### Guest Mode 97ac13d5f3SPatrick Williams 98ac13d5f3SPatrick WilliamsPatchtest's guest mode has been refactored to more closely mirror the 99ac13d5f3SPatrick Williamstypical Yocto Project image build workflow, but there are still some key 100ac13d5f3SPatrick Williamsdifferences to keep in mind. The primary objective is to provide a level 101ac13d5f3SPatrick Williamsof isolation from the host when testing patches pulled automatically 102ac13d5f3SPatrick Williamsfrom the mailing lists. When executed this way, the test process is 103ac13d5f3SPatrick Williamsessentially running random code from the internet and could be 104ac13d5f3SPatrick Williamscatastrophic if malicious bits or even poorly-handled edge cases aren't 105ac13d5f3SPatrick Williamsprotected against. In order to use this mode, the 106ac13d5f3SPatrick Williamshttps://git.yoctoproject.org/patchtest/ repository must be cloned and 107ac13d5f3SPatrick Williamsthe meta-patchtest layer added to bblayers.conf. 108ac13d5f3SPatrick Williams 109ac13d5f3SPatrick WilliamsThe general flow of guest mode is: 110ac13d5f3SPatrick Williams 111ac13d5f3SPatrick Williams1. Run patchtest-setup-sharedir --directory <dirname> to create a 112ac13d5f3SPatrick Williams directory for mounting 113ac13d5f3SPatrick Williams2. Collect patches via patchtest-get-series (or other manual step) into the 114ac13d5f3SPatrick Williams <dirname>/mboxes path 115ac13d5f3SPatrick Williams3. Ensure that a user with ID 1200 has appropriate read/write 116ac13d5f3SPatrick Williams permissions to <dirname> and <dirname>/mboxes, so that the 117ac13d5f3SPatrick Williams "patchtest" user in the core-image-patchtest image can function 118ac13d5f3SPatrick Williams4. Build the core-image-patchtest image 119ac13d5f3SPatrick Williams5. Run the core-image-patchtest image with the mounted sharedir, like 120ac13d5f3SPatrick Williams so: 121ac13d5f3SPatrick Williams `runqemu kvm nographic qemuparams="-snapshot -fsdev 122ac13d5f3SPatrick Williams local,id=test_mount,path=/workspace/yocto/poky/build/patchtestdir,security_model=mapped 123ac13d5f3SPatrick Williams -device virtio-9p-pci,fsdev=test_mount,mount_tag=test_mount -smp 4 -m 124ac13d5f3SPatrick Williams 2048"` 125ac13d5f3SPatrick Williams 126ac13d5f3SPatrick WilliamsPatchtest runs as an initscript for the core-image-patchtest image and 127ac13d5f3SPatrick Williamsshuts down after completion, so there is no input required from a user 128ac13d5f3SPatrick Williamsduring operation. Unlike in host mode, the guest is designed to 129ac13d5f3SPatrick Williamsautomatically generate test result files, in the same directory as the 130ac13d5f3SPatrick Williamstargeted patch files but with .testresult as an extension. These contain 131ac13d5f3SPatrick Williamsthe entire output of the patchtest run for each respective pass, 132ac13d5f3SPatrick Williamsincluding the PASS, FAIL, and SKIP indicators for each test run. 133ac13d5f3SPatrick Williams 134ac13d5f3SPatrick Williams## Contributing 135ac13d5f3SPatrick Williams 136*73bd93f1SPatrick WilliamsThe yocto mailing list (openembedded-core@lists.openembedded.org) is used for questions, 137ac13d5f3SPatrick Williamscomments and patch review. It is subscriber only, so please register before 138ac13d5f3SPatrick Williamsposting. 139ac13d5f3SPatrick Williams 140ac13d5f3SPatrick WilliamsWhen sending single patches, please use something like: 141ac13d5f3SPatrick Williams 142*73bd93f1SPatrick Williams git send-email -M -1 --to=openembedded-core@lists.openembedded.org --subject-prefix=OE-core][PATCH 143ac13d5f3SPatrick Williams 144ac13d5f3SPatrick Williams## Maintenance 145ac13d5f3SPatrick Williams----------- 146ac13d5f3SPatrick Williams 147ac13d5f3SPatrick WilliamsMaintainers: 148ac13d5f3SPatrick Williams Trevor Gamblin <tgamblin@baylibre.com> 149ac13d5f3SPatrick Williams 150ac13d5f3SPatrick Williams## Links 151ac13d5f3SPatrick Williams----- 152ac13d5f3SPatrick Williams[1] https://git.openembedded.org/openembedded-core/ 153ac13d5f3SPatrick Williams[2] https://www.yoctoproject.org/community/mailing-lists/ 154