1# Patchtest 2 3## Introduction 4 5Patchtest is a test framework for community patches based on the standard 6unittest python module. As input, it needs three elements to work properly: 7 8- a patch in mbox format (either created with `git format-patch` or fetched 9from 'patchwork') 10- a test suite 11- a target repository 12 13The first test suite intended to be used with patchtest is found in the 14openembedded-core repository [1], targeted for patches that get into the 15openembedded-core mailing list [2]. This suite is also intended as a 16baseline for development of similar suites for other layers as needed. 17 18Patchtest can either run on a host or a guest machine, depending on 19which environment you prefer. If you plan to test your own patches (a 20good practice before these are sent to the mailing list), the easiest 21way is to install and execute on your local host; in the other hand, if 22automatic testing is intended, the guest method is strongly recommended. 23The guest method requires the use of the patchtest layer, in addition to 24the tools available in oe-core: https://git.yoctoproject.org/patchtest/ 25 26## Installation 27 28As a tool for use with the Yocto Project, the [quick start 29guide](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html) 30contains the necessary prerequisites. In addition, patchtest relies on 31several Python modules for parsing and analysis, which can be installed 32by running `pip install -r meta/lib/patchtest/requirements.txt`. Note 33that git-pw is not automatically added to the user's PATH; by default, 34it is installed at ~/.local/bin/git-pw. 35 36For git-pw (and therefore scripts such as patchtest-get--series) to work, you need 37to provide a Patchwork instance in your user's .gitconfig, like so (the project 38can be specified using the --project argument): 39 40 git config --global pw.server "https://patchwork.yoctoproject.org/api/1.2/" 41 42To work with patchtest, you should have the following repositories cloned: 43 441. https://git.openembedded.org/openembedded-core/ (or https://git.yoctoproject.org/poky/) 452. https://git.openembedded.org/bitbake/ (if not using poky) 463. https://git.yoctoproject.org/patchtest (if using guest mode) 47 48## Usage 49 50### Obtaining Patches 51 52Patch files can be obtained directly from cloned repositories using `git 53format-patch -N` (where N is the number of patches starting from HEAD to 54generate). git-pw can also be used with filters for users, patch/series IDs, 55and timeboxes if specific patches are desired. For more information, see the 56git-pw [documentation](https://patchwork.readthedocs.io/projects/git-pw/en/latest/). 57 58Alternatively, `scripts/patchtest-get-series` can be used to pull mbox files from 59the Patchwork instance configured previously in .gitconfig. It uses a log file 60called ".series_test.log" to store and compare series IDs so that the same 61versions of a patch are not tested multiple times unintentionally. By default, 62it will pull up to five patch series from the last 30 minutes using oe-core as 63the target project, but these parameters can be configured using the `--limit`, 64`--interval`, and `--project` arguments respectively. For more information, run 65`patchtest-get-series -h`. 66 67### Host Mode 68 69To run patchtest on the host, do the following: 70 711. In openembedded-core/poky, do `source oe-init-build-env` 722. Generate patch files from the target repository by doing `git-format patch -N`, 73 where N is the number of patches starting at HEAD, or by using git-pw 74 or patchtest-get-series 753. Run patchtest on a patch file by doing the following: 76 77 patchtest --patch /path/to/patch/file 78 79 or, if you have stored the patch files in a directory, do: 80 81 patchtest --directory /path/to/patch/directory 82 83 For example, to test `master-gcc-Fix--fstack-protector-issue-on-aarch64.patch` against the oe-core test suite: 84 85 patchtest --patch master-gcc-Fix--fstack-protector-issue-on-aarch64.patch 86 87 If you want to use a different test suite or target repository, you can use the --testdir and --repodir flags: 88 89 patchtest --patch /path/to/patch/file --repodir /path/to/repo --testdir /path/to/test/dir 90 91### Guest Mode 92 93Patchtest's guest mode has been refactored to more closely mirror the 94typical Yocto Project image build workflow, but there are still some key 95differences to keep in mind. The primary objective is to provide a level 96of isolation from the host when testing patches pulled automatically 97from the mailing lists. When executed this way, the test process is 98essentially running random code from the internet and could be 99catastrophic if malicious bits or even poorly-handled edge cases aren't 100protected against. In order to use this mode, the 101https://git.yoctoproject.org/patchtest/ repository must be cloned and 102the meta-patchtest layer added to bblayers.conf. 103 104The general flow of guest mode is: 105 1061. Run patchtest-setup-sharedir --directory <dirname> to create a 107 directory for mounting 1082. Collect patches via patchtest-get-series (or other manual step) into the 109 <dirname>/mboxes path 1103. Ensure that a user with ID 1200 has appropriate read/write 111 permissions to <dirname> and <dirname>/mboxes, so that the 112 "patchtest" user in the core-image-patchtest image can function 1134. Build the core-image-patchtest image 1145. Run the core-image-patchtest image with the mounted sharedir, like 115 so: 116 `runqemu kvm nographic qemuparams="-snapshot -fsdev 117 local,id=test_mount,path=/workspace/yocto/poky/build/patchtestdir,security_model=mapped 118 -device virtio-9p-pci,fsdev=test_mount,mount_tag=test_mount -smp 4 -m 119 2048"` 120 121Patchtest is run by an initscript for the core-image-patchtest image and 122shuts down after completion, so there is no input required from a user 123during operation. Unlike in host mode, the guest is designed to 124automatically generate test result files, in the same directory as the 125targeted patch files but with .testresult as an extension. These contain 126the entire output of the patchtest run for each respective pass, 127including the PASS, FAIL, and SKIP indicators for each test run. 128 129### Running Patchtest Selftests 130 131Patchtest also includes selftests, which are currently in the form of 132several contrived patch files and a runner script found in 133`meta/lib/patchtest/selftest/`. In order to run these, the 134`meta-selftest` layer must be added to bblayers.conf. It is also 135recommended to set BB_SERVER_TIMEOUT (and thus enable memory-resident 136bitbake) in local.conf to reduce runtime, as the bitbake startup process 137will otherwise add to it significantly when restarted for each test 138patch. 139 140## Contributing 141 142The yocto mailing list (openembedded-core@lists.openembedded.org) is used for questions, 143comments and patch review. It is subscriber only, so please register before 144posting. 145 146When sending single patches, please use something like: 147 148 git send-email -M -1 --to=openembedded-core@lists.openembedded.org --subject-prefix=OE-core][PATCH 149 150## Maintenance 151----------- 152 153Maintainers: 154 Trevor Gamblin <tgamblin@baylibre.com> 155 156## Links 157----- 158[1] https://git.openembedded.org/openembedded-core/ 159[2] https://www.yoctoproject.org/community/mailing-lists/ 160