1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK 2 3*********************************** 4Project Testing and Release Process 5*********************************** 6 7Day to Day Development 8====================== 9 10This section details how the project tests changes, through automation 11on the Autobuilder or with the assistance of QA teams, through to making 12releases. 13 14The project aims to test changes against our test matrix before those 15changes are merged into the master branch. As such, changes are queued 16up in batches either in the ``master-next`` branch in the main trees, or 17in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton 18helps review and test patches and this is his testing tree). 19 20We have two broad categories of test builds, including "full" and 21"quick". On the Autobuilder, these can be seen as "a-quick" and 22"a-full", simply for ease of sorting in the UI. Use our Autobuilder 23:yocto_ab:`console view </typhoon/#/console>` to see where we manage most 24test-related items. 25 26Builds are triggered manually when the test branches are ready. The 27builds are monitored by the SWAT team. For additional information, see 28:yocto_wiki:`/Yocto_Build_Failure_Swat_Team`. 29If successful, the changes would usually be merged to the ``master`` 30branch. If not successful, someone would respond to the changes on the 31mailing list explaining that there was a failure in testing. The choice 32of quick or full would depend on the type of changes and the speed with 33which the result was required. 34 35The Autobuilder does build the ``master`` branch once daily for several 36reasons, in particular, to ensure the current ``master`` branch does 37build, but also to keep (:yocto_git:`yocto-testresults </yocto-testresults/>`), 38(:yocto_git:`buildhistory </poky-buildhistory/>`), and 39our sstate up to date. On the weekend, there is a ``master-next`` build 40instead to ensure the test results are updated for the less frequently 41run targets. 42 43Performance builds (``buildperf-\*`` targets in the console) are triggered 44separately every six hours and automatically push their results to the 45:yocto_git:`buildstats </yocto-buildstats/>` repository. 46 47The "quick" targets have been selected to be the ones which catch the 48most failures or give the most valuable data. We run "fast" ptests in 49this case for example but not the ones which take a long time. The quick 50target doesn't include ``\*-lsb`` builds for all architectures, some ``world`` 51builds and doesn't trigger performance tests or ``ltp`` testing. The full 52build includes all these things and is slower but more comprehensive. 53 54Release Builds 55============== 56 57The project typically has two major releases a year with a six month 58cadence in April and October. Between these there would be a number of 59milestone releases (usually four) with the final one being stabilization 60only along with point releases of our stable branches. 61 62The build and release process for these project releases is similar to 63that in :ref:`test-manual/test-process:day to day development`, in that the 64a-full target of the Autobuilder is used but in addition the form is 65configured to generate and publish artifacts and the milestone number, 66version, release candidate number and other information is entered. The 67box to "generate an email to QA" is also checked. 68 69When the build completes, an email is sent out using the ``send-qa-email`` 70script in the :yocto_git:`yocto-autobuilder-helper </yocto-autobuilder-helper>` 71repository to the list of people configured for that release. Release builds 72are placed into a directory in https://autobuilder.yocto.io/pub/releases on the 73Autobuilder which is included in the email. The process from here is 74more manual and control is effectively passed to release engineering. 75The next steps include: 76 77- QA teams respond to the email saying which tests they plan to run and 78 when the results will be available. 79 80- QA teams run their tests and share their results in the 81 :yocto_git:`yocto-testresults-contrib </yocto-testresults-contrib>` 82 repository, along with a summary of their findings. 83 84- Release engineering prepare the release as per their process. 85 86- Test results from the QA teams are included into the release in 87 separate directories and also uploaded to the 88 :yocto_git:`yocto-testresults </yocto-testresults>` 89 repository alongside the other test results for the given revision. 90 91- The QA report in the final release is regenerated using resulttool to 92 include the new test results and the test summaries from the teams 93 (as headers to the generated report). 94 95- The release is checked against the release checklist and release 96 readiness criteria. 97 98- A final decision on whether to release is made by the YP TSC who have 99 final oversight on release readiness. 100