1.. SPDX-License-Identifier: GPL-2.0+
2
3=========================================
4Automated testing of the DRM subsystem
5=========================================
6
7Introduction
8============
9
10Making sure that changes to the core or drivers don't introduce regressions can
11be very time-consuming when lots of different hardware configurations need to
12be tested. Moreover, it isn't practical for each person interested in this
13testing to have to acquire and maintain what can be a considerable amount of
14hardware.
15
16Also, it is desirable for developers to check for regressions in their code by
17themselves, instead of relying on the maintainers to find them and then
18reporting back.
19
20There are facilities in gitlab.freedesktop.org to automatically test Mesa that
21can be used as well for testing the DRM subsystem. This document explains how
22people interested in testing it can use this shared infrastructure to save
23quite some time and effort.
24
25
26Relevant files
27==============
28
29drivers/gpu/drm/ci/gitlab-ci.yml
30--------------------------------
31
32This is the root configuration file for GitLab CI. Among other less interesting
33bits, it specifies the specific version of the scripts to be used. There are
34some variables that can be modified to change the behavior of the pipeline:
35
36DRM_CI_PROJECT_PATH
37    Repository that contains the Mesa software infrastructure for CI
38
39DRM_CI_COMMIT_SHA
40    A particular revision to use from that repository
41
42UPSTREAM_REPO
43    URL to git repository containing the target branch
44
45TARGET_BRANCH
46    Branch to which this branch is to be merged into
47
48IGT_VERSION
49    Revision of igt-gpu-tools being used, from
50    https://gitlab.freedesktop.org/drm/igt-gpu-tools
51
52drivers/gpu/drm/ci/testlist.txt
53-------------------------------
54
55IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
56file, see below).
57
58drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
59----------------------------------------------------------
60
61Lists the known failures for a given driver on a specific hardware revision.
62
63drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
64-----------------------------------------------------------
65
66Lists the tests that for a given driver on a specific hardware revision are
67known to behave unreliably. These tests won't cause a job to fail regardless of
68the result. They will still be run.
69
70drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
71-----------------------------------------------------------
72
73Lists the tests that won't be run for a given driver on a specific hardware
74revision. These are usually tests that interfere with the running of the test
75list due to hanging the machine, causing OOM, taking too long, etc.
76
77
78How to enable automated testing on your tree
79============================================
80
811. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
82yet
83
842. In your kernel repo's configuration (eg.
85https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
86CI/CD configuration file from .gitlab-ci.yml to
87drivers/gpu/drm/ci/gitlab-ci.yml.
88
893. Next time you push to this repository, you will see a CI pipeline being
90created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
91
924. The various jobs will be run and when the pipeline is finished, all jobs
93should be green unless a regression has been found.
94
95
96How to update test expectations
97===============================
98
99If your changes to the code fix any tests, you will have to remove one or more
100lines from one or more of the files in
101drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
102affected by the change.
103
104
105How to expand coverage
106======================
107
108If your code changes make it possible to run more tests (by solving reliability
109issues, for example), you can remove tests from the flakes and/or skips lists,
110and then the expected results if there are any known failures.
111
112If there is a need for updating the version of IGT being used (maybe you have
113added more tests to it), update the IGT_VERSION variable at the top of the
114gitlab-ci.yml file.
115
116
117How to test your changes to the scripts
118=======================================
119
120For testing changes to the scripts in the drm-ci repo, change the
121DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
122drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
123janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
124
125
126How to incorporate external fixes in your testing
127=================================================
128
129Often, regressions in other trees will prevent testing changes local to the
130tree under test. These fixes will be automatically merged in during the build
131jobs from a branch in the target tree that is named as
132${TARGET_BRANCH}-external-fixes.
133
134If the pipeline is not in a merge request and a branch with the same name
135exists in the local tree, commits from that branch will be merged in as well.
136
137
138How to deal with automated testing labs that may be down
139========================================================
140
141If a hardware farm is down and thus causing pipelines to fail that would
142otherwise pass, one can disable all jobs that would be submitted to that farm
143by editing the file at
144https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
145