Date:   Tue, 17 May 2022 11:24:17 +0200
From:   Tomeu Vizoso <tomeu.vizoso@...labora.com>
To:     Neil Armstrong <narmstrong@...libre.com>,
        David Airlie <airlied@...ux.ie>,
        Jonathan Corbet <corbet@....net>,
        Carlo Caione <carlo@...one.org>,
        Kevin Hilman <khilman@...libre.com>,
        Heiko Stuebner <heiko@...ech.de>,
        Matthias Brugger <matthias.bgg@...il.com>
Cc:     linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
        dri-devel@...ts.freedesktop.org,
        linux-rockchip@...ts.infradead.org,
        linux-mediatek@...ts.infradead.org,
        linux-amlogic@...ts.infradead.org,
        linux-arm-kernel@...ts.infradead.org
Subject: Re: [RFC v4] drm: Add initial ci/ subdirectory

On 5/17/22 11:18 AM, Neil Armstrong wrote:
> On 17/05/2022 10:16, Tomeu Vizoso wrote:
>> And use it to store expectations about what the DRM drivers are
>> supposed to pass in the IGT test suite.
>>
>> Also include a configuration file that points to the out-of-tree CI
>> scripts.
>>
>> By storing the test expectations along the code we can make sure both
>> stay in sync with each other, and so we can know when a code change
>> breaks those expectations.
>>
>> This will allow all contributors to drm to reuse the infrastructure
>> already in gitlab.freedesktop.org to test the driver on several
>> generations of the hardware.
>>
>> v2:
>>    - Fix names of result expectation files to match SoC
>>    - Don't execute tests that are going to skip on all boards
>>
>> v3:
>>    - Remove tracking of dmesg output during test execution
>>
>> v4:
>>    - Move up to drivers/gpu/drm
>>    - Add support for a bunch of other drivers
>>    - Explain how to incorporate fixes for CI from a
>>      ${TARGET_BRANCH}-external-fixes branch
>>    - Remove tests that pass from expected results file, to reduce the
>>      size of in-tree files
>>    - Add docs about how to deal with outages in automated testing labs
>>    - Specify the exact SHA of the CI scripts to be used
>>
>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@...labora.com>
>> ---
>>   Documentation/gpu/automated_testing.rst       |  84 +++++++
>>   drivers/gpu/drm/ci/amdgpu.testlist            | 130 +++++++++++
>>   drivers/gpu/drm/ci/amdgpu_stoney_results.txt  |  17 ++
>>   drivers/gpu/drm/ci/gitlab-ci.yml              |  13 ++
>>   drivers/gpu/drm/ci/i915.testlist              | 219 ++++++++++++++++++
>>   drivers/gpu/drm/ci/i915_amly_results.txt      |   9 +
>>   drivers/gpu/drm/ci/i915_apl_results.txt       |   9 +
>>   drivers/gpu/drm/ci/i915_cml_results.txt       |  10 +
>>   drivers/gpu/drm/ci/i915_glk_results.txt       |  13 ++
>>   drivers/gpu/drm/ci/i915_kbl_results.txt       |   9 +
>>   drivers/gpu/drm/ci/i915_tgl_results.txt       |  12 +
>>   drivers/gpu/drm/ci/i915_whl_results.txt       |   9 +
>>   drivers/gpu/drm/ci/mediatek.testlist          | 102 ++++++++
>>   .../gpu/drm/ci/mediatek_mt8173_results.txt    |  11 +
>>   .../gpu/drm/ci/mediatek_mt8183_results.txt    |  11 +
>>   .../gpu/drm/ci/mediatek_mt8192_results.txt    |  11 +
>>   drivers/gpu/drm/ci/meson.testlist             |  71 ++++++
>>   drivers/gpu/drm/ci/meson_g12b_results.txt     |  11 +
>>   drivers/gpu/drm/ci/msm.testlist               | 144 ++++++++++++
>>   drivers/gpu/drm/ci/msm_apq8016_results.txt    |  77 ++++++
>>   drivers/gpu/drm/ci/msm_apq8096_results.txt    |  76 ++++++
>>   drivers/gpu/drm/ci/msm_sc7180_results.txt     |  18 ++
>>   drivers/gpu/drm/ci/msm_sdm845_results.txt     |  33 +++
>>   drivers/gpu/drm/ci/rockchip.testlist          | 188 +++++++++++++++
>>   .../gpu/drm/ci/rockchip_rk3288_results.txt    |  11 +
>>   .../gpu/drm/ci/rockchip_rk3399_results.txt    |  14 ++
>>   drivers/gpu/drm/ci/virtio_gpu.testlist        |  92 ++++++++
>>   drivers/gpu/drm/ci/virtio_gpu_results.txt     |  28 +++
>>   28 files changed, 1432 insertions(+)
>>   create mode 100644 Documentation/gpu/automated_testing.rst
>>   create mode 100644 drivers/gpu/drm/ci/amdgpu.testlist
>>   create mode 100644 drivers/gpu/drm/ci/amdgpu_stoney_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
>>   create mode 100644 drivers/gpu/drm/ci/i915.testlist
>>   create mode 100644 drivers/gpu/drm/ci/i915_amly_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_apl_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_cml_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_glk_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_kbl_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_tgl_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/i915_whl_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/mediatek.testlist
>>   create mode 100644 drivers/gpu/drm/ci/mediatek_mt8173_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/mediatek_mt8183_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/mediatek_mt8192_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/meson.testlist
>>   create mode 100644 drivers/gpu/drm/ci/meson_g12b_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/msm.testlist
>>   create mode 100644 drivers/gpu/drm/ci/msm_apq8016_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/msm_apq8096_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/msm_sc7180_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/msm_sdm845_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/rockchip.testlist
>>   create mode 100644 drivers/gpu/drm/ci/rockchip_rk3288_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/rockchip_rk3399_results.txt
>>   create mode 100644 drivers/gpu/drm/ci/virtio_gpu.testlist
>>   create mode 100644 drivers/gpu/drm/ci/virtio_gpu_results.txt
>>
>> diff --git a/Documentation/gpu/automated_testing.rst 
>> b/Documentation/gpu/automated_testing.rst
>> new file mode 100644
>> index 000000000000..61003ecc7b6e
>> --- /dev/null
>> +++ b/Documentation/gpu/automated_testing.rst
>> @@ -0,0 +1,84 @@
>> +.. SPDX-License-Identifier: GPL-2.0+
>> +
>> +=========================================
>> +Automated testing of the DRM subsystem
>> +=========================================
>> +
>> +
>> +Introduction
>> +============
>> +
>> +Making sure that changes to the core or drivers don't introduce 
>> regressions can be very time consuming when lots of different hardware 
>> configurations need to be tested. Moreover, it isn't practical for 
>> each person interested in this testing to have to acquire and maintain 
>> what can be a considerable amount of hardware.
>> +
>> +Also, it is desirable for developers to check for regressions in 
>> their code by themselves, instead of relying on the maintainers 
>> finding them and then reporting back.
>> +
>> +There are facilities in gitlab.freedesktop.org to automatically test 
>> Mesa that can be used as well for testing the DRM subsystem. This 
>> document explains how people interested in testing it can use this 
>> shared infrastructure to save quite some time and effort.
>> +
>> +
>> +Relevant files
>> +==============
>> +
>> +drivers/gpu/drm/ci/gitlab-ci.yml
>> +--------------------------------
>> +
>> +Specifies the specific version of the scripts to be used. GitLab CI 
>> will use the values defined in this file to fetch the right scripts.
>> +
>> +
>> +drivers/gpu/drm/ci/${DRIVER_NAME}.testlist
>> +------------------------------------------
>> +
>> +Specifies the tests that the current code is expected to be able to 
>> reliably run for ${DRIVER_NAME}. These tests are expected to not hang 
>> the DUT (Device Under Testing) when running on the revision they 
>> belong to, and to give consistent results.
>> +
>> +
>> +drivers/gpu/drm/ci/${DRIVER_NAME}_*_results.txt
>> +----------------------------------------
>> +
>> +Specifies the expected results of running this specific kernel 
>> revision on a given hardware configuration.
>> +
>> +
>> +How to enable automated testing on your tree
>> +============================================
>> +
>> +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you 
>> don't have one yet
>> +
>> +2. In your kernel repo's configuration (eg. 
>> https://gitlab.freedesktop.org/tomeu/linux/-/settings/ci_cd), change 
>> the CI/CD configuration file from .gitlab-ci.yml to 
>> drivers/gpu/drm/ci/gitlab-ci.yml.
>> +
>> +3. Next time you push to this repository, you will see a CI pipeline 
>> being created (eg. 
>> https://gitlab.freedesktop.org/tomeu/linux/-/pipelines)
>> +
>> +4. The various jobs will be run and when the pipeline is finished, 
>> all jobs should be green unless a regression has been found.
>> +
>> +
>> +How to update test expectations
>> +===============================
>> +
>> +If your changes to the code fix any tests, you will have to update 
>> one or more of the files in 
>> drivers/gpu/drm/ci/${DRIVER_NAME}_*_results.txt, for each of the test 
>> platforms affected by the change.
>> +
>> +If you have run a pipeline and it failed because of a mismatched test 
>> result, you will have been offered a patch at the end of the run that 
>> you can apply after reviewing it.
>> +
>> +
>> +How to expand coverage
>> +======================
>> +
>> +If your code changes makes it possible to run more tests (by solving 
>> reliability issues, for example), more tests can be added to this 
>> list, and then the expected results updated with the patch file that 
>> will be printed at the end of each job.
>> +
>> +If there is a need for updating the version of IGT being used (maybe 
>> you have added more tests to it), follow the instructions at 
>> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/tree/main#upreving-igt.
>> +
>> +
>> +How to test your changes to the scripts
>> +==========================================
>> +
>> +For testing changes to the scripts in the drm-ci repo, change the 
>> DRM_CI_PROJECT_PATH variable in drivers/gpu/drm/ci/gitlab-ci.yml to 
>> match your fork of the project (eg. tomeu/drm-ci). This fork needs to 
>> be in https://gitlab.freedesktop.org/.
>> +
>> +
>> +How to incorporate external fixes in your testing
>> +=================================================
>> +
>> +Often, regressions in other trees will prevent testing changes local 
>> to the tree under test. These fixes will be automatically merged in 
>> during the build jobs from a branch in the target tree that is named 
>> as ${TARGET_BRANCH}-external-fixes.
>> +
>> +If the pipeline is not in a merge request and a branch with the same 
>> name exists in the local tree, commits from that branch will be merged 
>> in as well.
>> +
>> +
>> +How to deal with automated testing labs that may be down
>> +========================================================
>> +
>> +If a hardware farm is down and thus causing pipelines to fail that 
>> would otherwise pass, one can disable all jobs that would be submitted 
>> to that farm by editing the file at 
>> https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml. 
>>
> 
> 
> [..]
> 
>> diff --git a/drivers/gpu/drm/ci/meson.testlist 
>> b/drivers/gpu/drm/ci/meson.testlist
>> new file mode 100644
>> index 000000000000..23744b3a2fb0
>> --- /dev/null
>> +++ b/drivers/gpu/drm/ci/meson.testlist
>> @@ -0,0 +1,71 @@
>> +# Keep alphabetically sorted by default
>> +
>> +igt@...e_auth@...client-simple
>> +igt@...e_auth@...client-master-drop
>> +igt@...e_auth@...ic-auth
>> +igt@...e_auth@...y-magics
>> +igt@...e_getclient
>> +igt@...e_getstats
>> +igt@...e_getversion
>> +igt@...e_setmaster_vs_auth
>> +igt@..._read@...alid-buffer
>> +igt@..._addfb_basic@...sed-handle
>> +igt@..._addfb_basic@...sed-pitches
>> +igt@..._addfb_basic@...sed-offsets
>> +igt@..._addfb_basic@...sed-modifier
>> +igt@..._addfb_basic@...acy-format
>> +igt@..._addfb_basic@...handle
>> +igt@..._addfb_basic@...ic
>> +igt@..._addfb_basic@...-pitch-0
>> +igt@..._addfb_basic@...-pitch-32
>> +igt@..._addfb_basic@...-pitch-63
>> +igt@..._addfb_basic@...-pitch-128
>> +igt@..._addfb_basic@...-pitch-256
>> +igt@..._addfb_basic@...-pitch-1024
>> +igt@..._addfb_basic@...-pitch-999
>> +igt@..._addfb_basic@...-pitch-65536
>> +igt@..._addfb_basic@...e-max
>> +igt@..._addfb_basic@...-wide
>> +igt@..._addfb_basic@...-high
>> +igt@..._addfb_basic@...too-small
>> +igt@..._addfb_basic@...ll-bo
>> +igt@..._addfb_basic@...fb25-modifier-no-flag
>> +igt@..._addfb_basic@...fb25-bad-modifier
>> +igt@..._addfb_basic@...alid-get-prop-any
>> +igt@..._addfb_basic@...alid-get-prop
>> +igt@..._addfb_basic@...alid-set-prop-any
>> +igt@..._addfb_basic@...alid-set-prop
>> +igt@..._addfb_basic@...ter-rmfb
>> +igt@..._atomic_interruptible@...acy-setmode@...e-a-edp-1
>> +igt@..._atomic_interruptible@...mic-setmode@...e-a-edp-1
>> +igt@..._atomic_interruptible@...acy-dpms@...e-a-edp-1
>> +igt@..._atomic_interruptible@...acy-pageflip@...e-a-edp-1
>> +igt@..._atomic_interruptible@...acy-cursor@...e-a-edp-1
>> +igt@..._atomic_interruptible@...versal-setplane-primary@...e-a-edp-1
>> +igt@..._atomic_interruptible@...versal-setplane-cursor@...e-a-edp-1
>> +igt@..._content_protection@LIC
>> +igt@..._getfb@...fb-handle-zero
>> +igt@..._getfb@...fb-handle-valid
>> +igt@..._getfb@...fb-handle-closed
>> +igt@..._getfb@...fb-handle-not-fb
>> +igt@..._getfb@...fb-addfb-different-handles
>> +igt@..._getfb@...fb-repeated-different-handles
>> +igt@..._getfb@...fb2-handle-zero
>> +igt@..._getfb@...fb2-handle-closed
>> +igt@..._getfb@...fb2-handle-not-fb
>> +igt@..._getfb@...fb2-into-addfb2
>> +igt@..._getfb@...fb-handle-protection
>> +igt@..._getfb@...fb2-handle-protection
>> +igt@..._hdmi_inject@...ect-4k
>> +igt@..._prop_blob@...ic
>> +igt@..._prop_blob@...b-prop-core
>> +igt@..._prop_blob@...b-prop-validate
>> +igt@..._prop_blob@...b-prop-lifetime
>> +igt@..._prop_blob@...b-multiple
>> +igt@..._prop_blob@...alid-get-prop-any
>> +igt@..._prop_blob@...alid-get-prop
>> +igt@..._prop_blob@...alid-set-prop-any
>> +igt@..._prop_blob@...alid-set-prop
>> +igt@..._setmode@...ic@...e-a-edp-1
>> +igt@..._setmode@...ic@...e-b-edp-1
>> +igt@..._sysfs_edid_timing
>> diff --git a/drivers/gpu/drm/ci/meson_g12b_results.txt 
>> b/drivers/gpu/drm/ci/meson_g12b_results.txt
>> new file mode 100644
>> index 000000000000..6997a75aa1ef
>> --- /dev/null
>> +++ b/drivers/gpu/drm/ci/meson_g12b_results.txt
>> @@ -0,0 +1,11 @@
>> +igt@..._read@...alid-buffer,fail
>> +igt@..._addfb_basic@...fb25-bad-modifier,fail
>> +igt@..._atomic_interruptible@...acy-setmode,skip
>> +igt@..._atomic_interruptible@...mic-setmode,skip
>> +igt@..._atomic_interruptible@...acy-dpms,skip
>> +igt@..._atomic_interruptible@...acy-pageflip,skip
>> +igt@..._atomic_interruptible@...acy-cursor,skip
>> +igt@..._atomic_interruptible@...versal-setplane-primary,skip
>> +igt@..._atomic_interruptible@...versal-setplane-cursor,skip
>> +igt@..._content_protection@lic,skip
>> +igt@..._setmode@...ic,skip
> 
> Can't those be removed from testlist instead ?
> I suppose those should not skip, right ?

You are right, these slipped in. The only skips in the results should be 
for tests that do run on other boards for the same driver.

> Anyway, this is great, please take my
> Reviewed-by: Neil Armstrong <narmstrong@...libre.com>
> for stuff related to the meson driver.

Thanks,

Tomeu

Powered by blists - more mailing lists