| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID:
<SA3PR13MB63726E6742A05C401D484147FD5F2@SA3PR13MB6372.namprd13.prod.outlook.com>
Date: Thu, 29 Feb 2024 02:44:47 +0000
From: "Bird, Tim" <Tim.Bird@...y.com>
To: Helen Koike <helen.koike@...labora.com>,
"linuxtv-ci@...uxtv.org"
<linuxtv-ci@...uxtv.org>,
"dave.pigott@...labora.com"
<dave.pigott@...labora.com>,
"mripard@...nel.org" <mripard@...nel.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
"linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
"gustavo.padovan@...labora.com" <gustavo.padovan@...labora.com>,
"pawiecz@...labora.com" <pawiecz@...labora.com>,
"spbnick@...il.com"
<spbnick@...il.com>,
"tales.aparecida@...il.com" <tales.aparecida@...il.com>,
"workflows@...r.kernel.org" <workflows@...r.kernel.org>,
"kernelci@...ts.linux.dev" <kernelci@...ts.linux.dev>,
"skhan@...uxfoundation.org" <skhan@...uxfoundation.org>,
"kunit-dev@...glegroups.com" <kunit-dev@...glegroups.com>,
"nfraprado@...labora.com" <nfraprado@...labora.com>,
"davidgow@...gle.com"
<davidgow@...gle.com>,
"cocci@...ia.fr" <cocci@...ia.fr>,
"Julia.Lawall@...ia.fr" <Julia.Lawall@...ia.fr>,
"laura.nao@...labora.com"
<laura.nao@...labora.com>,
"ricardo.canuelo@...labora.com"
<ricardo.canuelo@...labora.com>,
"kernel@...labora.com"
<kernel@...labora.com>,
"torvalds@...uxfoundation.org"
<torvalds@...uxfoundation.org>,
"gregkh@...uxfoundation.org"
<gregkh@...uxfoundation.org>
Subject: RE: [PATCH 1/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel
Testing
> -----Original Message-----
> From: Helen Koike <helen.koike@...labora.com>
>
....
> Hey all,
>
> You can check the validation of this patchset on:
> https://gitlab.collabora.com/koike/linux/-/pipelines/87035
>
> I would appreciate your feedback on this work, what do you think?
>
> If you would rate from 0 to 5, where:
>
> [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile.
> [ ] 1. I don't find it useful in its current form.
> [ ] 2. It might be useful to others, but not for me.
> [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow.
> [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow.
> [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :)
>
> Which rating would you select?
For me, this is a "5". I don't currently use gitlab, but I might install it just to test this out.
It looks like there are a large number of YAML files which define the integration between the
test scripts and gitlab. Also, there are a number of shell scripts to perform some of the setup
and tests. Do you have any idea how difficult it would be to use the shell scripts outside of
the CI system (e.g. manually)? That is, are there dependencies between the CI system
and the shell scripts?
I think the convention, of putting this kind of stuff under a "ci" directory, makes sense.
And it appears that the sub-dir structure allows for other CI systems to add their
own config and files also.
I have a few minor comments below.
>
> ---
> .gitlab-ci.yml | 2 +
> MAINTAINERS | 8 ++
> ci/gitlab-ci/bootstrap-gitlab-runner.sh | 55 +++++++++
> ci/gitlab-ci/ci-scripts/build-docs.sh | 35 ++++++
> ci/gitlab-ci/ci-scripts/build-kernel.sh | 35 ++++++
> ci/gitlab-ci/ci-scripts/ici-functions.sh | 104 ++++++++++++++++++
> ci/gitlab-ci/ci-scripts/install-smatch.sh | 13 +++
> .../ci-scripts/parse_commit_message.sh | 27 +++++
> ci/gitlab-ci/ci-scripts/run-checkpatch.sh | 20 ++++
> ci/gitlab-ci/ci-scripts/run-smatch.sh | 45 ++++++++
> ci/gitlab-ci/docker-compose.yaml | 18 +++
> ci/gitlab-ci/linux.code-workspace | 11 ++
> ci/gitlab-ci/yml/build.yml | 43 ++++++++
> ci/gitlab-ci/yml/cache.yml | 26 +++++
> ci/gitlab-ci/yml/container.yml | 36 ++++++
> ci/gitlab-ci/yml/gitlab-ci.yml | 71 ++++++++++++
> ci/gitlab-ci/yml/kernel-combinations.yml | 18 +++
> ci/gitlab-ci/yml/scenarios.yml | 12 ++
> ci/gitlab-ci/yml/scenarios/file-systems.yml | 21 ++++
> ci/gitlab-ci/yml/scenarios/media.yml | 21 ++++
> ci/gitlab-ci/yml/scenarios/network.yml | 21 ++++
> ci/gitlab-ci/yml/static-checks.yml | 21 ++++
> 22 files changed, 663 insertions(+)
> create mode 100644 .gitlab-ci.yml
> create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh
> create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh
> create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh
> create mode 100644 ci/gitlab-ci/docker-compose.yaml
> create mode 100644 ci/gitlab-ci/linux.code-workspace
> create mode 100644 ci/gitlab-ci/yml/build.yml
> create mode 100644 ci/gitlab-ci/yml/cache.yml
> create mode 100644 ci/gitlab-ci/yml/container.yml
> create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml
> create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml
> create mode 100644 ci/gitlab-ci/yml/scenarios.yml
> create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml
> create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml
> create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml
> create mode 100644 ci/gitlab-ci/yml/static-checks.yml
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> new file mode 100644
> index 0000000000000..fac523bb86866
> --- /dev/null
> +++ b/.gitlab-ci.yml
> @@ -0,0 +1,2 @@
> +include:
> + - ci/gitlab-ci/yml/gitlab-ci.yml
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 716b2e22598c8..aa0f65791c2ee 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -4998,6 +4998,14 @@ T: git git://linuxtv.org/media_tree.git
> F: Documentation/devicetree/bindings/media/i2c/chrontel,ch7322.yaml
> F: drivers/media/cec/i2c/ch7322.c
>
> +CI AUTOMATED TESTING
> +M: Helen Koike <helen.koike@...labora.com>
> +L: linux-kernel@...r.kernel.org
> +S: Maintained
> +T: git https://gitlab.collabora.com/koike/linux.git
> +F: .gitlab-ci.yml
> +F: ci/
> +
> CIRRUS LOGIC AUDIO CODEC DRIVERS
> M: James Schulman <james.schulman@...rus.com>
> M: David Rhodes <david.rhodes@...rus.com>
> diff --git a/ci/gitlab-ci/bootstrap-gitlab-runner.sh b/ci/gitlab-ci/bootstrap-gitlab-runner.sh
> new file mode 100755
> index 0000000000000..73238960d0880
> --- /dev/null
> +++ b/ci/gitlab-ci/bootstrap-gitlab-runner.sh
> @@ -0,0 +1,55 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -eo pipefail
> +
> +# Define variables
> +CONFIG_VOLUME="/srv/gitlab-runner/config" # Path to your GitLab Runner config
> +
> +# Check if RUNNER_REGISTRATION_TOKEN is set
> +if [ -z "${RUNNER_REGISTRATION_TOKEN}" ]; then
> + echo "Error: RUNNER_REGISTRATION_TOKEN is not set."
> + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again."
> + exit 1
> +fi
> +
> +# Check if GITLAB_URL is set
> +if [ -z "${GITLAB_URL}" ]; then
> + GITLAB_URL="https://gitlab.com/"
> + echo "Info: GITLAB_URL is not set. Using the default $GITLAB_URL"
> + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again."
> +fi
> +
> +# Check if docker-compose is installed
> +if ! command -v docker-compose &> /dev/null
> +then
> + echo "docker-compose could not be found. Please install it first."
> + exit 1
> +fi
> +
> +# Start the GitLab Runner using Docker Compose
> +echo "Starting GitLab Runner..."
> +docker-compose up -d
> +
> +# Wait for a few seconds to ensure the service is up
> +sleep 5
> +
> +# Register the GitLab Runner
> +echo "Registering GitLab Runner..."
> +docker run --rm -v ${CONFIG_VOLUME}:/etc/gitlab-runner gitlab/gitlab-runner register \
> + --non-interactive \
> + --url ${GITLAB_URL} \
> + --token ${RUNNER_REGISTRATION_TOKEN} \
> + --executor docker \
> + --docker-image "alpine:latest" \
> + --description "Docker Runner" \
> + --docker-privileged
> +
> +echo ""
> +echo "INFO: To configure the number of concurrent jobs, edit the value of"
> +echo "INFO: concurrent in ${CONFIG_VOLUME}/config.toml, than restart the GitLab"
> +echo "INFO: Runner using docker-compose restart"
> +echo ""
> +echo "GitLab Runner setup complete."
> diff --git a/ci/gitlab-ci/ci-scripts/build-docs.sh b/ci/gitlab-ci/ci-scripts/build-docs.sh
> new file mode 100755
> index 0000000000000..2053cb8794acf
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/build-docs.sh
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +source ci/gitlab-ci/ci-scripts/ici-functions.sh
> +
> +ici_get_patch_series_size
> +
> +# Get the list of modified files in the last $ICI_PATCH_SERIES_SIZE commits
> +MODIFIED_DOC_FILES=$(git diff HEAD~$ICI_PATCH_SERIES_SIZE --name-only -- Documentation/)
> +
> +make -j$(nproc) "$ICI_DOC_TYPE" 2>&1 | tee output.txt
> +
> +mkdir -p "${CI_PROJECT_DIR}/artifacts"
> +mv Documentation/output "${CI_PROJECT_DIR}/artifacts/Documentation-output"
> +
> +# Check if any of the MODIFIED_DOC_FILES generated a warning
> +# NOTE: the alternative solution was to touch the modified files and run make
> +# again, but too much warnings still appears
> +for file in $MODIFIED_DOC_FILES; do
> + if grep -qi "warning" output.txt && grep -q "$file" output.txt; then
> + echo "Warning found in $file"
> + exit 101
> + fi
> +done
> +
> +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then
> + # If the patch series size was not detected, exit with a warning
> + echo -n "The patch series size was not detected, we probably didn't check the"
> + echo " whole series. Exiting with a warning."
> + exit 101
> +fi
> diff --git a/ci/gitlab-ci/ci-scripts/build-kernel.sh b/ci/gitlab-ci/ci-scripts/build-kernel.sh
> new file mode 100755
> index 0000000000000..361826368a573
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/build-kernel.sh
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +source ci/gitlab-ci/ci-scripts/ici-functions.sh
> +
> +ici_prepare_build
> +
> +pushd build
> +
> +# compile the entire kernel
> +make CF=-D__CHECK_ENDIAN__ -C "$ICI_KERNEL_DIR" O=$(pwd) -j$(nproc) 2>&1 | tee output.txt
> +
> +export INSTALL_PATH="${CI_PROJECT_DIR}/artifacts/"
> +INSTALL_PATH+="kernel-install-${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config"
> +mkdir -p "$INSTALL_PATH"
> +
> +# install the kernel image to artifacts/kernel-install
> +make -C "$ICI_KERNEL_DIR" O=$(pwd) install INSTALL_PATH="$INSTALL_PATH"
> +
> +# install kernel modules to artifacts/kernel-install
> +make -C "$ICI_KERNEL_DIR" O=$(pwd) modules_install INSTALL_MOD_PATH="$INSTALL_PATH"
> +
> +# export config as artifact
> +cp .config "${CI_PROJECT_DIR}/artifacts/${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config"
> +
> +# if the compilation has warnings, exit with the warning code
> +if grep -iq "warning" output.txt; then
I'm not sure, but do you need a popd here in this early exit path?
> + exit 101
> +fi
> +
> +popd
> diff --git a/ci/gitlab-ci/ci-scripts/ici-functions.sh b/ci/gitlab-ci/ci-scripts/ici-functions.sh
> new file mode 100644
> index 0000000000000..df133b4d00103
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/ici-functions.sh
> @@ -0,0 +1,104 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +# internal CI bash functions
> +
> +# convention:
> +# KCI_<VARIABLE_NAME> for variables defined by the user (outside of this script)
> +# ICI_<VARIABLE_NAME> for variables defined internally for usage between scripts
> +# CI_<VARIABLE_NAME> for variables defined by GitLab CI
> +
> +
> +ici_prepare_build() {
> + BUILD_DIR="${1:-build}"
> +
> + echo ""
> + echo "Architecture: $KCI_KERNEL_ARCH"
> + echo "Defconfig: $KCI_DEFCONFIG"
> + echo ""
> +
> + # Get the current directory if KCI_KERNEL_DIR is not set
> + ICI_KERNEL_DIR="${KCI_KERNEL_DIR:-$(pwd)}"
> +
> + cd "$ICI_KERNEL_DIR" || { echo "Kernel directory not found"; exit 1; }
> +
> + # Clean up stale rebases that GitLab might not have removed when reusing a checkout dir
> + rm -rf .git/rebase-apply
> +
> + if [[ "$KCI_KERNEL_ARCH" = "arm64" ]]; then
> + GCC_ARCH="aarch64-linux-gnu"
> + elif [[ "$KCI_KERNEL_ARCH" = "arm" ]]; then
> + GCC_ARCH="arm-linux-gnueabihf"
> + else
> + GCC_ARCH="x86_64-linux-gnu"
> + fi
> +
> + # do not set ARCH and CROSS_COMPILE if KCI_KERNEL_ARCH is not set, useful for local run
> + if [ -n "$KCI_KERNEL_ARCH" ]; then
> + export ARCH=${KCI_KERNEL_ARCH}
> + export CROSS_COMPILE="${GCC_ARCH}-"
> + fi
> +
> + mkdir -p "$BUILD_DIR"
> +
> + pushd "$BUILD_DIR" || { echo "Failed to create $BUILD_DIR directory"; exit 1; }
> +
> + # generate defconfig
> + make -C "$ICI_KERNEL_DIR" O=$(pwd) $(basename ${KCI_DEFCONFIG-"defconfig"})
> +
> + # add extra configs from variable KCI_KCONFIGS_{ENABLE,DISABLE,MODULE}
> + for opt in $KCI_KCONFIGS_ENABLE; do
> + ../scripts/config --file .config --enable CONFIG_$opt
> + done
> + for opt in $KCI_KCONFIGS_DISABLE; do
> + ../scripts/config --file .config --disable CONFIG_$opt
> + done
> + for opt in $KCI_KCONFIGS_MODULE; do
> + ../scripts/config --file .config --module CONFIG_$opt
> + done
> +
> + if [ -n "$KCI_KCONFIGS_DISABLE" ] || [ -n "$KCI_KCONFIGS_ENABLE" ] ||
> + [ -n "$KCI_KCONFIGS_MODULE" ]; then
> + # execude olddefconfig only if we changed the default config, otherwise,
> + # let it raise warnings if any
> + make -C "$ICI_KERNEL_DIR" O=$(pwd) olddefconfig
> + fi
> +
> + popd
> +}
> +
> +ici_get_patch_series_size()
> +{
> + local CLONE_DEPTH
> + CLONE_DEPTH=$(git rev-list --count HEAD)
> + echo "The depth of the clone is $CLONE_DEPTH"
> +
> + # If this is in the context of a merge request, calculate the patch series
> + # size comparing to the target branch
> + if [ -n "$CI_MERGE_REQUEST_IID" ]; then
> + git fetch origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" --depth $CLONE_DEPTH
> + BASE_COMMIT="origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
> + ICI_PATCH_SERIES_SIZE=$(git rev-list --count ${BASE_COMMIT}.."$CI_COMMIT_SHA")
> +
> + # if KCI_PATCH_SERIES_SIZE is set, use it
> + elif [ -n "$KCI_PATCH_SERIES_SIZE" ]; then
> + ICI_PATCH_SERIES_SIZE="$KCI_PATCH_SERIES_SIZE"
> + else
> + ICI_PATCH_SERIES_SIZE=1
> + echo "WARNING: unable to detect the patch series size, using the default value of 1."
> + # shellcheck disable=SC2034
> + ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE=true
> + fi
> +
> + # Check if the clone depth is smaller than or equal to KCI_PATCH_SERIES_SIZE,
> + # otherwise the checkpatch.pl hangs
> + if [ "$ICI_PATCH_SERIES_SIZE" -ge "$CLONE_DEPTH" ]; then
> + echo -n "ERROR: the depth of the clone is $CLONE_DEPTH, smaller than or equal to the patch"
> + echo " series size. Update your GitLab configuration to increase the size of the clone."
> + return 1
> + fi
> +}
> diff --git a/ci/gitlab-ci/ci-scripts/install-smatch.sh b/ci/gitlab-ci/ci-scripts/install-smatch.sh
> new file mode 100755
> index 0000000000000..cae16a0e90fa8
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/install-smatch.sh
> @@ -0,0 +1,13 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +pushd /
At the root? really? Could this be /usr/local/src or $HOME/src or something?
or is this inside a container and doesn't matter?
> +git clone --depth 1 https://repo.or.cz/smatch.git
> +pushd smatch
> +make
> +popd
> +popd
> diff --git a/ci/gitlab-ci/ci-scripts/parse_commit_message.sh b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh
> new file mode 100755
> index 0000000000000..c9792c64ad51e
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh
> @@ -0,0 +1,27 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +# Get the last commit message
> +commit_message=$(git log -1 --pretty=%B)
> +
> +pattern='(KCI_[A-Za-z_]+)=("[^"]*"|[^ ]+)'
> +
> +while read -r line; do
> + if [[ $line =~ $pattern ]]; then
> + variable_name="${BASH_REMATCH[1]}"
> + variable_value="${BASH_REMATCH[2]}"
> +
> + # Remove quotes if present
> + variable_value="${variable_value%\"}"
> + variable_value="${variable_value#\"}"
> +
> + # Export the variable
> + export "$variable_name=$variable_value"
> +
> + echo "Exported $variable_name=$variable_value"
> + fi
> +done <<< "$commit_message"
> diff --git a/ci/gitlab-ci/ci-scripts/run-checkpatch.sh b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh
> new file mode 100755
> index 0000000000000..98585ec18be2b
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh
> @@ -0,0 +1,20 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +source ci/gitlab-ci/ci-scripts/ici-functions.sh
> +
> +ici_get_patch_series_size
> +
> +# shellcheck disable=SC2086
> +scripts/checkpatch.pl $KCI_CHECKPATCH_OPTIONS --git HEAD-"$ICI_PATCH_SERIES_SIZE"
> +
> +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then
> + # If the patch series size was not detected, exit with a warning
> + echo -n "The patch series size was not detected, we probably didn't check the whole series."
> + echo " Exiting with a warning."
> + exit 101
> +fi
> diff --git a/ci/gitlab-ci/ci-scripts/run-smatch.sh b/ci/gitlab-ci/ci-scripts/run-smatch.sh
> new file mode 100755
> index 0000000000000..ec054df9ef179
> --- /dev/null
> +++ b/ci/gitlab-ci/ci-scripts/run-smatch.sh
> @@ -0,0 +1,45 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +set -exo pipefail
> +
> +source ci/gitlab-ci/ci-scripts/ici-functions.sh
> +
> +ls -l
> +pwd
> +
> +# generate config file
> +ici_prepare_build
> +
> +ici_get_patch_series_size
> +
> +cp build/.config .
> +
> +# Get a list of modified .c files in the last ICI_PATCH_SERIES_SIZE commits
> +MODIFIED_C_FILES=$(git diff --name-only HEAD~$ICI_PATCH_SERIES_SIZE HEAD | grep '\.c$' || true)
> +
> +# Check if any .c files were modified
> +if [ -z "$MODIFIED_C_FILES" ]; then
> + echo "No .c files were modified in the last $ICI_PATCH_SERIES_SIZE commits."
> +else
> + echo "Running kchecker on modified .c files..."
> + mkdir -p "$CI_PROJECT_DIR"/artifacts
> +fi
> +
> +# Run kchecker on each modified .c file
> +for file in $MODIFIED_C_FILES; do
> + if [ -f "$file" ]; then
> + /smatch/smatch_scripts/kchecker "$file" | tee "$CI_PROJECT_DIR"/artifacts/smatch_warns.txt
> + else
> + echo "File not found: $file"
> + fi
> +done
> +
> +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then
> + # If the patch series size was not detected, exit with a warning
> + echo -n "The patch series size was not detected, we probably didn't check the whole series."
> + echo " Exiting with a warning."
> + exit 101
> +fi
> diff --git a/ci/gitlab-ci/docker-compose.yaml b/ci/gitlab-ci/docker-compose.yaml
> new file mode 100644
> index 0000000000000..7edf8055ca375
> --- /dev/null
> +++ b/ci/gitlab-ci/docker-compose.yaml
> @@ -0,0 +1,18 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +version: '3.8'
> +
> +services:
> + gitlab-runner:
> + image: gitlab/gitlab-runner:latest
> + container_name: gitlab-runner
> + restart: always
> + privileged: true
> + volumes:
> + - /srv/gitlab-runner/config:/etc/gitlab-runner
> + - /var/run/docker.sock:/var/run/docker.sock
> +
> +# To register the GitLab Runner, run the following command:
> +# docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register --url https://gitlab.com --token
> YOUR_REGISTRATION_TOKEN
> diff --git a/ci/gitlab-ci/linux.code-workspace b/ci/gitlab-ci/linux.code-workspace
> new file mode 100644
> index 0000000000000..dd76698e38c2c
> --- /dev/null
> +++ b/ci/gitlab-ci/linux.code-workspace
> @@ -0,0 +1,11 @@
> +{
> + "folders": [
> + {
> + "path": "../../../.."
> + },
> + {
> + "path": "../../../../../mesa"
> + }
> + ],
> + "settings": {}
> +}
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/build.yml b/ci/gitlab-ci/yml/build.yml
> new file mode 100644
> index 0000000000000..f81410c293f35
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/build.yml
> @@ -0,0 +1,43 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +.build:
> + extends: .use-debian/x86_64_build
> + stage: build
> +
> +build-kernel:
> + extends:
> + - .build
> + - .kernel-combinations
> + variables:
> + # ccache in gitlab-runner to speed up builds
> + CCACHE_BASEDIR: $CI_PROJECT_DIR
> + CCACHE_DIR: $CI_PROJECT_DIR/ccache
> + CCACHE_COMPILERCHECK: content
> + cache:
> + - key: ccache-$CI_JOB_NAME
> + paths:
> + - $CCACHE_DIR
> + before_script:
> + - export PATH="/usr/lib/ccache:$PATH"
> + - ccache --zero-stats || true
> + - ccache --show-stats || true
> + after_script:
> + - ccache --show-stats
> + script:
> + - ./ci/gitlab-ci/ci-scripts/build-kernel.sh
> +
> +build-docs:
> + extends:
> + - .build
> + parallel:
> + matrix:
> + - ICI_DOC_TYPE: "htmldocs"
> + # TODO: re-add pdfdocs once build errors are fixed
> + # - ICI_DOC_TYPE: "pdfdocs"
> + - ICI_DOC_TYPE: "latexdocs"
> + - ICI_DOC_TYPE: "epubdocs"
> + script:
> + - ./ci/gitlab-ci/ci-scripts/build-docs.sh
> + when: manual
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/cache.yml b/ci/gitlab-ci/yml/cache.yml
> new file mode 100644
> index 0000000000000..efe4fc69b11ca
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/cache.yml
> @@ -0,0 +1,26 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +# Smatch db is saved to a cached folder, so it can be used by other jobs and pipelines.
> +# It is set to manual so it can be run when needed
> +
> +.use-cache-smatch-db:
> + cache:
> + # TODO: check if this cache shouldn't be per architecture
> + - key: smatch-db
> + paths:
> + - /smatch/smatch_data
> +
> +smatch-db-generate:
> + stage: cache
> + extends:
> + - .kernel-combinations
> + - .use-debian/x86_64_build
> + - .use-cache-smatch-db
> + script:
> + - source ci/gitlab-ci/ci-scripts/ici-functions.sh
> + - ici_prepare_build
> + - cp build/.config .
> + - /smatch/smatch_scripts/build_kernel_data.sh
> + when: manual
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/container.yml b/ci/gitlab-ci/yml/container.yml
> new file mode 100644
> index 0000000000000..eecca95caca0e
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/container.yml
> @@ -0,0 +1,36 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +.prep-debian/x86_64_build:
> + variables:
> + FDO_DISTRIBUTION_VERSION: bookworm-slim
> + FDO_DISTRIBUTION_TAG: "2024-02-28-ci-test-1"
> +
> +
> +debian/x86_64_build:
> + extends:
> + - ".fdo.container-build@...ian"
> + - ".prep-debian/x86_64_build"
> + variables:
> + FDO_DISTRIBUTION_PACKAGES: >
> + make bc flex bison pahole mount build-essential
> + libcairo-dev libdw-dev libjson-c-dev libkmod2
> + libkmod-dev libpciaccess-dev libproc2-dev libudev-dev
> + libunwind-dev python3-docutils bc python3-ply
> + libssl-dev ccache
> + gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu
> + gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf
> + perl git kmod python3-git python3-yaml python3-dulwich
> + sqlite3 libsqlite3-dev libdbd-sqlite3-perl libssl-dev libtry-tiny-perl
> + python3-sphinx imagemagick graphviz dvipng python3-venv fonts-noto-cjk
> + latexmk librsvg2-bin texlive-lang-chinese texlive-xetex
> + FDO_DISTRIBUTION_EXEC: ./ci/gitlab-ci/ci-scripts/install-smatch.sh
> + stage: container
> +
> +
> +.use-debian/x86_64_build:
> + extends:
> + - ".fdo.distribution-image@...ian"
> + - ".prep-debian/x86_64_build"
> + needs: [debian/x86_64_build]
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml
> new file mode 100644
> index 0000000000000..57b9c02904713
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/gitlab-ci.yml
> @@ -0,0 +1,71 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +workflow:
> + name: $PIPELINE_NAME
> + rules:
> + # when triggered as a multi-project pipeline for an MR
> + - if: $CI_PIPELINE_SOURCE == 'pipeline' && $PARENT_MERGE_REQUEST_IID != null && $PARENT_MERGE_REQUEST_IID != ""
> + variables:
> + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH!$PARENT_MERGE_REQUEST_IID'
> + # when triggered as a multi-project pipeline
> + - if: $CI_PIPELINE_SOURCE == 'pipeline'
> + variables:
> + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH'
> + # when triggered via a schedule
> + - if: $CI_PIPELINE_SOURCE == 'schedule'
> + variables:
> + PIPELINE_NAME: 'Scheduled pipeline for $ONLY_JOB_NAME'
> + # for merge requests
> + - if: $CI_MERGE_REQUEST_ID
> + # when triggered via the REST api
> + - if: $CI_PIPELINE_SOURCE == 'api'
> + # for the tip of the default branch
> + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
> + # when triggered via a trigger token
> + - if: $CI_PIPELINE_SOURCE == 'trigger'
> + # when triggered from a button press in the web interface
> + - if: $CI_PIPELINE_SOURCE == 'web'
> + # for branch tips without open MRs, ignoring special branches
> + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_OPEN_MERGE_REQUESTS == null
> + # when forced via '-o ci.variable="FORCE_CI=true"' during pushing
> + - if: $FORCE_CI == 'true'
> +
> +variables:
> + FDO_UPSTREAM_REPO: helen.fornazier/linux # The repo where to look for cached images
> + # ccache builds in gitlab-runner to speed up builds
> + SMATCH_DB_DIR: /smatch/smatch_data
> + # exit code of bash script on `script` will be the exit code of the job
> + FF_USE_NEW_BASH_EVAL_STRATEGY: "true"
> +
> +default:
> + artifacts:
> + paths:
> + - artifacts/
> + when: always
> +
> +include:
> + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci-
> fairy.yml'
> + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-
> /raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml'
> +
> + - ci/gitlab-ci/yml/kernel-combinations.yml
> + - ci/gitlab-ci/yml/container.yml
> + - ci/gitlab-ci/yml/cache.yml
> + - ci/gitlab-ci/yml/build.yml
> + - ci/gitlab-ci/yml/static-checks.yml
> + - ci/gitlab-ci/yml/scenarios.yml
> +
> +before_script:
> + - source ci/gitlab-ci/ci-scripts/parse_commit_message.sh
> +
> +.use-debian/x86_64_build:
> + allow_failure:
> + # Code to exit with a warning
> + exit_codes: 101
> +
> +stages:
> + - container
> + - static-checks
> + - build
> + - cache
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/kernel-combinations.yml b/ci/gitlab-ci/yml/kernel-combinations.yml
> new file mode 100644
> index 0000000000000..87f08579ffd62
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/kernel-combinations.yml
> @@ -0,0 +1,18 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +.kernel-combinations:
> + parallel:
> + matrix:
> + - KCI_KERNEL_ARCH: "x86_64"
> + KCI_DEFCONFIG: "x86_64_defconfig"
> + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH"
> +
> + - KCI_KERNEL_ARCH: "arm64"
> + KCI_DEFCONFIG: "defconfig"
> + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH"
> +
> + - KCI_KERNEL_ARCH: "arm"
> + KCI_DEFCONFIG: "multi_v7_defconfig"
> + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH"
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/scenarios.yml b/ci/gitlab-ci/yml/scenarios.yml
> new file mode 100644
> index 0000000000000..11598c3b6f2a9
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/scenarios.yml
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +# Extend the CI by including a test scenario here. The scenario will be
> +# activated if KCI_SCENARIO is set to the scenario name. KCI_SCENARIO can be
> +# defined in the gitlab-ci.yml file, or through GitLab UI.
> +
> +include:
> + - local: 'ci/gitlab-ci/yml/scenarios/$KCI_SCENARIO.yml'
> + rules:
> + - if: '$KCI_SCENARIO'
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/scenarios/file-systems.yml b/ci/gitlab-ci/yml/scenarios/file-systems.yml
> new file mode 100644
> index 0000000000000..66456600a9dd3
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/scenarios/file-systems.yml
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +# Write here specific configurations and extensions for the given scenario
> +
> +# Example - overwrite kernel combinations in the pipeline
> +# .kernel-combinations:
> +# parallel:
> +# matrix:
> +# - KCI_KERNEL_ARCH: "x86_64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/scenarios/media.yml b/ci/gitlab-ci/yml/scenarios/media.yml
> new file mode 100644
> index 0000000000000..66456600a9dd3
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/scenarios/media.yml
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +# Write here specific configurations and extensions for the given scenario
> +
> +# Example - overwrite kernel combinations in the pipeline
> +# .kernel-combinations:
> +# parallel:
> +# matrix:
> +# - KCI_KERNEL_ARCH: "x86_64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> \ No newline at end of file
> diff --git a/ci/gitlab-ci/yml/scenarios/network.yml b/ci/gitlab-ci/yml/scenarios/network.yml
> new file mode 100644
> index 0000000000000..66456600a9dd3
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/scenarios/network.yml
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +# Write here specific configurations and extensions for the given scenario
> +
> +# Example - overwrite kernel combinations in the pipeline
> +# .kernel-combinations:
> +# parallel:
> +# matrix:
> +# - KCI_KERNEL_ARCH: "x86_64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm64"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> +#
> +# - KCI_KERNEL_ARCH: "arm"
> +# KCI_DEFCONFIG: "path/to/my/config/file_config"
> +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..."
> \ No newline at end of file
Do you have any real examples of scenario files? These all seem like
placeholders.
> diff --git a/ci/gitlab-ci/yml/static-checks.yml b/ci/gitlab-ci/yml/static-checks.yml
> new file mode 100644
> index 0000000000000..a99a1ea910f40
> --- /dev/null
> +++ b/ci/gitlab-ci/yml/static-checks.yml
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@...labora.com>
> +
> +.static-checks:
> + stage: static-checks
> + extends:
> + - .use-debian/x86_64_build
> +
> +checkpatch:
> + extends: .static-checks
> + script:
> + - ci/gitlab-ci/ci-scripts/run-checkpatch.sh
> +
> +smatch:
> + extends:
> + - .static-checks
> + - .kernel-combinations
> + - .use-cache-smatch-db
> + script:
> + - ci/gitlab-ci/ci-scripts/run-smatch.sh
> \ No newline at end of file
> --
> 2.40.1
>
Looks very interesting!
-- Tim
Powered by blists - more mailing lists