linux-kernel - Re: LKFT CI: improving Networking selftests results when validating stable kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1bda012e-817a-45be-82e2-03ac78c58034@stanley.mountain>
Date: Wed, 13 Nov 2024 20:08:12 +0300
From: Dan Carpenter <dan.carpenter@...aro.org>
To: Matthieu Baerts <matttbe@...nel.org>,
	Greg KH <gregkh@...uxfoundation.org>, Shuah Khan <shuah@...nel.org>
Cc: Linux Kernel Functional Testing <lkft@...aro.org>,
	Kernel Selftests <linux-kselftest@...r.kernel.org>,
	Netdev <netdev@...r.kernel.org>,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
	Willem de Bruijn <willemb@...gle.com>,
	Naresh Kamboju <naresh.kamboju@...aro.org>,
	Ido Schimmel <idosch@...dia.com>, stable@...r.kernel.org
Subject: Re: LKFT CI: improving Networking selftests results when validating
 stable kernels

On Fri, Nov 08, 2024 at 07:21:59PM +0100, Matthieu Baerts wrote:
> KSelftests from the same version
> --------------------------------
> 
> According to the doc [2], kselftests should support all previous kernel
> versions. The LKFT CI is then using the kselftests from the last stable
> release to validate all stable versions. Even if there are good reasons
> to do that, we would like to ask for an opt-out for this policy for the
> networking tests: this is hard to maintain with the increased
> complexity, hard to validate on all stable kernels before applying
> patches, and hard to put in place in some situations. As a result, many
> tests are failing on older kernels, and it looks like it is a lot of
> work to support older kernels, and to maintain this.
> 
> Many networking tests are validating the internal behaviour that is not
> exposed to the userspace. A typical example: some tests look at the raw
> packets being exchanged during a test, and this behaviour can change
> without modifying how the userspace is interacting with the kernel. The
> kernel could expose capabilities, but that's not something that seems
> natural to put in place for internal behaviours that are not exposed to
> end users. Maybe workarounds could be used, e.g. looking at kernel
> symbols, etc. Nut that doesn't always work, increase the complexity, and
> often "false positive" issue will be noticed only after a patch hits
> stable, and will cause a bunch of tests to be ignored.
> 
> Regarding fixes, ideally they will come with a new or modified test that
> can also be backported. So the coverage can continue to grow in stable
> versions too.
> 
> Do you think that from the kernel v6.12 (or before?), the LKFT CI could
> run the networking kselftests from the version that is being validated,
> and not from a newer one? So validating the selftests from v6.12.1 on a
> v6.12.1, and not the ones from a future v6.16.y on a v6.12.42.
> 

These kinds of decisions are something that Greg and Shuah need to decide on.

You would still need some way to automatically detect that kselftest is running
on an old kernel and disable the networking checks.  Otherwise when random
people on the internet try to run selftests they would run into issues.

regards,
dan carpenter