linux-kernel - Re: [PATCH RFC] kselftest: devices: Add test to detect missing devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <f9a457c8-f558-4c45-96e0-baa97d143c7b@notapiano>
Date: Thu, 1 Aug 2024 15:15:11 -0400
From: Nícolas F. R. A. Prado <nfraprado@...labora.com>
To: Shuah Khan <skhan@...uxfoundation.org>
Cc: Shuah Khan <shuah@...nel.org>,
	Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
	"Bird, Tim" <Tim.Bird@...y.com>,
	Laura Nao <laura.nao@...labora.com>,
	Saravana Kannan <saravanak@...gle.com>,
	"Rafael J. Wysocki" <rafael@...nel.org>,
	AngeloGioacchino Del Regno <angelogioacchino.delregno@...labora.com>,
	kernel@...labora.com, kernelci@...ts.linux.dev,
	linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org
Subject: Re: [PATCH RFC] kselftest: devices: Add test to detect missing
 devices

On Wed, Jul 31, 2024 at 05:19:45PM -0600, Shuah Khan wrote:
> On 7/24/24 15:40, Nícolas F. R. A. Prado wrote:
> > Introduce a new test to identify regressions causing devices to go
> > missing on the system.
> > 
> > For each bus and class on the system the test checks the number of
> > devices present against a reference file, which needs to have been
> > generated by the program at a previous point on a known-good kernel, and
> > if there are missing devices they are reported.
> 
> Can you elaborate on how to generate reference file? It isn't clear.

Indeed, I'll make that information clearer in future versions.

The reference file is generated by passing the --generate-reference flag to the
test:

./exist.py --generate-reference

It will be printed as standard output.

> 
> > 
> > Signed-off-by: Nícolas F. R. A. Prado <nfraprado@...labora.com>
> > ---
> > Hi,
> > 
> > Key points about this test:
> > * Goal: Identify regressions causing devices to go missing on the system
> > * Focus:
> >    * Ease of maintenance: the reference file is generated programatically
> >    * Minimum of false-positives: the script makes as few assumptions as possible
> >      about the stability of device identifiers to ensure renames/refactors don't
> >      trigger false-positives
> > * How it works: For each bus and class on the system the test checks the number
> >    of devices present against a reference file, which needs to have been
> >    generated by the program at a previous point on a known-good kernel, and if
> >    there are missing devices they are reported.
> > * Comparison to other tests: It might be possible(*) to replace the discoverable
> >    devices test [1] with this. The benefits of this test is that it's easier
> >    to setup and maintain and has wider coverage of devices.
> > 
> > Additional detail:
> > * Having more devices on the running system than the reference does not cause a
> >    failure, but a warning is printed in that case to suggest that the reference
> >    be updated.
> > * Missing devices are detected per bus/class based on the number of devices.
> >    When the test fails, the known metadata for each of the expected and detected
> >    devices is printed and some simple similitarity comparison is done to suggest
> >    the devices that are the most likely to be missing.
> > * The proposed place to store the generated reference files is the
> >    'platform-test-parameters' repository in KernelCI [2].
> 
> How would a user run this on their systems - do they need to access
> this repository in KernelCI?

No, that repository would just be a place where people could find pre-generated
reference files (which we'll be using when running this test in KernelCI), but
anyone can always generate their own reference files and store them wherever
they want.

> 
> This is what I see when I run the test on my system:
> 
> make -C tools/testing/selftests/devices/exist/ run_tests
> make: Entering directory '/linux/linux_6.11/tools/testing/selftests/devices/exist'
> TAP version 13
> 1..1
> # timeout set to 45
> # selftests: devices/exist: exist.py
> # TAP version 13
> # # No matching reference file found (tried './LENOVO,20XH005JUS.yaml')

First generate the reference file for your system like so:

tools/testing/selftests/devices/exist/exist.py --generate-reference > tools/testing/selftests/devices/exist/LENOVO,20XH005JUS.yaml

Then you can run the test and it should work.

Thanks,
Nícolas