linux-kernel - Re: [PATCH RFC] kselftest: devices: Add test to detect missing devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <9d0b73ce-704c-4633-bb11-06ca4cb7a9a1@linuxfoundation.org>
Date: Wed, 31 Jul 2024 17:19:45 -0600
From: Shuah Khan <skhan@...uxfoundation.org>
To: Nícolas F. R. A. Prado <nfraprado@...labora.com>,
 Shuah Khan <shuah@...nel.org>,
 Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Cc: "Bird, Tim" <Tim.Bird@...y.com>, Laura Nao <laura.nao@...labora.com>,
 Saravana Kannan <saravanak@...gle.com>, "Rafael J. Wysocki"
 <rafael@...nel.org>,
 AngeloGioacchino Del Regno <angelogioacchino.delregno@...labora.com>,
 kernel@...labora.com, kernelci@...ts.linux.dev,
 linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org,
 Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH RFC] kselftest: devices: Add test to detect missing
 devices

On 7/24/24 15:40, Nícolas F. R. A. Prado wrote:
> Introduce a new test to identify regressions causing devices to go
> missing on the system.
> 
> For each bus and class on the system the test checks the number of
> devices present against a reference file, which needs to have been
> generated by the program at a previous point on a known-good kernel, and
> if there are missing devices they are reported.

Can you elaborate on how to generate reference file? It isn't clear.

> 
> Signed-off-by: Nícolas F. R. A. Prado <nfraprado@...labora.com>
> ---
> Hi,
> 
> Key points about this test:
> * Goal: Identify regressions causing devices to go missing on the system
> * Focus:
>    * Ease of maintenance: the reference file is generated programatically
>    * Minimum of false-positives: the script makes as few assumptions as possible
>      about the stability of device identifiers to ensure renames/refactors don't
>      trigger false-positives
> * How it works: For each bus and class on the system the test checks the number
>    of devices present against a reference file, which needs to have been
>    generated by the program at a previous point on a known-good kernel, and if
>    there are missing devices they are reported.
> * Comparison to other tests: It might be possible(*) to replace the discoverable
>    devices test [1] with this. The benefits of this test is that it's easier
>    to setup and maintain and has wider coverage of devices.
> 
> Additional detail:
> * Having more devices on the running system than the reference does not cause a
>    failure, but a warning is printed in that case to suggest that the reference
>    be updated.
> * Missing devices are detected per bus/class based on the number of devices.
>    When the test fails, the known metadata for each of the expected and detected
>    devices is printed and some simple similitarity comparison is done to suggest
>    the devices that are the most likely to be missing.
> * The proposed place to store the generated reference files is the
>    'platform-test-parameters' repository in KernelCI [2].

How would a user run this on their systems - do they need to access
this repository in KernelCI?

This is what I see when I run the test on my system:

make -C tools/testing/selftests/devices/exist/ run_tests
make: Entering directory '/linux/linux_6.11/tools/testing/selftests/devices/exist'
TAP version 13
1..1
# timeout set to 45
# selftests: devices/exist: exist.py
# TAP version 13
# # No matching reference file found (tried './LENOVO,20XH005JUS.yaml')
# # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
not ok 1 selftests: devices/exist: exist.py # exit=1
make: Leaving directory '/linux/linux_6.11/tools/testing/selftests/devices/exist'

thanks,
-- Shuah