linux-kernel - Improving the testing quality of the kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <02c3926c-84fe-4526-a746-6964f5290c14@collabora.com>
Date: Tue, 23 Apr 2024 20:17:18 +0500
From: Muhammad Usama Anjum <usama.anjum@...labora.com>
To: Shuah Khan <shuah@...nel.org>, gustavo.padovan@...labora.com,
 "open list : KERNEL SELFTEST FRAMEWORK" <linux-kselftest@...r.kernel.org>,
 Laura Nao <laura.nao@...labora.com>,
 Shreeya Patel <shreeya.patel@...labora.com>,
 Nícolas F. R. A. Prado <nfraprado@...labora.com>,
 Helen Mae Koike Fornazier <helen.koike@...labora.com>
Cc: Muhammad Usama Anjum <usama.anjum@...labora.com>,
 automated-testing@...ts.yoctoproject.org,
 "kernel@...labora.com" <kernel@...labora.com>, kernelci@...ts.linux.dev,
 open list <linux-kernel@...r.kernel.org>
Subject: Improving the testing quality of the kernel

Hi,

I've been working with colleagues at Collabora to improve the testing
quality overall for kselftests since several months. We had identified
following key points to improve:
* Make non-TAP conformant tests, TAP conformant to catch the bugs/failures
in the CI and non-CI environment. Without clear success or failure message,
it is difficult and at times impossible to notice the bugs and which part
of the test or sub-test has failed.
* Add config fragment if it is already isn't present.
* Improve ancient tests to be robust. KernelCI helps a lot in finding
flakiness or non-robust pieces of code by building it in range of
configurations and running it on range of hardware.

As new KernelCI is in works, we are identifying which kselftest suites
could be most suitable to be enabled in the start to keep the noise to a
minimum. The criteria to enable a suite on KernelCI is:
* The test suite is TAP compliant
* The test suite passes in preliminary testing on x86_64 and ARM64 platforms
* The test suite builds fine with Clang in-addition to gcc

To facilitate transparency and track our progress, I am diligently
maintaining a spreadsheet [1] with detailed information about each suite's
status and requirements.

Furthermore, I have commenced exploration into KUnit testing, and initial
results are promising. I plan to delve deeper into KUnit testing and will
provide further updates in the coming days.

Your feedback, insights and collaboration on any aspect of these
initiatives would be highly valuable. We greatly appreciate your
collaboration and support as we continue to enhance the testing quality of
kselftests.

[1]
https://docs.google.com/spreadsheets/d/1XRCgxlY1b74aIOIXQ7qDR-62KTJmuV1ow9st4fpeVhA/edit?usp=sharing

-- 
BR,
Muhammad Usama Anjum