linux-kernel - Re: Converting kselftest test modules to kunit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a3083ad4-e9dc-40da-bf57-8391bcd96a6c@collabora.com>
Date: Mon, 29 Jul 2024 12:55:50 +0500
From: Muhammad Usama Anjum <usama.anjum@...labora.com>
To: Shuah Khan <skhan@...uxfoundation.org>, Kees Cook
 <keescook@...omium.org>, davidgow@...gle.com
Cc: Muhammad Usama Anjum <usama.anjum@...labora.com>,
 "open list : KERNEL SELFTEST FRAMEWORK" <linux-kselftest@...r.kernel.org>,
 open list <linux-kernel@...r.kernel.org>, kunit-dev@...glegroups.com,
 "kernel@...labora.com" <kernel@...labora.com>, Shuah Khan <shuah@...nel.org>
Subject: Re: Converting kselftest test modules to kunit

On 7/27/24 12:35 AM, Shuah Khan wrote:
> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
>> Hi Kees and All,
>>
>> There are several tests in kselftest subsystem which load modules to tests
>> the internals of the kernel. Most of these test modules are just loaded by
>> the kselftest, their status isn't read and reported to the user logs. Hence
>> they don't provide benefit of executing those tests.
>>
>> I've found patches from Kees where he has been converting such kselftests
>> to kunit tests [1]. The probable motivation is to move tests output of
>> kselftest subsystem which only triggers tests without correctly reporting
>> the results. On the other hand, kunit is there to test the kernel's
>> internal functions which can't be done by userspace.
>>
>> Kselftest:    Test user facing APIs from userspace
>> Kunit:        Test kernel's internal functions from kernelspace
>>
>> This brings me to conclusion that kselftest which are loading modules to
>> test kernelspace should be converted to kunit tests. I've noted several
>> such kselftests.
>>
>> This is just my understanding. Please mention if I'm correct above or more
>> reasons to support kselftest test modules transformation into kunit test.
>>
>> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
>>
> 
> Please make sure you aren't taking away the ability to run these tests during
> boot. 
The kunit tests are usually run at boot time. They can be run later as
well. So I'm not trying to remove any functionality. Rather adding a way
where failures would actually be detected.

> It doesn't make sense to convert every single test especially when it
> is intended to be run during boot without dependencies - not as a kunit test
> but a regression test during boot.
I started investigating when these lib kselftests were just loading the
test module without checking if test actually passed/failed (which proves
that this type of kselftests can never detect regression as it doesn't
process the results). It was strange. Hence I found out the conversion of
such kselftests to kunit tests done by Kees and started this thread for
discussion before doing actual work and sending patches.

> 
> bitmap is one example - pay attention to the config help test - bitmap
> one clearly states it runs regression testing during boot. Any test that
> says that isn't a candidate for conversion.
Please can you explain what do you mean by bitmap testing during boot? As
far as I understand, the kselftests are usespace tests and are run after
kernel has booted completely and userspace has started. It cannot be
regarded as testing during boot.

> 
> I am going to nack any such conversions.
> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum