Message-ID: <f1a6d13b-a5b8-45c2-bd45-b44b5b70a25d@linuxfoundation.org>
Date: Tue, 30 Jul 2024 16:55:00 -0600
From: Shuah Khan <skhan@...uxfoundation.org>
To: Muhammad Usama Anjum <usama.anjum@...labora.com>,
 Kees Cook <keescook@...omium.org>, davidgow@...gle.com
Cc: "open list : KERNEL SELFTEST FRAMEWORK"
 <linux-kselftest@...r.kernel.org>, open list <linux-kernel@...r.kernel.org>,
 kunit-dev@...glegroups.com, "kernel@...labora.com" <kernel@...labora.com>,
 Shuah Khan <shuah@...nel.org>, Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: Converting kselftest test modules to kunit

On 7/29/24 01:55, Muhammad Usama Anjum wrote:
> On 7/27/24 12:35 AM, Shuah Khan wrote:
>> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
>>> Hi Kees and All,
>>>
>>> There are several tests in kselftest subsystem which load modules to tests
>>> the internals of the kernel. Most of these test modules are just loaded by
>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>> they don't provide benefit of executing those tests.
>>>
>>> I've found patches from Kees where he has been converting such kselftests
>>> to kunit tests [1]. The probable motivation is to move tests output of
>>> kselftest subsystem which only triggers tests without correctly reporting
>>> the results. On the other hand, kunit is there to test the kernel's
>>> internal functions which can't be done by userspace.
>>>
>>> Kselftest:    Test user facing APIs from userspace
>>> Kunit:        Test kernel's internal functions from kernelspace
>>>
>>> This brings me to conclusion that kselftest which are loading modules to
>>> test kernelspace should be converted to kunit tests. I've noted several
>>> such kselftests.
>>>
>>> This is just my understanding. Please mention if I'm correct above or more
>>> reasons to support kselftest test modules transformation into kunit test.
>>>
>>> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
>>>
>>
>> Please make sure you aren't taking away the ability to run these tests during
>> boot.
> The kunit tests are usually run at boot time. They can be run later as
> well. So I'm not trying to remove any functionality. Rather adding a way
> where failures would actually be detected.
> 
>> It doesn't make sense to convert every single test especially when it
>> is intended to be run during boot without dependencies - not as a kunit test
>> but a regression test during boot.
> I started investigating when these lib kselftests were just loading the
> test module without checking if test actually passed/failed (which proves
> that this type of kselftests can never detect regression as it doesn't
> process the results). It was strange. Hence I found out the conversion of
> such kselftests to kunit tests done by Kees and started this thread for
> discussion before doing actual work and sending patches.
> 
>>

I explained this on two other threads. If you don't know how this test
is used why are you making this change?

>> bitmap is one example - pay attention to the config help test - bitmap
>> one clearly states it runs regression testing during boot. Any test that
>> says that isn't a candidate for conversion.
> Please can you explain what do you mean by bitmap testing during boot? As
> far as I understand, the kselftests are usespace tests and are run after
> kernel has booted completely and userspace has started. It cannot be
> regarded as testing during boot.
> 

I responded to David - let me know you would like me to repeat it.

thanks,
-- Shuah

Powered by blists - more mailing lists