Message-ID: <714e7642-6f92-4e41-aa36-c854668e0bb0@linuxfoundation.org>
Date: Tue, 30 Jul 2024 09:55:22 -0600
From: Shuah Khan <skhan@...uxfoundation.org>
To: David Gow <davidgow@...gle.com>, Randy Dunlap <rdunlap@...radead.org>,
 kees@...nel.org, Muhammad Usama Anjum <usama.anjum@...labora.com>
Cc: Yury Norov <yury.norov@...il.com>,
 Andrew Morton <akpm@...ux-foundation.org>,
 Rasmus Villemoes <linux@...musvillemoes.dk>, Shuah Khan <shuah@...nel.org>,
 linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org,
 John Hubbard <jhubbard@...dia.com>, kernel@...labora.com,
 Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH 2/3] bitmap: Rename module

On 7/30/24 04:10, David Gow wrote:
> On Mon, 29 Jul 2024 at 22:09, Randy Dunlap <rdunlap@...radead.org> wrote:
>>
>>
>>
>> On 7/29/24 1:07 AM, Muhammad Usama Anjum wrote:
>>> On 7/27/24 10:35 PM, Yury Norov wrote:
>>>> On Fri, Jul 26, 2024 at 04:06:57PM +0500, Muhammad Usama Anjum wrote:
>>>>> Rename module to bitmap_kunit and rename the configuration option
>>>>> compliant with kunit framework.
>>>>
>>>> ... , so those enabling bitmaps testing in their configs by setting
>>>> "CONFIG_TEST_BITMAP=y" will suddenly get it broken, and will likely
>>>> not realize it until something nasty will happen.
>>> CONFIG_TEST_BITMAP was being enabled by the kselftest suite lib. The bitmap
>>> test and its config option would disappear. The same test can be run by
>>> just enabling KUNIT default config option:
>>>
>>> KUNIT_ALL_TESTS=y enables this bitmap config by default.
>>>
>>>>
>>>> Sorry, NAK for config rename.
>>>>
>>
>> I agree with Yury. Using KUNIT takes away test coverage for people who
>> are willing to run selftests but not use KUNIT.

This is incorrect. There are selftest that are used to

- regression test a subsystem or a abi during boot
- spot check on a running system to debug and test by loading
   a test module.

> 
> I can see the point that renaming the config option is just churn, but
> is there a reason people would run the bitmap selftest but be unable
> or unwilling to use KUnit?
> 
> Beyond a brief period of adjustment (which could probably be made
> quite minimal with a wrapper script or something), there shouldn't
> really be any fundamental difference: KUnit tests can already run at
> boot, be configured with a config option, and write output to the
> kernel log. There's nothing really being taken away here, and the
> bonus of having easier access to run the tests with KUnit's tooling
> (or have them automatically run by systems which run KUnit tests)
> would seem worthwhile to me, especially since it's optional. And
> CONFIG_KUNIT shouldn't be heavy enough to cause problems.
> 
> Obviously I can't force people to use KUnit, but this is exactly the
> sort of test which would fit KUnit well, and -- forgive me if I'm
> missing something -- the only real argument against it I'm hearing is
> "it's different". And while that's valid (as I said, churn for churn's
> sake isn't great), none of the "people who are willing to run
> selftests but not use KUnit" have given reasons why. Especially since
> this is the sort of test (testing in-kernel functions) we're
> encouraging people to write with KUnit in
> Documentation/dev-tools/testing-overview.rst -- if there are good
> reasons people are refusing to run these, maybe we need to fix those
> or change the recommendation.

It isn't about kunit vs. kselftest. It is about overall kernel validation
and ability to test effectively by developers and users.

KUnit isn't ideal for cases where people would want to check a subsystem
on a running kernel - KUnit covers some use-cases and kselftest covers
others.

What happens if we are debugging a problem that requires us to debug on
a running system? Please don't go converting kselftest into kunit without
understanding how these frameworks are intended to be used.

Yes kselftest results need to be looked at. Write a parser which can
be improved. What you are doing is reducing the coverage and talking
away the ability to debug and test on running system.

Fix what needs to be fixed instead of deleting tests.

Think through the use-cases before advocating KUnit is suitable for
all testing needs and talking about being able to force or not force
people to use one or the other.

Reports aren't everything. The primary reason we have these tests is for
developers to be able to test. Reports can be improved and shouldn't
come at the expense of coverage and testing. Any patch that does that
will be NACKed.

thanks,
-- Shuah

Powered by blists - more mailing lists