Message-ID: <CAJ-ks9nCsCfuayotpqgNytHAZ758w2oSU58dEDMXwQRU7vqisQ@mail.gmail.com>
Date: Mon, 10 Feb 2025 17:01:50 -0500
From: Tamir Duberstein <tamird@...il.com>
To: Yury Norov <yury.norov@...il.com>
Cc: John Hubbard <jhubbard@...dia.com>, Geert Uytterhoeven <geert@...ux-m68k.org>, 
	David Gow <davidgow@...gle.com>, Andrew Morton <akpm@...ux-foundation.org>, 
	Madhavan Srinivasan <maddy@...ux.ibm.com>, Michael Ellerman <mpe@...erman.id.au>, 
	Nicholas Piggin <npiggin@...il.com>, Christophe Leroy <christophe.leroy@...roup.eu>, 
	Naveen N Rao <naveen@...nel.org>, Rasmus Villemoes <linux@...musvillemoes.dk>, 
	Shuah Khan <shuah@...nel.org>, Kees Cook <kees@...nel.org>, 
	Muhammad Usama Anjum <usama.anjum@...labora.com>, linux-kernel@...r.kernel.org, 
	linux-m68k@...ts.linux-m68k.org, linuxppc-dev@...ts.ozlabs.org, 
	linux-kselftest@...r.kernel.org, Brad Figg <bfigg@...dia.com>, 
	David Hildenbrand <david@...hat.com>, Michal Hocko <mhocko@...e.com>, Jason Gunthorpe <jgg@...dia.com>
Subject: Re: distro support for CONFIG_KUNIT: [PATCH 0/3] bitmap: convert
 self-test to KUnit

On Mon, Feb 10, 2025 at 3:20 PM Yury Norov <yury.norov@...il.com> wrote:
>
> On Mon, Feb 10, 2025 at 11:35:48AM -0800, John Hubbard wrote:
> > On 2/9/25 11:54 PM, Geert Uytterhoeven wrote:
> > > On Sat, 8 Feb 2025 at 18:53, Yury Norov <yury.norov@...il.com> wrote:
> > > > On Fri, Feb 07, 2025 at 03:14:01PM -0500, Tamir Duberstein wrote:
> > > > > On 7/27/24 12:35 AM, Shuah Khan wrote:
> > ...
> > > > > The crux of the argument seems to be that the config help text is taken
> > > > > to describe the author's intent with the fragment "at boot". I think
> > >
> > > IMO, "at boot" is a misnomer, as most tests can be either builtin
> > > or modular.
> >
> > Right.
> >
> > >
> > > > KUNIT is disabled in defconfig, at least on x86_64. It is also disabled
> > > > on my Ubuntu 24.04 machine. If I take your patches, I'll be unable to
> >
> > OK so I just bought a shiny new test machine, and installed one of the
> > big name distros on it, hoping they've moved ahead and bought into the kunit
> > story...
> >
> > $ grep KUNIT /boot/config-6.8.0-52-generic
> > # CONFIG_KUNIT is not set
> >
> > ...gagghh! No such luck. One more data point, in support of Yuri's complaint. :)
> >
> > >
> > > I think distros should start setting CONFIG_KUNIT=m.
> >
> > Yes they should! kunit really does have important advantages for many use
> > cases, including bitmaps here, and "CONFIG_KUNIT is not set" is the main
> > obstacle.
>
> Hi John, Geert, Tamir,
>
> Can you please explain in details which advantages KUNIT brings to
> the test_bitmap.c, find_bit_benchmark.c and other low-level tests?

I can try, but I'm not the expert, and David Gow can probably elaborate further.

As I understand it the main benefit of KUnit is standardization and
speed (and standardization _is_ speed). KUnit makes it very easy for
me, a person who has not previously contributed to any of the bitmap
code, to run those tests, and it requires zero configuration, it all
just works. It's basically just `tools/testing/kunit/kunit.py run
bitmap`, and I get the test results in a human-readable format. The
same benefit applies on the author side: test facilities are
standardized, so once you get to know the tools, all the tests start
to look the same: you can jump in and contribute without having to
first learn the so-called local "testing framework".

The important part is that this all applies to ~all other tests
written in KUnit. I can even run them *all* trivially:
`tools/testing/kunit/kunit.py run`.

Anecdotally I've also noticed there are bots running those KUnit tests
e.g. see https://lore.kernel.org/all/20250207-blackholedev-kunit-convert-v1-1-8ef0dc1ff881@gmail.com/
where a test I converted was immediately flagged by a robot as having
dubious type coercion.

None of these are must-haves, they are just (to me) a nice way to make
the kernel more approachable for new contributors.

As for pure benchmarks like
find_bit_benchmark.c and
test_bitmap.c::`test_bitmap_{read,write}_perf` specifically: I believe
the benefits are super limited or even negative: AFAIK KUnit is
designed to generally suppress output (in the userspace reporter, not
in the kernel) unless a test fails, so I wouldn't hurry to use it for
these.

> I'm not strongly against moving under KUNIT's hat, but I do:
>  - respect commitment of my contributors, so I don't want to wipe git
>    history for no serious reason;
>  - respect time of my testers, so no extra dependencies;
>  - respect time of reviewers.

These are valid concerns. Certainly the testing case is the most
compelling and folks are clearly interested in lowering those
barriers. I don't have any influence in this area, but I am grateful
to John for starting the conversation.

As I mentioned in the previous thread: I think we could keep
`test_bitmap_{read,write}_perf` in test_bitmap.c and get the best of
both worlds. WDYT?

> Tamir,
>
> If it comes to v2, can you please begin your series with an exhaustive
> and clear answer to the following questions:
>  - What do the tests miss now?
>  - What do _you_ need from the tests? Describe your test scenario.
>  - How exactly KUNIT helps you testing bitmaps and friends better?
>  - Is there a way to meet your needs with a less invasive approach,
>    particularly without run-time dependencies?

Hopefully I've answered these above. I can include some of it in a v2,
but perhaps the general pitch for KUnit is better placed in
documentation or slides from a conference?

Cheers.

Tamir

Powered by blists - more mailing lists