Message-ID: <CABVgOS=KZrM2dWyp1HzVS0zh7vquLxmTY2T2Ti53DQADrW+sJg@mail.gmail.com>
Date: Tue, 11 Feb 2025 15:51:54 +0800
From: David Gow <davidgow@...gle.com>
To: Yury Norov <yury.norov@...il.com>
Cc: John Hubbard <jhubbard@...dia.com>, Geert Uytterhoeven <geert@...ux-m68k.org>, 
	Tamir Duberstein <tamird@...il.com>, Andrew Morton <akpm@...ux-foundation.org>, 
	Madhavan Srinivasan <maddy@...ux.ibm.com>, Michael Ellerman <mpe@...erman.id.au>, 
	Nicholas Piggin <npiggin@...il.com>, Christophe Leroy <christophe.leroy@...roup.eu>, 
	Naveen N Rao <naveen@...nel.org>, Rasmus Villemoes <linux@...musvillemoes.dk>, 
	Shuah Khan <shuah@...nel.org>, Kees Cook <kees@...nel.org>, 
	Muhammad Usama Anjum <usama.anjum@...labora.com>, linux-kernel@...r.kernel.org, 
	linux-m68k@...ts.linux-m68k.org, linuxppc-dev@...ts.ozlabs.org, 
	linux-kselftest@...r.kernel.org, Brad Figg <bfigg@...dia.com>, 
	David Hildenbrand <david@...hat.com>, Michal Hocko <mhocko@...e.com>, Jason Gunthorpe <jgg@...dia.com>
Subject: Re: distro support for CONFIG_KUNIT: [PATCH 0/3] bitmap: convert
 self-test to KUnit

On Tue, 11 Feb 2025 at 04:20, Yury Norov <yury.norov@...il.com> wrote:
>
> On Mon, Feb 10, 2025 at 11:35:48AM -0800, John Hubbard wrote:
> > On 2/9/25 11:54 PM, Geert Uytterhoeven wrote:
> > > On Sat, 8 Feb 2025 at 18:53, Yury Norov <yury.norov@...il.com> wrote:
> > > > On Fri, Feb 07, 2025 at 03:14:01PM -0500, Tamir Duberstein wrote:
> > > > > On 7/27/24 12:35 AM, Shuah Khan wrote:
> > ...
> > > > > The crux of the argument seems to be that the config help text is taken
> > > > > to describe the author's intent with the fragment "at boot". I think
> > >
> > > IMO, "at boot" is a misnomer, as most tests can be either builtin
> > > or modular.
> >
> > Right.
> >
> > >
> > > > KUNIT is disabled in defconfig, at least on x86_64. It is also disabled
> > > > on my Ubuntu 24.04 machine. If I take your patches, I'll be unable to
> >
> > OK so I just bought a shiny new test machine, and installed one of the
> > big name distros on it, hoping they've moved ahead and bought into the kunit
> > story...
> >
> > $ grep KUNIT /boot/config-6.8.0-52-generic
> > # CONFIG_KUNIT is not set
> >
> > ...gagghh! No such luck. One more data point, in support of Yuri's complaint. :)
> >
> > >
> > > I think distros should start setting CONFIG_KUNIT=m.
> >
> > Yes they should! kunit really does have important advantages for many use
> > cases, including bitmaps here, and "CONFIG_KUNIT is not set" is the main
> > obstacle.
>
> Hi John, Geert, Tamir,
>
> Can you please explain in details which advantages KUNIT brings to
> the test_bitmap.c, find_bit_benchmark.c and other low-level tests?
>

First, I'd agree with Tamir and others who've pointed out that KUnit
is not really intended for benchmarks, so I definitely don't want to
oversell it for find_bit_benchmark.c and others (though it has been
used for benchmarks in a few cases, it's not perfect).

The main advantages of KUnit are that:
- it's already used for a bunch of lib/ tests (so quite familiar to a
number of contributors/users),
- tests can be run at boot without needing any userspace at all (if built-in),
- KUnit tests are already being automatically run as regression tests,
and it's relatively easy to run "all tests" associated with a config
- there's a whole bunch of tooling which automates configuring,
building, and running a kernel with these tests, across several
architectures
  (e.g., passing --arch arm64 or --arch m68k to automatically build
and run the tests in QEMU for those architectures)
- there's tooling to nicely print, format, and summarise the results
of any number of suites.
- and, as a result of the above, it's very quick and easy to run a
test (or whole bunch of tests) in an isolated environment.

KUnit also has very few runtime dependencies from a developer
point-of-view (python, optionally QEMU).

It is, however, less focused on the "debug a running system" case,
hence the requirement for CONFIG_KUNIT to be enabled, and the tooling
largely being focused on the "run all of the tests in UML or a VM"
case.

> I'm not strongly against moving under KUNIT's hat, but I do:
>  - respect commitment of my contributors, so I don't want to wipe git
>    history for no serious reason;

It should be possible to preserve the git history during a port,
though it admittedly will have a bunch of porting changes which make
it more difficult to navigate with git blame. But it should at least
be better than a total wipe.

>  - respect time of my testers, so no extra dependencies;

I think KUnit is a bit of a mixed bag here. In many cases, KUnit tests
can be run pretty quickly, either because the tester's environment
already has CONFIG_KUNIT enabled, or possibly under some kind of
virtualization with the tooling (either QEMU, or on x86,
User-Mode-Linux). But, as mentioned above, I'd not consider it ideal
for performance testing.

I am very sympathetic to not wanting to add a dependency here. I think
KUnit is a pretty minimal dependency as far as they go (it selects
CONFIG_GLOB and nothing else, needs no userspace, and even the tooling
only really requires python on top of the basic toolchain), but it's
definitely a further dependency for the
'test-the-currently-running-kernel' use-case.

>  - respect time of reviewers.

This is an area where I think KUnit could be an advantage, as it'd be
easy for reviewers to simply run
./tools/testing/kunit/kunit.py run bitmap
as a spot check on the code. And if they want to test on other
architectures, the --arch option can be used to do so quickly.
(And, of course, it can still be loaded as a module (or even built-in)
on a physical system or other VM, albeit with the requirement for
CONFIG_KUNIT.)



As for distros enabling CONFIG_KUNIT by default, as mentioned, my
understanding is that the Fedora/Red Hat distros are doing it, as are
some branches of Android, but I doubt anyone else is. This makes
sense: enabling CONFIG_KUNIT isn't free (though the overhead should be
very low if no test is running, just a few static branches and an
extra check in the module loader), and there's always been some
reticence (which I've shared) in enabling test features on production
kernels. So it's definitely something I was recommending against a few
years ago.

However, we've definitely made sure that CONFIG_KUNIT is less
invasive, with things like hooks being kept behind static branches,
and made the decision to have _running_ a test taint the kernel, so it
should be relatively safe to just have the framework enabled (and even
safe to have it loaded), and pretty obvious when a test is run. We've
also got a bunch of extra options to disable running tests
automatically when they're built-in/loaded, both at runtime and
compile time.

So I'd definitely be happy with other distros choosing to enable
CONFIG_KUNIT by default now: I think it's much safer than it was, and
you won't be going against the grain. But equally, I'd understand
having it disabled in production: the cost, while very small, is
nonzero.

Cheers,
-- David

Powered by blists - more mailing lists