Message-ID: <20250620145812-e6fc0e34-28bd-4a8a-8376-63c3ac6b6b30@linutronix.de>
Date: Fri, 20 Jun 2025 15:18:20 +0200
From: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
To: David Gow <davidgow@...gle.com>
Cc: Masahiro Yamada <masahiroy@...nel.org>, 
	Nathan Chancellor <nathan@...nel.org>, Andrew Morton <akpm@...ux-foundation.org>, 
	Willy Tarreau <w@....eu>, Thomas Weißschuh <linux@...ssschuh.net>, 
	Brendan Higgins <brendan.higgins@...ux.dev>, Rae Moar <rmoar@...gle.com>, Shuah Khan <shuah@...nel.org>, 
	Jonathan Corbet <corbet@....net>, Nicolas Schier <nicolas.schier@...ux.dev>, 
	Paul Walmsley <paul.walmsley@...ive.com>, Palmer Dabbelt <palmer@...belt.com>, 
	Albert Ou <aou@...s.berkeley.edu>, Alexandre Ghiti <alex@...ti.fr>, 
	Christophe Leroy <christophe.leroy@...roup.eu>, linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com, linux-doc@...r.kernel.org, 
	linux-riscv@...ts.infradead.org, workflows@...r.kernel.org
Subject: Re: [PATCH v3 00/16] kunit: Introduce UAPI testing framework

On Fri, Jun 20, 2025 at 05:37:24PM +0800, David Gow wrote:
> On Wed, 11 Jun 2025 at 15:38, Thomas Weißschuh
> <thomas.weissschuh@...utronix.de> wrote:
> >
> > Currently testing of userspace and in-kernel API use two different
> > frameworks. kselftests for the userspace ones and Kunit for the
> > in-kernel ones. Besides their different scopes, both have different
> > strengths and limitations:
> >
> > Kunit:
> > * Tests are normal kernel code.
> > * They use the regular kernel toolchain.
> > * They can be packaged and distributed as modules conveniently.
> >
> > Kselftests:
> > * Tests are normal userspace code
> > * They need a userspace toolchain.
> >   A kernel cross toolchain is likely not enough.
> > * A fair amout of userland is required to run the tests,
> >   which means a full distro or handcrafted rootfs.
> > * There is no way to conveniently package and run kselftests with a
> >   given kernel image.
> > * The kselftests makefiles are not as powerful as regular kbuild.
> >   For example they are missing proper header dependency tracking or more
> >   complex compiler option modifications.
> >
> > Therefore kunit is much easier to run against different kernel
> > configurations and architectures.
> > This series aims to combine kselftests and kunit, avoiding both their
> > limitations. It works by compiling the userspace kselftests as part of
> > the regular kernel build, embedding them into the kunit kernel or module
> > and executing them from there. If the kernel toolchain is not fit to
> > produce userspace because of a missing libc, the kernel's own nolibc can
> > be used instead.
> > The structured TAP output from the kselftest is integrated into the
> > kunit KTAP output transparently, the kunit parser can parse the combined
> > logs together.
> >
> > Further room for improvements:
> > * Call each test in its completely dedicated namespace
> > * Handle additional test files besides the test executable through
> >   archives. CPIO, cramfs, etc.
> > * Compatibility with kselftest_harness.h (in progress)
> > * Expose the blobs in debugfs
> > * Provide some convience wrappers around compat userprogs
> > * Figure out a migration path/coexistence solution for
> >   kunit UAPI and tools/testing/selftests/
> >
> > Output from the kunit example testcase, note the output of
> > "example_uapi_tests".
> >
> > $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
> > ...
> > Running tests with:
> > $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
> > [11:53:53] ================== example (10 subtests) ===================
> > [11:53:53] [PASSED] example_simple_test
> > [11:53:53] [SKIPPED] example_skip_test
> > [11:53:53] [SKIPPED] example_mark_skipped_test
> > [11:53:53] [PASSED] example_all_expect_macros_test
> > [11:53:53] [PASSED] example_static_stub_test
> > [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
> > [11:53:53] [PASSED] example_priv_test
> > [11:53:53] =================== example_params_test  ===================
> > [11:53:53] [SKIPPED] example value 3
> > [11:53:53] [PASSED] example value 2
> > [11:53:53] [PASSED] example value 1
> > [11:53:53] [SKIPPED] example value 0
> > [11:53:53] =============== [PASSED] example_params_test ===============
> > [11:53:53] [PASSED] example_slow_test
> > [11:53:53] ======================= (4 subtests) =======================
> > [11:53:53] [PASSED] procfs
> > [11:53:53] [PASSED] userspace test 2
> > [11:53:53] [SKIPPED] userspace test 3: some reason
> > [11:53:53] [PASSED] userspace test 4
> > [11:53:53] ================ [PASSED] example_uapi_test ================
> > [11:53:53] ===================== [PASSED] example =====================
> > [11:53:53] ============================================================
> > [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
> > [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running
> >
> > Based on v6.15-rc1.
> >
> > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
> > ---
> > Changes in v3:
> > - Reintroduce CONFIG_CC_CAN_LINK_STATIC
> > - Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC
> > - Properly handle 'clean' target for userprogs
> > - Use ramfs over tmpfs to reduce dependencies
> > - Inherit userprogs byte order and ABI from kernel
> > - Drop now unnecessary "#ifndef NOLIBC"
> > - Pick up review tags
> > - Drop usage of __private in blob.h,
> >   sparse complains and it is not really necessary
> > - Fix execution on loongarch when using clang
> > - Drop userprogs libgcc handling, it was ugly and is not yet necessary
> > - Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@linutronix.de
> >
> > Changes in v2:
> > - Rebase onto v6.15-rc1
> > - Add documentation and kernel docs
> > - Resolve invalid kconfig breakages
> > - Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux"
> > - Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series
> > - Replace patch prefix "kconfig" with "kbuild"
> > - Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest()
> > - Generate private, conflict-free symbols in the blob framework
> > - Handle kselftest exit codes
> > - Handle SIGABRT
> > - Forward output also to kunit debugfs log
> > - Install a fd=0 stdin filedescriptor
> > - Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de
> >
> > ---
> 
> 
> Thanks very much for persisting with this, and sorry for the delayed response.

Thanks for the review.

> I've taken quite a liking to it: it'd definitely have made my life
> easier more than once.
>
> As a more general wrapping of selftests in KUnit tests, I do think
> that there's still some risk of confusion as to when a KUnit UAPI test
> makes sense versus a simple selftest. The UAPI tests are definitely
> (IMO) easier to build and run, but won't be easier to debug, or to run
> on an existing, non-test system as a part of troubleshooting (which
> has been a complaint when selftests have been ported to KUnit in the
> past).

The tests I am currently running with this framework are actually real
kselftests. They primarily live in tools/testing/selftests/ but I have a
wrapper in the "real" source tree and also build as part of KUnit.
This gives the advantages of both systems.
FWIW, the KUnit UAPI tests still exist as regular binaries in the output
tree and can also be used on their own.

> Nevertheless, I'm pretty happy to have this be a part of KUnit, though
> I have three slight reservations:
> 1. There's no real _user_ of this yet -- save for the small test of
> /proc/self/comm in the example -- I'd like to see a real-world test
> using this.

As mentioned before I am using this with real kselftests.
So far I have plugged in:

* all of tools/testing/selftests/vDSO/
* all of tools/testing/selftests/timens/
* a bit of tools/testing/selftests/x86/
* a bit of tools/testing/selftests/timers/

The selftests require a few small changes but these are mostly to resolve
warnings introduced by the more struct userprogs CFLAGS.
Or make the tests compatible with nolibc by changing syscall constants
from the generic SYS_foo to the Linux UAPI variant __NR_foo.

If you want to take a look at the (very WIP) code:
https://git.kernel.org/pub/scm/linux/kernel/git/thomas.weissschuh/linux.git/log/?h=kunit-kselftests-integration

In addition, some extensions to nolibc and the vDSO selftest libraries were
necessary. But all of that has been upstreamed already.

> 2. There's a fair bit of complexity here, and we're already a bit
> behind with KUnit reviews. I'd love it if you could commit to helping
> maintain the KUnit parts of this in MAINTAINERS.

Ack. I'll add this in the next revision.

> 3. We need to make sure that there's a clear understanding of when to
> use this, versus in-kernel KUnit tests, versus kselftest. This'll
> probably involve (a) making sure Shuah is on board -- or at least not
> strongly opposed, and (b) updating
> Documentation/dev-tools/testing-overview.rst.

As mentioned above, I think for most testcases both can be used from the same
codebase. So far I don't have any other requirements, although I am fairly sure
those will come up at some point. Let's go there when necessary.

I talked to Shuah about the proposal before and she had a positive reaction.
But I still would be very happy to get more feedback from her, also about the
posted patches.

I'll also update testing-overview.rst.

> But thanks very much -- it's working well in my testing here, and
> running the tests is very pleasant.

Good to hear!

<snip>

Powered by blists - more mailing lists