Message-ID: <CABVgOSn5tGDj5rnR=a133ntv3GeoXQLnHRBg9HRf86hWve7T1w@mail.gmail.com>
Date: Tue, 18 Feb 2025 16:20:06 +0800
From: David Gow <davidgow@...gle.com>
To: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
Cc: Masahiro Yamada <masahiroy@...nel.org>, Nathan Chancellor <nathan@...nel.org>, 
	Nicolas Schier <nicolas@...sle.eu>, Andrew Morton <akpm@...ux-foundation.org>, Willy Tarreau <w@....eu>, 
	Thomas Weißschuh <linux@...ssschuh.net>, 
	Brendan Higgins <brendan.higgins@...ux.dev>, Rae Moar <rmoar@...gle.com>, 
	Shuah Khan <shuah@...nel.org>, Jonathan Corbet <corbet@....net>, linux-kbuild@...r.kernel.org, 
	linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org, 
	kunit-dev@...glegroups.com, linux-doc@...r.kernel.org
Subject: Re: [PATCH 00/12] kunit: Introduce UAPI testing framework

On Mon, 17 Feb 2025 at 19:00, Thomas Weißschuh
<thomas.weissschuh@...utronix.de> wrote:
>
> Currently testing of userspace and in-kernel API use two different
> frameworks. kselftests for the userspace ones and Kunit for the
> in-kernel ones. Besides their different scopes, both have different
> strengths and limitations:
>
> Kunit:
> * Tests are normal kernel code.
> * They use the regular kernel toolchain.
> * They can be packaged and distributed as modules conveniently.
>
> Kselftests:
> * Tests are normal userspace code
> * They need a userspace toolchain.
>   A kernel cross toolchain is likely not enough.
> * A fair amout of userland is required to run the tests,
>   which means a full distro or handcrafted rootfs.
> * There is no way to conveniently package and run kselftests with a
>   given kernel image.
> * The kselftests makefiles are not as powerful as regular kbuild.
>   For example they are missing proper header dependency tracking or more
>   complex compiler option modifications.
>
> Therefore kunit is much easier to run against different kernel
> configurations and architectures.
> This series aims to combine kselftests and kunit, avoiding both their
> limitations. It works by compiling the userspace kselftests as part of
> the regular kernel build, embedding them into the kunit kernel or module
> and executing them from there. If the kernel toolchain is not fit to
> produce userspace because of a missing libc, the kernel's own nolibc can
> be used instead.
> The structured TAP output from the kselftest is integrated into the
> kunit KTAP output transparently, the kunit parser can parse the combined
> logs together.

Wow -- this is really neat! Thanks for putting this together.

I haven't had a chance to play with it in detail yet, but here are a
few initial / random thoughts:
- Having support for running things from userspace within a KUnit test
seems like it's something that could be really useful for testing
syscalls (and maybe other mm / exec code as well).
- I don't think we can totally combine kselftests and KUnit for all
tests (some of the selftests definitely require more complicated
dependencies than I think KUnit would want to reasonably support or
require).
- The in-kernel KUnit framework doesn't have any knowledge of the
structure or results of a uapi test. It'd be nice to at least be able
to get the process exit status, and bubble up a basic
'passed'/'skipped'/'failed' so that we're not reporting success for
failed tests (and so that simple test executables could run without
needing to output their own KTAP if they only run one test).
- Equally, for some selftests, it's probably a pain to have to write a
kernel module if there's nothing that needs to be done in the kernel.
Maybe such tests could still be built with nolibc and a kernel
toolchain, but be triggered directly from the python tooling (e.g. as
the 'init' process).
- There still seems to be some increased requirements over plain KUnit
at the moment: I'm definitely seeing issues from not having the right
libgcc installed for all architectures. (Though it's working for most
of them, which is very neat!)
- This is a great example of how having standardised result formats is useful!
- If this is going to change or blur the boundary between "this is a
ksefltest" and "this is a kunit test", we probably will need to update
Documentation/dev-tools/testing-overview.rst -- it probably needs some
clarifications there _anyway_, so this is probably a good point to
ensure everyone's on the same page.

Do you have a particular non-example test you'd like to either write
or port to use this? I think it'd be great to see some real-world
examples of where this'd be most useful.

Either way, I'll keep playing with this a bit over the next few days.
I'd love to hear what Shuah and Rae think, as well, as this involves
kselftest and KTAP a lot.

Cheers,
-- David

>
> Further room for improvements:
> * Call each test in its completely dedicated namespace
> * Handle additional test files besides the test executable through
>   archives. CPIO, cramfs, etc.
> * Compatibility with kselftest_harness.h (in progress)
> * Expose the blobs in debugfs
> * Provide some convience wrappers around compat userprogs
> * Figure out a migration path/coexistence solution for
>   kunit UAPI and tools/testing/selftests/
>
> Output from the kunit example testcase, note the output of
> "example_uapi_tests".
>
> $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
> ...
> Running tests with:
> $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
> [11:53:53] ================== example (10 subtests) ===================
> [11:53:53] [PASSED] example_simple_test
> [11:53:53] [SKIPPED] example_skip_test
> [11:53:53] [SKIPPED] example_mark_skipped_test
> [11:53:53] [PASSED] example_all_expect_macros_test
> [11:53:53] [PASSED] example_static_stub_test
> [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
> [11:53:53] [PASSED] example_priv_test
> [11:53:53] =================== example_params_test  ===================
> [11:53:53] [SKIPPED] example value 3
> [11:53:53] [PASSED] example value 2
> [11:53:53] [PASSED] example value 1
> [11:53:53] [SKIPPED] example value 0
> [11:53:53] =============== [PASSED] example_params_test ===============
> [11:53:53] [PASSED] example_slow_test
> [11:53:53] ======================= (4 subtests) =======================
> [11:53:53] [PASSED] procfs
> [11:53:53] [PASSED] userspace test 2
> [11:53:53] [SKIPPED] userspace test 3: some reason
> [11:53:53] [PASSED] userspace test 4
> [11:53:53] ================ [PASSED] example_uapi_test ================
> [11:53:53] ===================== [PASSED] example =====================
> [11:53:53] ============================================================
> [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
> [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running
>
> Based on v6.14-rc1 and the series
> "tools/nolibc: compatibility with -Wmissing-prototypes" [0].
> For compatibility with LLVM/clang another series is needed [1].
>
> [0] https://lore.kernel.org/lkml/20250123-nolibc-prototype-v1-0-e1afc5c1999a@weissschuh.net/
> [1] https://lore.kernel.org/lkml/20250213-kbuild-userprog-fixes-v1-0-f255fb477d98@linutronix.de/
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
> ---
> Thomas Weißschuh (12):
>       kconfig: implement CONFIG_HEADERS_INSTALL for Usermode Linux
>       kconfig: introduce CONFIG_ARCH_HAS_NOLIBC
>       kbuild: userprogs: respect CONFIG_WERROR
>       kbuild: userprogs: add nolibc support
>       kbuild: introduce blob framework
>       kunit: tool: Add test for nested test result reporting
>       kunit: tool: Don't overwrite test status based on subtest counts
>       kunit: tool: Parse skipped tests from kselftest.h
>       kunit: Introduce UAPI testing framework
>       kunit: uapi: Add example for UAPI tests
>       kunit: uapi: Introduce preinit executable
>       kunit: uapi: Validate usability of /proc
>
>  Documentation/kbuild/makefiles.rst                 |  12 +
>  Makefile                                           |   5 +-
>  include/kunit/uapi.h                               |  17 ++
>  include/linux/blob.h                               |  21 ++
>  init/Kconfig                                       |   2 +
>  lib/Kconfig.debug                                  |   1 -
>  lib/kunit/Kconfig                                  |   9 +
>  lib/kunit/Makefile                                 |  17 +-
>  lib/kunit/kunit-example-test.c                     |  17 ++
>  lib/kunit/kunit-uapi-example.c                     |  58 +++++
>  lib/kunit/uapi-preinit.c                           |  61 +++++
>  lib/kunit/uapi.c                                   | 250 +++++++++++++++++++++
>  scripts/Makefile.blobs                             |  19 ++
>  scripts/Makefile.build                             |   6 +
>  scripts/Makefile.clean                             |   2 +-
>  scripts/Makefile.userprogs                         |  18 +-
>  scripts/blob-wrap.c                                |  27 +++
>  tools/include/nolibc/Kconfig.nolibc                |  18 ++
>  tools/testing/kunit/kunit_parser.py                |  13 +-
>  tools/testing/kunit/kunit_tool_test.py             |   9 +
>  .../test_is_test_passed-failure-nested.log         |  10 +
>  .../test_data/test_is_test_passed-kselftest.log    |   3 +-
>  22 files changed, 584 insertions(+), 11 deletions(-)
> ---
> base-commit: 20e952894066214a80793404c9578d72ef89c5e0
> change-id: 20241015-kunit-kselftests-56273bc40442
>
> Best regards,
> --
> Thomas Weißschuh <thomas.weissschuh@...utronix.de>
>

Download attachment "smime.p7s" of type "application/pkcs7-signature" (5294 bytes)

Powered by blists - more mailing lists