| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CABVgOSn+530YJ3OPNJQncLDQNbd9JEDtZ04Amyyxk57jOVYUyQ@mail.gmail.com>
Date: Fri, 20 Jun 2025 17:37:24 +0800
From: David Gow <davidgow@...gle.com>
To: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
Cc: Masahiro Yamada <masahiroy@...nel.org>, Nathan Chancellor <nathan@...nel.org>,
Andrew Morton <akpm@...ux-foundation.org>, Willy Tarreau <w@....eu>,
Thomas Weißschuh <linux@...ssschuh.net>,
Brendan Higgins <brendan.higgins@...ux.dev>, Rae Moar <rmoar@...gle.com>,
Shuah Khan <shuah@...nel.org>, Jonathan Corbet <corbet@....net>,
Nicolas Schier <nicolas.schier@...ux.dev>, Paul Walmsley <paul.walmsley@...ive.com>,
Palmer Dabbelt <palmer@...belt.com>, Albert Ou <aou@...s.berkeley.edu>,
Alexandre Ghiti <alex@...ti.fr>, Christophe Leroy <christophe.leroy@...roup.eu>,
linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com,
linux-doc@...r.kernel.org, linux-riscv@...ts.infradead.org,
workflows@...r.kernel.org
Subject: Re: [PATCH v3 00/16] kunit: Introduce UAPI testing framework
On Wed, 11 Jun 2025 at 15:38, Thomas Weißschuh
<thomas.weissschuh@...utronix.de> wrote:
>
> Currently testing of userspace and in-kernel API use two different
> frameworks. kselftests for the userspace ones and Kunit for the
> in-kernel ones. Besides their different scopes, both have different
> strengths and limitations:
>
> Kunit:
> * Tests are normal kernel code.
> * They use the regular kernel toolchain.
> * They can be packaged and distributed as modules conveniently.
>
> Kselftests:
> * Tests are normal userspace code
> * They need a userspace toolchain.
> A kernel cross toolchain is likely not enough.
> * A fair amout of userland is required to run the tests,
> which means a full distro or handcrafted rootfs.
> * There is no way to conveniently package and run kselftests with a
> given kernel image.
> * The kselftests makefiles are not as powerful as regular kbuild.
> For example they are missing proper header dependency tracking or more
> complex compiler option modifications.
>
> Therefore kunit is much easier to run against different kernel
> configurations and architectures.
> This series aims to combine kselftests and kunit, avoiding both their
> limitations. It works by compiling the userspace kselftests as part of
> the regular kernel build, embedding them into the kunit kernel or module
> and executing them from there. If the kernel toolchain is not fit to
> produce userspace because of a missing libc, the kernel's own nolibc can
> be used instead.
> The structured TAP output from the kselftest is integrated into the
> kunit KTAP output transparently, the kunit parser can parse the combined
> logs together.
>
> Further room for improvements:
> * Call each test in its completely dedicated namespace
> * Handle additional test files besides the test executable through
> archives. CPIO, cramfs, etc.
> * Compatibility with kselftest_harness.h (in progress)
> * Expose the blobs in debugfs
> * Provide some convience wrappers around compat userprogs
> * Figure out a migration path/coexistence solution for
> kunit UAPI and tools/testing/selftests/
>
> Output from the kunit example testcase, note the output of
> "example_uapi_tests".
>
> $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
> ...
> Running tests with:
> $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
> [11:53:53] ================== example (10 subtests) ===================
> [11:53:53] [PASSED] example_simple_test
> [11:53:53] [SKIPPED] example_skip_test
> [11:53:53] [SKIPPED] example_mark_skipped_test
> [11:53:53] [PASSED] example_all_expect_macros_test
> [11:53:53] [PASSED] example_static_stub_test
> [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
> [11:53:53] [PASSED] example_priv_test
> [11:53:53] =================== example_params_test ===================
> [11:53:53] [SKIPPED] example value 3
> [11:53:53] [PASSED] example value 2
> [11:53:53] [PASSED] example value 1
> [11:53:53] [SKIPPED] example value 0
> [11:53:53] =============== [PASSED] example_params_test ===============
> [11:53:53] [PASSED] example_slow_test
> [11:53:53] ======================= (4 subtests) =======================
> [11:53:53] [PASSED] procfs
> [11:53:53] [PASSED] userspace test 2
> [11:53:53] [SKIPPED] userspace test 3: some reason
> [11:53:53] [PASSED] userspace test 4
> [11:53:53] ================ [PASSED] example_uapi_test ================
> [11:53:53] ===================== [PASSED] example =====================
> [11:53:53] ============================================================
> [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
> [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running
>
> Based on v6.15-rc1.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
> ---
> Changes in v3:
> - Reintroduce CONFIG_CC_CAN_LINK_STATIC
> - Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC
> - Properly handle 'clean' target for userprogs
> - Use ramfs over tmpfs to reduce dependencies
> - Inherit userprogs byte order and ABI from kernel
> - Drop now unnecessary "#ifndef NOLIBC"
> - Pick up review tags
> - Drop usage of __private in blob.h,
> sparse complains and it is not really necessary
> - Fix execution on loongarch when using clang
> - Drop userprogs libgcc handling, it was ugly and is not yet necessary
> - Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@linutronix.de
>
> Changes in v2:
> - Rebase onto v6.15-rc1
> - Add documentation and kernel docs
> - Resolve invalid kconfig breakages
> - Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux"
> - Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series
> - Replace patch prefix "kconfig" with "kbuild"
> - Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest()
> - Generate private, conflict-free symbols in the blob framework
> - Handle kselftest exit codes
> - Handle SIGABRT
> - Forward output also to kunit debugfs log
> - Install a fd=0 stdin filedescriptor
> - Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de
>
> ---
Thanks very much for persisting with this, and sorry for the delayed response.
I've taken quite a liking to it: it'd definitely have made my life
easier more than once.
As a more general wrapping of selftests in KUnit tests, I do think
that there's still some risk of confusion as to when a KUnit UAPI test
makes sense versus a simple selftest. The UAPI tests are definitely
(IMO) easier to build and run, but won't be easier to debug, or to run
on an existing, non-test system as a part of troubleshooting (which
has been a complaint when selftests have been ported to KUnit in the
past).
Nevertheless, I'm pretty happy to have this be a part of KUnit, though
I have three slight reservations:
1. There's no real _user_ of this yet -- save for the small test of
/proc/self/comm in the example -- I'd like to see a real-world test
using this.
2. There's a fair bit of complexity here, and we're already a bit
behind with KUnit reviews. I'd love it if you could commit to helping
maintain the KUnit parts of this in MAINTAINERS.
3. We need to make sure that there's a clear understanding of when to
use this, versus in-kernel KUnit tests, versus kselftest. This'll
probably involve (a) making sure Shuah is on board -- or at least not
strongly opposed, and (b) updating
Documentation/dev-tools/testing-overview.rst.
But thanks very much -- it's working well in my testing here, and
running the tests is very pleasant.
Cheers,
-- David
> Thomas Weißschuh (16):
> kbuild: userprogs: avoid duplicating of flags inherited from kernel
> kbuild: userprogs: also inherit byte order and ABI from kernel
> init: re-add CONFIG_CC_CAN_LINK_STATIC
> kbuild: userprogs: add nolibc support
> kbuild: introduce CONFIG_ARCH_HAS_NOLIBC
> kbuild: doc: add label for userprogs section
> kbuild: introduce blob framework
> kunit: tool: Add test for nested test result reporting
> kunit: tool: Don't overwrite test status based on subtest counts
> kunit: tool: Parse skipped tests from kselftest.h
> kunit: Always descend into kunit directory during build
> kunit: qemu_configs: loongarch: Enable LSX/LSAX
> kunit: Introduce UAPI testing framework
> kunit: uapi: Add example for UAPI tests
> kunit: uapi: Introduce preinit executable
> kunit: uapi: Validate usability of /proc
>
> Documentation/dev-tools/kunit/api/index.rst | 5 +
> Documentation/dev-tools/kunit/api/uapi.rst | 12 +
> Documentation/kbuild/makefiles.rst | 38 ++-
> MAINTAINERS | 2 +
> Makefile | 7 +-
> include/kunit/uapi.h | 24 ++
> include/linux/blob.h | 31 +++
> init/Kconfig | 7 +
> lib/Makefile | 4 -
> lib/kunit/Kconfig | 10 +
> lib/kunit/Makefile | 20 +-
> lib/kunit/kunit-example-test.c | 15 ++
> lib/kunit/kunit-example-uapi.c | 54 ++++
> lib/kunit/uapi-preinit.c | 63 +++++
> lib/kunit/uapi.c | 294 +++++++++++++++++++++
> scripts/Makefile.blobs | 19 ++
> scripts/Makefile.build | 6 +
> scripts/Makefile.clean | 2 +-
> scripts/Makefile.userprogs | 13 +-
> scripts/blob-wrap.c | 27 ++
> tools/include/nolibc/Kconfig.nolibc | 15 ++
> tools/testing/kunit/kunit_parser.py | 13 +-
> tools/testing/kunit/kunit_tool_test.py | 9 +
> tools/testing/kunit/qemu_configs/loongarch.py | 2 +
> .../test_is_test_passed-failure-nested.log | 10 +
> .../test_data/test_is_test_passed-kselftest.log | 3 +-
> 26 files changed, 686 insertions(+), 19 deletions(-)
> ---
> base-commit: f07a3558c4a5d76f3fea004075e5151c4516d055
> change-id: 20241015-kunit-kselftests-56273bc40442
>
> Best regards,
> --
> Thomas Weißschuh <thomas.weissschuh@...utronix.de>
>
Download attachment "smime.p7s" of type "application/pkcs7-signature" (5281 bytes)
Powered by blists - more mailing lists