| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20250523-winged-donkey-of-advance-fffac7@l-nschier-aarch64> Date: Mon, 26 May 2025 16:19:01 +0200 From: Nicolas Schier <nicolas.schier@...ux.dev> To: Thomas Weißschuh <thomas.weissschuh@...utronix.de> Cc: Masahiro Yamada <masahiroy@...nel.org>, Nathan Chancellor <nathan@...nel.org>, Andrew Morton <akpm@...ux-foundation.org>, Willy Tarreau <w@....eu>, Thomas Weißschuh <linux@...ssschuh.net>, Brendan Higgins <brendan.higgins@...ux.dev>, David Gow <davidgow@...gle.com>, Rae Moar <rmoar@...gle.com>, Shuah Khan <shuah@...nel.org>, Jonathan Corbet <corbet@....net>, Christophe Leroy <christophe.leroy@...roup.eu>, linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org, linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com, linux-doc@...r.kernel.org Subject: Re: [PATCH v2 00/11] kunit: Introduce UAPI testing framework On Mon, Apr 07, 2025 at 09:42:37AM +0200, Thomas Weißschuh wrote: > Currently testing of userspace and in-kernel API use two different > frameworks. kselftests for the userspace ones and Kunit for the > in-kernel ones. Besides their different scopes, both have different > strengths and limitations: > > Kunit: > * Tests are normal kernel code. > * They use the regular kernel toolchain. > * They can be packaged and distributed as modules conveniently. > > Kselftests: > * Tests are normal userspace code > * They need a userspace toolchain. > A kernel cross toolchain is likely not enough. > * A fair amout of userland is required to run the tests, > which means a full distro or handcrafted rootfs. > * There is no way to conveniently package and run kselftests with a > given kernel image. > * The kselftests makefiles are not as powerful as regular kbuild. > For example they are missing proper header dependency tracking or more > complex compiler option modifications. > > Therefore kunit is much easier to run against different kernel > configurations and architectures. > This series aims to combine kselftests and kunit, avoiding both their > limitations. It works by compiling the userspace kselftests as part of > the regular kernel build, embedding them into the kunit kernel or module > and executing them from there. If the kernel toolchain is not fit to > produce userspace because of a missing libc, the kernel's own nolibc can > be used instead. > The structured TAP output from the kselftest is integrated into the > kunit KTAP output transparently, the kunit parser can parse the combined > logs together. > > Further room for improvements: > * Call each test in its completely dedicated namespace > * Handle additional test files besides the test executable through > archives. CPIO, cramfs, etc. > * Compatibility with kselftest_harness.h (in progress) > * Expose the blobs in debugfs > * Provide some convience wrappers around compat userprogs > * Figure out a migration path/coexistence solution for > kunit UAPI and tools/testing/selftests/ > > Output from the kunit example testcase, note the output of > "example_uapi_tests". > > $ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example > ... > Running tests with: > $ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt > [11:53:53] ================== example (10 subtests) =================== > [11:53:53] [PASSED] example_simple_test > [11:53:53] [SKIPPED] example_skip_test > [11:53:53] [SKIPPED] example_mark_skipped_test > [11:53:53] [PASSED] example_all_expect_macros_test > [11:53:53] [PASSED] example_static_stub_test > [11:53:53] [PASSED] example_static_stub_using_fn_ptr_test > [11:53:53] [PASSED] example_priv_test > [11:53:53] =================== example_params_test =================== > [11:53:53] [SKIPPED] example value 3 > [11:53:53] [PASSED] example value 2 > [11:53:53] [PASSED] example value 1 > [11:53:53] [SKIPPED] example value 0 > [11:53:53] =============== [PASSED] example_params_test =============== > [11:53:53] [PASSED] example_slow_test > [11:53:53] ======================= (4 subtests) ======================= > [11:53:53] [PASSED] procfs > [11:53:53] [PASSED] userspace test 2 > [11:53:53] [SKIPPED] userspace test 3: some reason > [11:53:53] [PASSED] userspace test 4 > [11:53:53] ================ [PASSED] example_uapi_test ================ > [11:53:53] ===================== [PASSED] example ===================== > [11:53:53] ============================================================ > [11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5 > [11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running > > Based on v6.15-rc1. > > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@...utronix.de> > --- > Changes in v2: > - Rebase onto v6.15-rc1 > - Add documentation and kernel docs > - Resolve invalid kconfig breakages > - Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux" > - Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series > - Replace patch prefix "kconfig" with "kbuild" > - Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest() > - Generate private, conflict-free symbols in the blob framework > - Handle kselftest exit codes > - Handle SIGABRT > - Forward output also to kunit debugfs log > - Install a fd=0 stdin filedescriptor > - Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de > > --- > Thomas Weißschuh (11): > kbuild: userprogs: add nolibc support > kbuild: introduce CONFIG_ARCH_HAS_NOLIBC > kbuild: doc: add label for userprogs section > kbuild: introduce blob framework > kunit: tool: Add test for nested test result reporting > kunit: tool: Don't overwrite test status based on subtest counts > kunit: tool: Parse skipped tests from kselftest.h > kunit: Introduce UAPI testing framework > kunit: uapi: Add example for UAPI tests > kunit: uapi: Introduce preinit executable > kunit: uapi: Validate usability of /proc > > Documentation/dev-tools/kunit/api/index.rst | 5 + > Documentation/dev-tools/kunit/api/uapi.rst | 12 + > Documentation/kbuild/makefiles.rst | 37 ++- > MAINTAINERS | 2 + > include/kunit/uapi.h | 24 ++ > include/linux/blob.h | 32 +++ > init/Kconfig | 2 + > lib/kunit/.kunitconfig | 2 + > lib/kunit/Kconfig | 11 + > lib/kunit/Makefile | 18 +- > lib/kunit/kunit-example-test.c | 15 ++ > lib/kunit/kunit-example-uapi.c | 56 ++++ > lib/kunit/uapi-preinit.c | 65 +++++ > lib/kunit/uapi.c | 294 +++++++++++++++++++++ > scripts/Makefile.blobs | 19 ++ > scripts/Makefile.build | 6 + > scripts/Makefile.clean | 2 +- > scripts/Makefile.userprogs | 16 +- > scripts/blob-wrap.c | 27 ++ > tools/include/nolibc/Kconfig.nolibc | 13 + > tools/testing/kunit/kunit_parser.py | 13 +- > tools/testing/kunit/kunit_tool_test.py | 9 + > .../test_is_test_passed-failure-nested.log | 10 + > .../test_data/test_is_test_passed-kselftest.log | 3 +- > 24 files changed, 682 insertions(+), 11 deletions(-) > --- > base-commit: bf9962cc9ec3ac1dae2bf81b126657c1c49c348a > change-id: 20241015-kunit-kselftests-56273bc40442 > > Best regards, > -- > Thomas Weißschuh <thomas.weissschuh@...utronix.de> > Hi Thomas, sorry for the long delay. I started reviewing but am not completely through the whole set (especially, I did not really look at the kunit patches, yet) but would like to send you at least some feedback already. In general, I really like the idea and your approach and am looking forward for an integration. Kind regards, Nicolas
Powered by blists - more mailing lists