Message-Id: <20191216220555.245089-1-brendanhiggins@google.com>
Date:   Mon, 16 Dec 2019 14:05:49 -0800
From:   Brendan Higgins <brendanhiggins@...gle.com>
To:     jdike@...toit.com, richard@....at, anton.ivanov@...bridgegreys.com,
        arnd@...db.de, keescook@...omium.org, skhan@...uxfoundation.org,
        alan.maguire@...cle.com, yzaikin@...gle.com, davidgow@...gle.com,
        akpm@...ux-foundation.org, rppt@...ux.ibm.com
Cc:     gregkh@...uxfoundation.org, sboyd@...nel.org, logang@...tatee.com,
        mcgrof@...nel.org, knut.omang@...cle.com,
        linux-um@...ts.infradead.org, linux-arch@...r.kernel.org,
        linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com,
        linux-kernel@...r.kernel.org,
        Brendan Higgins <brendanhiggins@...gle.com>
Subject: [RFC v1 0/6] kunit: create a centralized executor to dispatch all
 KUnit tests

## TL;DR

This patchset adds a centralized executor to dispatch tests rather than
relying on late_initcall to schedule each test suite separately along
with a couple of new features that depend on it.

## What am I trying to do?

Conceptually, I am trying to provide a mechanism by which test suites
can be grouped together so that they can be reasoned about collectively.
The last two patches in this series add features which depend on this:

RFC 5/6 Prints out a test plan right before KUnit tests are run[1]; this
        is valuable because it makes it possible for a test harness to
        detect whether the number of tests run matches the number of
        tests expected to be run, ensuring that no tests silently
        failed.

RFC 6/6 Add a new kernel command-line option which allows the user to
        specify that the kernel poweroff, halt, or reboot after
        completing all KUnit tests; this is very handy for running KUnit
        tests on UML or a VM so that the UML/VM process exits cleanly
        immediately after running all tests without needing a special
        initramfs.

In addition, by dispatching tests from a single location, we can
guarantee that all KUnit tests run after late_init is complete, which
was a concern during the initial KUnit patchset review (this has not
been a problem in practice, but resolving with certainty is nevertheless
desirable).

Other use cases for this exist, but the above features should provide an
idea of the value that this could provide.

## What work remains to be done?

These patches were based on patches in our non-upstream branch[2], so we
have a pretty good idea that they are useable as presented;
nevertheless, some of the changes done in this patchset could
*definitely* use some review by subsystem experts (linker scripts, init,
etc), and will likely change a lot after getting feedback.

The biggest thing that I know will require additional attention is
integrating this patchset with the KUnit module support patchset[3]. I
have not even attempted to build these patches on top of the module
support patches as I would like to get people's initial thoughts first
(especially Alan's :-) ). I think that making these patches work with
module support should be fairly straight forward, nevertheless.

Brendan Higgins (5):
  vmlinux.lds.h: add linker section for KUnit test suites
  arch: um: add linker section for KUnit test suites
  kunit: test: create a single centralized executor for all tests
  init: main: add KUnit to kernel init
  kunit: test: add test plan to KUnit TAP format

David Gow (1):
  kunit: Add 'kunit_shutdown' option

 arch/um/include/asm/common.lds.S              |  4 +
 include/asm-generic/vmlinux.lds.h             |  8 ++
 include/kunit/test.h                          | 16 ++--
 init/main.c                                   |  4 +
 lib/kunit/Makefile                            |  3 +-
 lib/kunit/executor.c                          | 74 ++++++++++++++++++
 lib/kunit/test.c                              | 11 ---
 tools/testing/kunit/kunit_kernel.py           |  2 +-
 tools/testing/kunit/kunit_parser.py           | 76 +++++++++++++++----
 .../test_is_test_passed-all_passed.log        |  1 +
 .../test_data/test_is_test_passed-crash.log   |  1 +
 .../test_data/test_is_test_passed-failure.log |  1 +
 12 files changed, 170 insertions(+), 31 deletions(-)
 create mode 100644 lib/kunit/executor.c

[1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
[2]: https://kunit-review.googlesource.com/c/linux/+/1037
[3]: https://patchwork.kernel.org/project/linux-kselftest/list/?series=211727

-- 
2.24.1.735.g03f4e72817-goog

Powered by blists - more mailing lists