Date: Mon, 5 Feb 2024 23:03:50 -0500
From: Rae Moar <rmoar@...gle.com>
To: "Bird, Tim" <Tim.Bird@...y.com>
Cc: "frowand.list@...il.com" <frowand.list@...il.com>, "davidgow@...gle.com" <davidgow@...gle.com>, 
	"keescook@...omium.org" <keescook@...omium.org>, "shuah@...nel.org" <shuah@...nel.org>, 
	"brendanhiggins@...gle.com" <brendanhiggins@...gle.com>, "dlatypov@...gle.com" <dlatypov@...gle.com>, 
	"tytso@...gle.com" <tytso@...gle.com>, 
	"gustavo.padovan@...labora.com" <gustavo.padovan@...labora.com>, 
	"ricardo.canuelo@...labora.com" <ricardo.canuelo@...labora.com>, 
	"guillaume.tucker@...labora.com" <guillaume.tucker@...labora.com>, "corbet@....net" <corbet@....net>, 
	"kernelci@...ts.linux.dev" <kernelci@...ts.linux.dev>, 
	"linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>, 
	"kunit-dev@...glegroups.com" <kunit-dev@...glegroups.com>, 
	"linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>, 
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [KTAP V2 PATCH v2] ktap_v2: add test metadata

On Wed, Jan 31, 2024 at 5:22 PM Bird, Tim <Tim.Bird@...y.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Rae Moar <rmoar@...gle.com>
> > Add specification for test metadata to the KTAP v2 spec.
> >
> > KTAP v1 only specifies the output format of very basic test information:
> > test result and test name. Any additional test information either gets
> > added to general diagnostic data or is not included in the output at all.
> >
> > The purpose of KTAP metadata is to create a framework to include and
> > easily identify additional important test information in KTAP.
> >
> > KTAP metadata could include any test information that is pertinent for
> > user interaction before or after the running of the test. For example,
> > the test file path or the test speed.
> >
> > Since this includes a large variety of information, this specification
> > will recognize notable types of KTAP metadata to ensure consistent format
> > across test frameworks. See the full list of types in the specification.
> >
> > Example of KTAP Metadata:
> >
> >  KTAP version 2
> >  # ktap_test: main
> >  # ktap_arch: uml
> >  1..1
> >      KTAP version 2
> >      # ktap_test: suite_1
> >      # ktap_subsystem: example
> >      # ktap_test_file: lib/test.c
> >      1..2
> >      ok 1 test_1
> >      # ktap_test: test_2
> >      # ktap_speed: very_slow
> >      # custom_is_flaky: true
> >      ok 2 test_2
> >  ok 1 test_suite
> >
> > The changes to the KTAP specification outline the format, location, and
> > different types of metadata.
> >
> > Here is a link to a version of the KUnit parser that is able to parse test
> > metadata lines for KTAP version 2. Note this includes test metadata
> > lines for the main level of KTAP.
> >
> > Link: https://kunit-review.googlesource.com/c/linux/+/5889
> >
> > Signed-off-by: Rae Moar <rmoar@...gle.com>
> > ---
> >  Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++-
> >  1 file changed, 159 insertions(+), 4 deletions(-)
> >
> > diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> > index ff77f4aaa6ef..4480eaf5bbc3 100644
> > --- a/Documentation/dev-tools/ktap.rst
> > +++ b/Documentation/dev-tools/ktap.rst
> > @@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test
> >  can have subtests), each of which can contain both diagnostic data -- e.g., log
> >  lines -- and a final result. The test structure and results are
> >  machine-readable, whereas the diagnostic data is unstructured and is there to
> > -aid human debugging.
> > +aid human debugging. One exception to this is test metadata lines - a type
> > +of diagnostic lines. Test metadata is used to identify important supplemental
> > +test information and can be machine-readable.
> >
> >  KTAP output is built from four different types of lines:
> >  - Version lines
> >  - Plan lines
> >  - Test case result lines
> > -- Diagnostic lines
> > +- Diagnostic lines (including test metadata)
> >
> >  In general, valid KTAP output should also form valid TAP output, but some
> >  information, in particular nested test results, may be lost. Also note that
> >  there is a stagnant draft specification for TAP14, KTAP diverges from this in
> > -a couple of places (notably the "Subtest" header), which are described where
> > -relevant later in this document.
> > +a couple of places, which are described where relevant later in this document.
> >
> >  Version lines
> >  -------------
> > @@ -166,6 +167,154 @@ even if they do not start with a "#": this is to capture any other useful
> >  kernel output which may help debug the test. It is nevertheless recommended
> >  that tests always prefix any diagnostic output they have with a "#" character.
> >
> > +KTAP metadata lines
> > +-------------------
> > +
> > +KTAP metadata lines are a subset of diagnostic lines that are used to include
> > +and easily identify important supplemental test information in KTAP.
> > +
> > +.. code-block:: none
> > +
> > +     # <prefix>_<metadata type>: <metadata value>
> > +
> > +The <prefix> indicates where to find the specification for the type of
> > +metadata. The metadata types listed below use the prefix "ktap" (See Types of
> > +KTAP Metadata).
> > +
> > +Types that are instead specified by an individual test framework use the
> > +framework name as the prefix. For example, a metadata type documented by the
> > +kselftest specification would use the prefix "kselftest". Any metadata type
> > +that is not listed in a specification must use the prefix "custom". Note the
> > +prefix must not include spaces or the characters ":" or "_".
> > +
> > +The format of <metadata type> and <value> varies based on the type. See the
> > +individual specification. For "custom" types the <metadata type> can be any
> > +string excluding ":", spaces, or newline characters and the <value> can be any
> > +string.
> > +
> > +**Location:**
> > +
> > +The first KTAP metadata entry for a test must be "# ktap_test: <test name>",
> > +which acts as a header to associate metadata with the correct test.
> > +
> > +For test cases, the location of the metadata is between the prior test result
> > +line and the current test result line. For test suites, the location of the
> > +metadata is between the suite's version line and test plan line. See the
> > +example below.
> > +
> > +KTAP metadata for a test does not need to be contiguous. For example, a kernel
> > +warning or other diagnostic output could interrupt metadata lines. However, it
> > +is recommended to keep a test's metadata lines together when possible, as this
> > +improves readability.
> > +
> > +**Here is an example of using KTAP metadata:**
> > +
> > +::
> > +
> > +        KTAP version 2
> > +        # ktap_test: main
> > +        # ktap_arch: uml
> > +        1..1
> > +          KTAP version 2
> > +          # ktap_test: suite_1
> > +          # ktap_subsystem: example
> > +          # ktap_test_file: lib/test.c
> > +          1..2
> > +          ok 1 test_1
> > +          # ktap_test: test_2
> > +          # ktap_speed: very_slow
> > +          # custom_is_flaky: true
> > +          ok 2 test_2
> > +        # suite_1 passed
> > +        ok 1 suite_1
> > +
> > +In this example, the tests are running on UML. The test suite "suite_1" is part
> > +of the subsystem "example" and belongs to the file "lib/example_test.c". It has
> Unless I'm missing something, the file is specified in the example above as
> list/test.c, not "lib/example_test.c".
>
> Other than that, the documentation looks fine to me.
>  -- Tim
>

Hello!

Thanks for looking over the documentation. And sorry about that typo
in the explanation. It should say "belongs to the file "lib/test.c"".
I will change that for the next version.

Thanks!
-Rae

>

Powered by blists - more mailing lists