Message-ID: 
 <SA3PR13MB63728D82B285DD69266AADDEFD7C2@SA3PR13MB6372.namprd13.prod.outlook.com>
Date: Wed, 31 Jan 2024 22:22:33 +0000
From: "Bird, Tim" <Tim.Bird@...y.com>
To: Rae Moar <rmoar@...gle.com>,
        "frowand.list@...il.com"
	<frowand.list@...il.com>,
        "davidgow@...gle.com" <davidgow@...gle.com>,
        "keescook@...omium.org" <keescook@...omium.org>,
        "shuah@...nel.org"
	<shuah@...nel.org>,
        "brendanhiggins@...gle.com" <brendanhiggins@...gle.com>,
        "dlatypov@...gle.com" <dlatypov@...gle.com>
CC: "tytso@...gle.com" <tytso@...gle.com>,
        "gustavo.padovan@...labora.com"
	<gustavo.padovan@...labora.com>,
        "ricardo.canuelo@...labora.com"
	<ricardo.canuelo@...labora.com>,
        "guillaume.tucker@...labora.com"
	<guillaume.tucker@...labora.com>,
        "corbet@....net" <corbet@....net>,
        "kernelci@...ts.linux.dev" <kernelci@...ts.linux.dev>,
        "linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
        "kunit-dev@...glegroups.com" <kunit-dev@...glegroups.com>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: [KTAP V2 PATCH v2] ktap_v2: add test metadata



> -----Original Message-----
> From: Rae Moar <rmoar@...gle.com>
> Add specification for test metadata to the KTAP v2 spec.
> 
> KTAP v1 only specifies the output format of very basic test information:
> test result and test name. Any additional test information either gets
> added to general diagnostic data or is not included in the output at all.
> 
> The purpose of KTAP metadata is to create a framework to include and
> easily identify additional important test information in KTAP.
> 
> KTAP metadata could include any test information that is pertinent for
> user interaction before or after the running of the test. For example,
> the test file path or the test speed.
> 
> Since this includes a large variety of information, this specification
> will recognize notable types of KTAP metadata to ensure consistent format
> across test frameworks. See the full list of types in the specification.
> 
> Example of KTAP Metadata:
> 
>  KTAP version 2
>  # ktap_test: main
>  # ktap_arch: uml
>  1..1
>      KTAP version 2
>      # ktap_test: suite_1
>      # ktap_subsystem: example
>      # ktap_test_file: lib/test.c
>      1..2
>      ok 1 test_1
>      # ktap_test: test_2
>      # ktap_speed: very_slow
>      # custom_is_flaky: true
>      ok 2 test_2
>  ok 1 test_suite
> 
> The changes to the KTAP specification outline the format, location, and
> different types of metadata.
> 
> Here is a link to a version of the KUnit parser that is able to parse test
> metadata lines for KTAP version 2. Note this includes test metadata
> lines for the main level of KTAP.
> 
> Link: https://kunit-review.googlesource.com/c/linux/+/5889
> 
> Signed-off-by: Rae Moar <rmoar@...gle.com>
> ---
>  Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++-
>  1 file changed, 159 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> index ff77f4aaa6ef..4480eaf5bbc3 100644
> --- a/Documentation/dev-tools/ktap.rst
> +++ b/Documentation/dev-tools/ktap.rst
> @@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test
>  can have subtests), each of which can contain both diagnostic data -- e.g., log
>  lines -- and a final result. The test structure and results are
>  machine-readable, whereas the diagnostic data is unstructured and is there to
> -aid human debugging.
> +aid human debugging. One exception to this is test metadata lines - a type
> +of diagnostic lines. Test metadata is used to identify important supplemental
> +test information and can be machine-readable.
> 
>  KTAP output is built from four different types of lines:
>  - Version lines
>  - Plan lines
>  - Test case result lines
> -- Diagnostic lines
> +- Diagnostic lines (including test metadata)
> 
>  In general, valid KTAP output should also form valid TAP output, but some
>  information, in particular nested test results, may be lost. Also note that
>  there is a stagnant draft specification for TAP14, KTAP diverges from this in
> -a couple of places (notably the "Subtest" header), which are described where
> -relevant later in this document.
> +a couple of places, which are described where relevant later in this document.
> 
>  Version lines
>  -------------
> @@ -166,6 +167,154 @@ even if they do not start with a "#": this is to capture any other useful
>  kernel output which may help debug the test. It is nevertheless recommended
>  that tests always prefix any diagnostic output they have with a "#" character.
> 
> +KTAP metadata lines
> +-------------------
> +
> +KTAP metadata lines are a subset of diagnostic lines that are used to include
> +and easily identify important supplemental test information in KTAP.
> +
> +.. code-block:: none
> +
> +	# <prefix>_<metadata type>: <metadata value>
> +
> +The <prefix> indicates where to find the specification for the type of
> +metadata. The metadata types listed below use the prefix "ktap" (See Types of
> +KTAP Metadata).
> +
> +Types that are instead specified by an individual test framework use the
> +framework name as the prefix. For example, a metadata type documented by the
> +kselftest specification would use the prefix "kselftest". Any metadata type
> +that is not listed in a specification must use the prefix "custom". Note the
> +prefix must not include spaces or the characters ":" or "_".
> +
> +The format of <metadata type> and <value> varies based on the type. See the
> +individual specification. For "custom" types the <metadata type> can be any
> +string excluding ":", spaces, or newline characters and the <value> can be any
> +string.
> +
> +**Location:**
> +
> +The first KTAP metadata entry for a test must be "# ktap_test: <test name>",
> +which acts as a header to associate metadata with the correct test.
> +
> +For test cases, the location of the metadata is between the prior test result
> +line and the current test result line. For test suites, the location of the
> +metadata is between the suite's version line and test plan line. See the
> +example below.
> +
> +KTAP metadata for a test does not need to be contiguous. For example, a kernel
> +warning or other diagnostic output could interrupt metadata lines. However, it
> +is recommended to keep a test's metadata lines together when possible, as this
> +improves readability.
> +
> +**Here is an example of using KTAP metadata:**
> +
> +::
> +
> +        KTAP version 2
> +        # ktap_test: main
> +        # ktap_arch: uml
> +        1..1
> +          KTAP version 2
> +          # ktap_test: suite_1
> +          # ktap_subsystem: example
> +          # ktap_test_file: lib/test.c
> +          1..2
> +          ok 1 test_1
> +          # ktap_test: test_2
> +          # ktap_speed: very_slow
> +          # custom_is_flaky: true
> +          ok 2 test_2
> +        # suite_1 passed
> +        ok 1 suite_1
> +
> +In this example, the tests are running on UML. The test suite "suite_1" is part
> +of the subsystem "example" and belongs to the file "lib/example_test.c". It has
Unless I'm missing something, the file is specified in the example above as
list/test.c, not "lib/example_test.c".

Other than that, the documentation looks fine to me.
 -- Tim

Powered by blists - more mailing lists