Date: Tue, 2 Jul 2024 16:36:11 -0400
From: Rae Moar <rmoar@...gle.com>
To: Frank Rowand <frowand.list@...il.com>
Cc: David Gow <davidgow@...gle.com>, Shuah Khan <skhan@...uxfoundation.org>, 
	Kees Cook <keescook@...omium.org>, Tim.Bird@...y.com, 
	Brendan Higgins <brendanhiggins@...gle.com>, Jonathan Corbet <corbet@....net>, rmr167@...il.com, 
	guillaume.tucker@...labora.com, dlatypov@...gle.com, kernelci@...ups.io, 
	kunit-dev@...glegroups.com, linux-kselftest@...r.kernel.org, 
	linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [RFC] ktap_v2: KTAP specification transition method

Hello everyone,

It has been a few months since there has been activity regarding the
second version of KTAP. I wanted to bring this topic back up to the
surface.

Currently, Frank has compiled a list of KTAPv2 patches here at this
link: https://elinux.org/Test_Results_Format_Notes#KTAP_version_2

I am interested in starting the process of accepting these patches as
changes to the KTAP documentation as KTAPv2.

First, a decision that needs to be made is which branch should be used
to accept these changes. Frank has a git repository for KTAPv2.
However, it is my understanding he has retired. As a reviewer of
KUnit, I am happy to take the patches in through the KUnit branch.
Would this work for everyone?

Second, we need to finalize the changes. A current list of proposed
KTAPv2 patches is as follows:

[PATCH v3 0/2] begin KTAP spec v2 process
[PATCH v3 1/2] ktap_v2: change version to 2-rc in KTAP specification
[PATCH v3 2/2] ktap_v2: change "version 1" to "version 2" in examples
[KTAP V2 PATCH] ktap_v2: add skip test result
[KTAP V2 PATCH v4] ktap_v2: add test metadata

Note the patch on adding a skip test result has not yet been reviewed.
So please take a look at this change if interested.

As a rule, for any feature of KTAPv1 that is replaced in KTAPv2, it
will stay in the documentation as allowed but deprecated to allow a
smooth transition.

If this process sounds good to people, I will try to get discussions
going on current patches and ask for reviews. My goal is by the end of
the summer, I can apply the approved patches and send them through a
chosen branch (potentially KUnit, as discussed above).

Let me know what you think. Thanks!
-Rae


On Sun, Mar 26, 2023 at 7:25 PM Frank Rowand <frowand.list@...il.com> wrote:
>
> In the middle of the thread about a patch to add the skip test result,
> I suggested documenting the process of deprecating the KTAP v1 Specification
> method of marking a skipped test:
>
>   https://lore.kernel.org/all/490271eb-1429-2217-6e38-837c6e5e328b@gmail.com/T/#u
>
> In a reply to that email I suggested that we ought to have a process to transition
> the KTAP Specification from v1 to v2, and possibly v3 and future.
>
> This email is meant to be the root of that discussion.
>
> My initial thinking is that there are at least three different types of project
> and/or community that may have different needs in this area.
>
> Type 1 - project controls both the test output generation and the test output
> parsing tool.  Both generation and parsing code are in the same repository
> and/or synchronized versions are distributed together.
>
> Devicetree unittests are an example of Type 1.  I plan to maintain changes
> of test output to KTAP v2 format in coordination with updating the parser
> to process KTAP v2 data.
>
> Type 2 - project controls both the test output generation and the test output
> parsing tool.  The test output generation and a parser modifications may be
> controlled by the project BUT there are one or more external testing projects
> that (1) may have their own parsers, and (2) may have a single framework that
> tests multiple versions of the tests.
>
> I think that kselftest and kunit tests are probably examples of Type 2.  I also
> think that DT unittests will become a Type 2 project as a result of converting
> to KTAP v2 data.
>
> Type 3 - project may create and maintain some tests, but is primarily a consumer
> of tests created by other projects.  Type 3 projects typically have a single
> framework that is able to execute and process multiple versions of the tests.
>
> The Fuego test project is an example of Type 3.
>
> Maybe adding all of this complexity of different Types in my initial thinking
> was silly -- maybe everything in this topic is governed by the more complex
> Type 3.
>
> My thinking was that the three different Types of project would be impacted
> in different ways by transition plans.  Type 3 would be the most impacted,
> so I wanted to be sure that any transition plan especially considered their
> needs.
>
> There is an important aspect of the KTAP format that might ease the transition
> from one version to another: All KTAP formatted results begin with a "version
> line", so as soon as a parser has processed the first line of a test, it can
> apply the appropriate KTAP Specification version to all subsequent lines of
> test output.  A parser implementation could choose to process all versions,
> could choose to invoke a version specific parser, or some other approach
> all together.
>
> In the "add skip test results" thread, I suggested deprecating the v1
> method of marking a skipped test in v2, with a scheduled removal of
> the v1 method in v3.  But since the KTAP format version is available
> in the very first line of test output, is it necessary to do a slow
> deprecation and removal over two versions?
>
> One argument to doing a two version deprecation/removal process is that
> a parser that is one version older the the test output _might_ be able
> to process the test output without error, but would not be able to take
> advantage of features added in the newer version of the Specification.
>
> My opinion is that a two version deprecation/removal process will slow
> the Specification update process and lead to more versions of the
> Specification over a given time interval.
>
> A one version deprecation/removal process puts more of a burden on Type 3
> projects and external parsers for Type 2 projects to implement parsers
> that can process the newer Specification more quickly and puts a burden
> on test maintainers to delay a move to the newer Specification, or possibly
> pressure to support selection of more than one Specification version format
> for output data.
>
> One additional item...  On the KTAP Specification version 2 process wiki page,
> I suggested that it is "desirable for test result parsers that understand the
> KTAP Specification version 2 data also be able to parse version 1 data."
> With the implication "Converting version 1 compliant data to version 2 compliant
> data should not require a "flag day" switch of test result parsers."  If this
> thread discussion results in a different decision, I will update the wiki.
>
> Thoughts?
>
> -Frank
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@...glegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/6d4afb49-3cb9-f176-61a2-5bbaab698644%40gmail.com.

Powered by blists - more mailing lists