linux-kernel - Re: RFC - kernel selftest result documentation (KTAP)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <19617b29-0298-5caf-39f2-ab08a7fddb82@gmail.com>
Date:   Fri, 19 Jun 2020 13:52:55 -0500
From:   Frank Rowand <frowand.list@...il.com>
To:     Kees Cook <keescook@...omium.org>, "Bird, Tim" <Tim.Bird@...y.com>
Cc:     "shuah@...nel.org" <shuah@...nel.org>,
        "linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
        Brendan Higgins <brendanhiggins@...gle.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        David Gow <davidgow@...gle.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        Frank Rowand <frowand.list@...il.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)

On 2020-06-16 18:52, Kees Cook wrote:
> On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote:
>> From: Kees Cook <keescook@...omium.org>
>>> Note: making the plan line required differs from TAP13 and TAP14. I
>>> think it's the right choice, but we should be clear.
>>
>> [...]
>> With regards to making it optional or not, I don't have a strong
>> preference.  The extra info seems helpful in some circumstances.
>> I don't know if it's too onerous to make it a requirement or not.
>> I'd prefer if it was always there (either at the beginning or the end),
>> but if there is some situation where it's quite difficult to calculate,
>> then it would be best not to mandate it. I can't think of any impossible
>> situations at the moment.
> 
> I think we should require one of:
> 
> - starting plan line
> - ending plan line

> - ending with something that indicates "I'm done, but I have no idea how
>   many tests actually ran" (Maybe "1..?")

I understand the desire to be able to say "I don't know how many tests
actually ran", but the use of it should be discouraged, even if available.

-Frank

> 
> To me, the point of the plan line is to be able to say "this test did,
> in fact, finish". So even if some test can't even count how many tests
> it _ran_, it can at least say "I am now finished".
> 
>>> TAP13/14 makes description optional, are we making it required (I think
>>> we should). There seems to be a TAP13/14 "convention" of starting
>>> <description> with "- ", which I'm on the fence about it. It does make
>>> parsing maybe a little easier.
>>
>> I would like the description to be required.
>> I don't have a strong opinion on the dash.  I'm OK with either one (dash
>> or no dash), but we should make kselftest and KUnit consistent.
> 
> I find the dash to be distracting -- it doesn't help me scan it, and it
> doesn't help a parser (which only needs to find "#").
> 
>>>> Differences between kernel test result format and TAP13:
>>>>  - in KTAP the "# SKIP" directive is placed after the description on
>>>>    the test result line
> 
> I sent a bunch of clean-ups for kselftest.h recently[1], but it looks
> like we'll need more for adding "description" to skip (right now it only
> prints the SKIP reason).
> 
> [1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@chromium.org/
> 
>>> Yes Documentation/*.rst Not sure on name yet, but where do kselftest
>>> docs live? :)
>> Documentation/dev-tools/kselftest.rst
>>
>> I'll put this at: Documentation/dev-tools/test-results-format.rst
> 
> Sounds good!
>