linux-kernel - Re: RFC - kernel selftest result documentation (KTAP)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Sat, 20 Jun 2020 09:51:06 -0500
From:   Frank Rowand <frowand.list@...il.com>
To:     Paolo Bonzini <pbonzini@...hat.com>,
        Kees Cook <keescook@...omium.org>,
        Brendan Higgins <brendanhiggins@...gle.com>
Cc:     "Bird, Tim" <Tim.Bird@...y.com>,
        "shuah@...nel.org" <shuah@...nel.org>,
        "linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        David Gow <davidgow@...gle.com>,
        Frank Rowand <frowand.list@...il.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)

On 2020-06-19 17:58, Paolo Bonzini wrote:
> On 19/06/20 20:47, Frank Rowand wrote:
>> Or if the entire test depends on the missing config then Bail out might
>> be appropriate.
> 
> No, in that case you want
> 
> 	1..0 # SKIP: unsupported configuration
> 
> The spec is not clear if "Bail out!" is an error condition or just a
> warning that only part of the test was run, but prove(1) and Automake
> both treat it as the former, for example.
> 
> For example, an ENOSPC error creating a temporary file could be turned
> into a bail-out, while an ENOSYS would be a skip.
> 
> Paolo
> 

I think that "Bail out!" needs to be more clearly defined by the spec,
and we should come up with an intent of what it is intended to mean.

The TAP 13 spec gives a "Bail out!" example of "MySQL is not running."
The spec gives this example after the general guidance of (among other
things) "missing dependencies".  A missing config _could_ be thought
of as a missing dependency.

To me, the key differentiator for "Bail out!" is "that further tests
are useless".  In other words, the detected problem means that
every subsequent part of the tests will fail for the same reason.

If _some_ subsequent parts of the tests will NOT fail for the same
reason then SKIP becomes more reasonable.  What percent of subsequent
parts of the tests is large enough to comprise "_some_" is probably
a value judgement.

For the case that I was commenting on, with the limited amount of
description provided, my preference is also SKIP, which is what I
was suggesting.