linux-kernel - Re: RFC - kernel selftest result documentation (KTAP)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <202006161653.15C278A5@keescook>
Date:   Tue, 16 Jun 2020 16:58:46 -0700
From:   Kees Cook <keescook@...omium.org>
To:     Brendan Higgins <brendanhiggins@...gle.com>
Cc:     "Bird, Tim" <Tim.Bird@...y.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        "shuah@...nel.org" <shuah@...nel.org>,
        "linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        David Gow <davidgow@...gle.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)

On Tue, Jun 16, 2020 at 12:44:28PM -0700, Brendan Higgins wrote:
> On Tue, Jun 16, 2020 at 9:42 AM Bird, Tim <Tim.Bird@...y.com> wrote:
> > > From: Paolo Bonzini <pbonzini@...hat.com>
> > >
> > > On 15/06/20 21:07, Bird, Tim wrote:
> > > >> Note: making the plan line required differs from TAP13 and TAP14. I
> > > >> think it's the right choice, but we should be clear.
> > >
> > > As an aside, where is TAP14?
> > By TAP14, I was referring to the current, undocumented, KUnit
> > conventions.
> 
> Not so. TAP14 is the proposed next version of TAP13:
> 
> https://github.com/TestAnything/testanything.github.io/pull/36
> https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md

I was reading this (I haven't compared to the blob above):

https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md

> Based on the discussion, it seems like most of the things we wanted
> from TAP14 would probably make it in if TAP ever accepts another pull
> request.

Were our leading diagnostic lines part of their latest spec? I thought
we were pretty far off in left field for that particular bit.

> > My personal preference is to have the dash.  I think it's more human readable.
> > I note that the TAP spec has examples of result lines both with and without
> > the dash, so even the spec is ambiguous on this.   I think not mandating it
> > either way is probably best.  For regex parsers, it's easy to ignore with '[-]?'
> > outside the pattern groups that grab the number and description.
> 
> I don't think we care, because we don't use it.

Yeah, I'm in the same place. I don't care -- I would just like a
determination. (The "implied" nature of it in TAP14 bothers me.)

> > > XFAIL/XPASS are different from SKIP.  I personally don't have a need for
> > > them, but kselftests includes XFAIL/XPASS exit codes and they aren't
> > > reflected into selftests/kselftest/runner.sh.
> > >
> > > Likewise, kselftest.h has ksft_inc_xfail_cnt but not
> > > ksft_test_result_xfail/ksft_test_result_xpass.

I proposed fixing that recently[1]. seccomp uses XFAIL for "I have
detected you lack the config to test this, so I can't say it's working
or not, because it only looks like a failure without the config."

[1] https://lore.kernel.org/lkml/20200611224028.3275174-7-keescook@chromium.org/

-- 
Kees Cook