linux-kernel - Re: [PATCH] kunit: tool: Fix bug in parsing test plan

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <Z8nKmGwSu9RvxbWc@google.com>
Date: Thu, 6 Mar 2025 16:17:28 +0000
From: Brendan Jackman <jackmanb@...gle.com>
To: Rae Moar <rmoar@...gle.com>
Cc: David Gow <davidgow@...gle.com>, shuah@...nel.org,
	linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com,
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH] kunit: tool: Fix bug in parsing test plan

Hi Rae,

On Thu, Mar 06, 2025 at 11:02:13AM -0500, Rae Moar wrote:
> On Thu, Mar 6, 2025 at 7:26 AM Brendan Jackman <jackmanb@...gle.com> wrote:
> > Since I reported this bug, I discovered that the example above is in
> > fact valid TAP:
> >
> > > The plan [...] must appear once, whether at the beginning or end of the output.
> >
> > From https://testanything.org/tap-version-13-specification.html
> 
> Hi!
> This brings up an interesting question because the parser has been
> mainly geared towards parsing KTAP
> (https://docs.kernel.org/dev-tools/ktap.html) rather than TAP.
> (Although we do try to have backwards compatibility with TAP v14
> "Subtest" lines)
> 
> For example,
> 
> TAP version 13
> 1..1
>   TAP version 13
>   1..1
>   ok 1 test_case
> ok 1 test_suite
> 
> This would be accepted by the parser without error because it is valid
> KTAP even though it is not valid TAP v13.
> 
> The scenario above that caused the infinite loop would be incorrect
> KTAP (which requires the test plan to follow a version line) but
> correct TAP v13. So do we accept it without error? Ideally, we would
> parse based on the version given in the version line.
> 
> Just an interesting thought. Either way, I will remove the error for
> now as our parameterized tests don't properly produce a test plan,
> which causes errors.

OK yeah sounds good. Now I think about it, I should note that I am
abusing the KUnit scripts to parse the result of a kselftest run. I
just assumed that this would be supported but actually there's no
particular reason I should have assumed that :)

I would like to have general support in the tree for parsing test
output ([K]TAP it's not really a human-readable format IMO... even in
the best case like nice tidy KUnit tests I find the structure very
hard to read. And without something like 'kunit.py parse' I find it
extremely difficult to get a high-level gestalt of how a test run
went). But that doesn't mean kunit.py is responsible for the whole
kernel tree's output! Still, it would be nice to handle it to the
extent that's practical (and at least, with no infinite loops :D).