linux-kernel - Re: [PATCH v3 1/2] kunit: tool: Fix bug in parsing test plan

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <D8L0E1Q6XSWX.2ZJ6N0FMGKEJ4@google.com>
Date: Thu, 20 Mar 2025 10:07:57 +0000
From: Brendan Jackman <jackmanb@...gle.com>
To: Rae Moar <rmoar@...gle.com>
Cc: David Gow <davidgow@...gle.com>, <shuah@...nel.org>, 
	<linux-kselftest@...r.kernel.org>, <kunit-dev@...glegroups.com>, 
	<linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v3 1/2] kunit: tool: Fix bug in parsing test plan

On Wed Mar 19, 2025 at 9:11 PM UTC, Rae Moar wrote:
> On Mon, Mar 17, 2025 at 12:13 PM Brendan Jackman <jackmanb@...gle.com> wrote:
> >
> > On Fri, 14 Mar 2025 at 06:37, David Gow <davidgow@...gle.com> wrote:
> > >
> > > On Fri, 14 Mar 2025 at 03:27, Rae Moar <rmoar@...gle.com> wrote:
> > > >
> > > > A bug was identified where the KTAP below caused an infinite loop:
> > > >
> > > >  TAP version 13
> > > >  ok 4 test_case
> > > >  1..4
> > > >
> > > > The infinite loop was caused by the parser not parsing a test plan
> > > > if following a test result line.
> > > >
> > > > Fix this bug by parsing test plan line to avoid the infinite loop.
> >
> > Hi Rae,
> >
> > With this change and this input:
> >
> > https://gist.githubusercontent.com/bjackman/220265699f346e16161c6534b115019b/raw/a2e0e1aa75c0d8ab9814708b028ec78810a0471b/run_vmtests.sh.tap
> >
> > The infinite loop is gone, but it's still hallucinating a [CRASHED] result:
> >
> > [16:07:15] # SUMMARY: PASS=17 SKIP=0 FAIL=1
> > [16:07:15] [CRASHED]
> > ...
> > [16:07:15] Testing complete. Ran 19 tests: passed: 17, failed: 1, crashed: 1
>
> Hi! Thanks for the response. This is an interesting problem. Should a
> test plan at the bottom cause a crash because no tests were found
> after? Again with KTAP, a crash would make sense. I feel this example
> demonstrates why there is a need for a general parser that can parse
> kselftest output as well as KUnit.  I'll see how difficult it would be
> to change the parser to accommodate removing the crash in a new
> version. Thanks!

Yeah, if this is difficult, it would be fine for the KUnit tool to
just say "sorry this is a KTAP parser not a general TAP parser"

With your new patch:

https://lore.kernel.org/linux-kernel/20250319223351.1517262-1-rmoar@google.com/

I get this:

[10:00:41] # SUMMARY: PASS=17 SKIP=0 FAIL=1
[10:00:41] [ERROR] Test: : No more test results!
[10:00:41] [NO TESTS RUN]

Which I think is basically fine.

We do want something that can parse this stuff properly but I tihnk
that's a separate little project.

Also, vanilla non-K TAP... is honestly a terrible format.  A flat
structure for the test results (i.e. no nesting) is a bit of a joke -
even for just run_vmtests.sh which has a pretty tractable amount of
tests, it's already a pain to mentally keep track of which results
come from where. Maybe if I want to be able to parse the output of
tests I should just switch them to using KTAP.