linux-kernel - Re: [PATCH v4 08/15] kunit: tool: Don't overwrite test status based on subtest counts

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250704145203-1dbf1c70-5ce9-4654-9995-4b794a8ee96e@linutronix.de>
Date: Fri, 4 Jul 2025 14:55:36 +0200
From: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
To: Rae Moar <rmoar@...gle.com>
Cc: Masahiro Yamada <masahiroy@...nel.org>, 
	Nathan Chancellor <nathan@...nel.org>, Andrew Morton <akpm@...ux-foundation.org>, 
	Willy Tarreau <w@....eu>, Thomas Weißschuh <linux@...ssschuh.net>, 
	Brendan Higgins <brendan.higgins@...ux.dev>, David Gow <davidgow@...gle.com>, Shuah Khan <shuah@...nel.org>, 
	Jonathan Corbet <corbet@....net>, Nicolas Schier <nicolas.schier@...ux.dev>, 
	Christophe Leroy <christophe.leroy@...roup.eu>, linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com, linux-doc@...r.kernel.org, 
	workflows@...r.kernel.org
Subject: Re: [PATCH v4 08/15] kunit: tool: Don't overwrite test status based
 on subtest counts

Hi Rae,

On Thu, Jul 03, 2025 at 05:30:02PM +0200, Thomas Weißschuh wrote:
> On Tue, Jul 01, 2025 at 05:11:59PM -0400, Rae Moar wrote:

<snip>

> > > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@...utronix.de>
> > > Reviewed-by: David Gow <davidgow@...gle.com>
> > > ---
> > >  tools/testing/kunit/kunit_parser.py                                  | 5 +++++
> > >  tools/testing/kunit/kunit_tool_test.py                               | 3 ++-
> > >  tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
> > >  3 files changed, 10 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> > > index c176487356e6c94882046b19ea696d750905b8d5..2478beb28fc3db825855ad46200340e884da7df1 100644
> > > --- a/tools/testing/kunit/kunit_parser.py
> > > +++ b/tools/testing/kunit/kunit_parser.py
> > > @@ -686,6 +686,11 @@ def bubble_up_test_results(test: Test) -> None:
> > >                 counts.add_status(status)
> > >         elif test.counts.get_status() == TestStatus.TEST_CRASHED:
> > >                 test.status = TestStatus.TEST_CRASHED
> > > +       if not test.ok_status():
> > > +               for t in subtests:
> > > +                       if not t.ok_status():
> > > +                               counts.add_status(t.status)
> > > +                               break
> > 
> > Here instead I recommend checking if not test.ok_status() and
> > test.counts.get_status() == TestStatus.SUCCESS and if so
> > counts.add_status(status)
> 
> Thanks for the recommendation. I tried this and it works well for this specific
> testcase, but unfortunately all kinds of othes tests are now broken.
> I'll look into it some more, but any hints are highly appreciated.
> It has been a while since I looked at the code.

The following variant passes all tests. What do you think?

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 333cd3a4a56b..5338489dcbe4 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
        elif test.counts.get_status() == TestStatus.TEST_CRASHED:
                test.status = TestStatus.TEST_CRASHED
 
+       if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
+               counts.add_status(status)
+