linux-kernel - Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <CAGS_qxo75FDOpG3e3eskWMx3CoveiyG+ujM-sxuLWw4+7myh5g@mail.gmail.com>
Date:   Fri, 30 Oct 2020 15:40:10 -0700
From:   Daniel Latypov <dlatypov@...gle.com>
To:     David Gow <davidgow@...gle.com>
Cc:     Brendan Higgins <brendanhiggins@...gle.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH] kunit: tool: fix extra trailing \n in parsed test output

On Fri, Oct 30, 2020 at 12:22 AM David Gow <davidgow@...gle.com> wrote:
>
> On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov <dlatypov@...gle.com> wrote:
> >
> > On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@...gle.com> wrote:
> > >
> > > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@...gle.com> wrote:
> > > >
> > > > For simplcity, strip all trailing whitespace from parsed output.
> > > > I imagine no one is printing out meaningful trailing whitespace via
> > > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't.
> > > >
> > > > At some point, the lines from `isolate_kunit_output()` started having
> > > > trailing \n, which results in artifacty output like this:
> > > >
> > > > $ ./tools/testing/kunit/kunit.py run
> > > > [16:16:46] [FAILED] example_simple_test
> > > > [16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > > >
> > > > [16:16:46]     Expected 1 + 1 == 3, but
> > > >
> > > > [16:16:46]         1 + 1 == 2
> > > >
> > > > [16:16:46]         3 == 3
> > > >
> > > > [16:16:46]     not ok 1 - example_simple_test
> > > >
> > > > [16:16:46]
> > > >
> > > > After this change:
> > > > [16:16:46]     # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > > > [16:16:46]     Expected 1 + 1 == 3, but
> > > > [16:16:46]         1 + 1 == 2
> > > > [16:16:46]         3 == 3
> > > > [16:16:46]     not ok 1 - example_simple_test
> > > > [16:16:46]
> > > >
> > > > Signed-off-by: Daniel Latypov <dlatypov@...gle.com>
> > > > ---
> > >
> > > Thanks! This is a long-overdue fix, and it worked well for me.
> > >
> > > Tested-by: David Gow <davidgow@...gle.com>
> > >
> > > One comment below:
> > >
> > > >  tools/testing/kunit/kunit_parser.py | 3 ++-
> > > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> > > > index 8019e3dd4c32..e68b1c66a73f 100644
> > > > --- a/tools/testing/kunit/kunit_parser.py
> > > > +++ b/tools/testing/kunit/kunit_parser.py
> > > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult:
> > > >         total_tests = 0
> > > >         failed_tests = 0
> > > >         crashed_tests = 0
> > > > -       test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
> > > > +       test_result = parse_test_result(list(
> > > > +            l.rstrip() for l in isolate_kunit_output(kernel_output)))
> > >
> > > Could we do this inside isolate_kunit_output() instead? That seems
> > > like it'd be a more logical place for it (removing the newline is a
> > > sort of isolating the output), and it'd avoid making this line quite
> > > as horrifyingly nested.
> >
> > Good point.
> > We could either do it on each yield (messy), or before, i.e.
> >
> > diff --git a/tools/testing/kunit/kunit_parser.py
> > b/tools/testing/kunit/kunit_parser.py
> > index 8019e3dd4c32..14d35deb96cd 100644
> > --- a/tools/testing/kunit/kunit_parser.py
> > +++ b/tools/testing/kunit/kunit_parser.py
> > @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|'
> >  def isolate_kunit_output(kernel_output):
> >         started = False
> >         for line in kernel_output:
> > +               line = line.rstrip()  # line always has a trailing \n
> >                 if kunit_start_re.search(line):
> >                         prefix_len = len(line.split('TAP version')[0])
> >                         started = True
> >
> > I had some vague concerns about this as
> >   kunit_start_re = re.compile(r'TAP version [0-9]+$')
> > has that anchor at the end.
> >
> > This could ostensibly make it match more things than before.
> > Since I'm using rstrip() out of laziness, that means strings like
> >   '<prefix we allow for some reason>TAP version 42\t\n'
> > will now also match.
> >
> > I don't really think that's an issue, but I'd sent this as a more
> > conservative change initially.
> > I can send the diff above as a replacement for this patch.
>
> I prefer this if it works. From my cursory testing, it does (though
> the kunt_tool_tests.py tests will need updating). At the very least,
> I'm able to get it to work with --alltests / allyesconfig (with a few
> options tactically disabled), which was the main reason we needed
> isolate_kunit_output() in the first place.
>
> So, unless anyone can find a real-world case where this breaks
> something, let's go with this.

Sounds good.
Sent https://lore.kernel.org/linux-kselftest/20201030223853.554597-1-dlatypov@google.com

>
> Cheers,
> -- David
>
> >
> > >
> > > >         if test_result.status == TestStatus.NO_TESTS:
> > > >                 print(red('[ERROR] ') + yellow('no tests run!'))
> > > >         elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
> > > >
> > > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93
> > > > --
> > > > 2.29.0.rc1.297.gfa9743e501-goog
> > > >