linux-kernel - Re: [PATCH v2 2/4] kunit: tool: Support skipped tests in kunit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAFd5g44VS==1yULMRS-JMxrArj9GFJRkuDCxoxnZHcj3PVbFHg@mail.gmail.com>
Date:   Fri, 4 Jun 2021 14:30:13 -0700
From:   Brendan Higgins <brendanhiggins@...gle.com>
To:     Daniel Latypov <dlatypov@...gle.com>
Cc:     David Gow <davidgow@...gle.com>,
        Alan Maguire <alan.maguire@...cle.com>,
        Shuah Khan <skhan@...uxfoundation.org>,
        Marco Elver <elver@...gle.com>,
        KUnit Development <kunit-dev@...glegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v2 2/4] kunit: tool: Support skipped tests in kunit_tool

On Tue, Jun 1, 2021 at 8:46 AM Daniel Latypov <dlatypov@...gle.com> wrote:
>
> On Fri, May 28, 2021 at 12:59 AM David Gow <davidgow@...gle.com> wrote:
> >
> > Add support for the SKIP directive to kunit_tool's TAP parser.
> >
> > Skipped tests now show up as such in the printed summary. The number of
> > skipped tests is counted, and if all tests in a suite are skipped, the
> > suite is also marked as skipped. Otherwise, skipped tests do affect the
> > suite result.
> >
> > Example output:
> > [00:22:34] ======== [SKIPPED] example_skip ========
> > [00:22:34] [SKIPPED] example_skip_test # SKIP this test should be skipped
> > [00:22:34] [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped
> > [00:22:34] ============================================================
> > [00:22:34] Testing complete. 2 tests run. 0 failed. 0 crashed. 2 skipped.
> >
> > Signed-off-by: David Gow <davidgow@...gle.com>
>
> Reviewed-by: Daniel Latypov <dlatypov@...gle.com>
>
> Some minor remarks, but this looks good to me.
>
> Though I'm surprised there has not been any bikeshedding done about
> the color of the SKIPPED output.
> So I'll throw an opinion out there.
> I think yellow is fine, but I did somewhat recently change another
> similar tool to go from yellow => cyan for SKIPPED. The motivation
> there was to have a color for "flaky" tests that stood out, and the
> most appropriate ANSI color seemed to be yellow (between green for
> PASSED and red for FAILED).
> And I don't know if KUnit tool will ever get to the point where we
> automatically rerun tests on failure, as I can see an argument for
> that logic living a layer above.

I do have some sympathy for using a different color for each type of
message. I am not arguing against cyan, but I am also OK with yellow.
However, if we get to the point where we support flaky warnings, what
if we used orange for flaky?