linux-kernel - Re: [PATCH v2 2/4] kunit: tool: Support skipped tests in kunit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAGS_qxocLmAV9TDw1dqqhJa=UZMt=sa9ohqJpcQ=xFT=gz0CGQ@mail.gmail.com>
Date:   Fri, 4 Jun 2021 14:44:04 -0700
From:   Daniel Latypov <dlatypov@...gle.com>
To:     Brendan Higgins <brendanhiggins@...gle.com>
Cc:     David Gow <davidgow@...gle.com>,
        Alan Maguire <alan.maguire@...cle.com>,
        Shuah Khan <skhan@...uxfoundation.org>,
        Marco Elver <elver@...gle.com>,
        KUnit Development <kunit-dev@...glegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v2 2/4] kunit: tool: Support skipped tests in kunit_tool

On Fri, Jun 4, 2021 at 2:30 PM 'Brendan Higgins' via KUnit Development
<kunit-dev@...glegroups.com> wrote:
>
> On Tue, Jun 1, 2021 at 8:46 AM Daniel Latypov <dlatypov@...gle.com> wrote:
> >
> > On Fri, May 28, 2021 at 12:59 AM David Gow <davidgow@...gle.com> wrote:
> > >
> > > Add support for the SKIP directive to kunit_tool's TAP parser.
> > >
> > > Skipped tests now show up as such in the printed summary. The number of
> > > skipped tests is counted, and if all tests in a suite are skipped, the
> > > suite is also marked as skipped. Otherwise, skipped tests do affect the
> > > suite result.
> > >
> > > Example output:
> > > [00:22:34] ======== [SKIPPED] example_skip ========
> > > [00:22:34] [SKIPPED] example_skip_test # SKIP this test should be skipped
> > > [00:22:34] [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped
> > > [00:22:34] ============================================================
> > > [00:22:34] Testing complete. 2 tests run. 0 failed. 0 crashed. 2 skipped.
> > >
> > > Signed-off-by: David Gow <davidgow@...gle.com>
> >
> > Reviewed-by: Daniel Latypov <dlatypov@...gle.com>
> >
> > Some minor remarks, but this looks good to me.
> >
> > Though I'm surprised there has not been any bikeshedding done about
> > the color of the SKIPPED output.
> > So I'll throw an opinion out there.
> > I think yellow is fine, but I did somewhat recently change another
> > similar tool to go from yellow => cyan for SKIPPED. The motivation
> > there was to have a color for "flaky" tests that stood out, and the
> > most appropriate ANSI color seemed to be yellow (between green for
> > PASSED and red for FAILED).
> > And I don't know if KUnit tool will ever get to the point where we
> > automatically rerun tests on failure, as I can see an argument for
> > that logic living a layer above.
>
> I do have some sympathy for using a different color for each type of
> message. I am not arguing against cyan, but I am also OK with yellow.
> However, if we get to the point where we support flaky warnings, what
> if we used orange for flaky?

We can always change the colors later.
I don't seriously think this should hold up the patch.
I was partially just bemused that no one had chimed in about the
color, as that is the apocryphal thing to bikeshed. Was curious if
broaching the topic would elicit any response, but given the silence,
I think we can truly say that no one cares.

Also, orange only works if we're going to assume 256-color support,
which I'd be against*.

Given that, our options are:
Black, Red, Green, Yellow, Blue, Magenta, Cyan, White
(and the "bright" variations thereof)

So yellow is the closest we'd get, which is why I'd made the shift to
cyan for the aforementioned other projects.

>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@...glegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/CAFd5g44VS%3D%3D1yULMRS-JMxrArj9GFJRkuDCxoxnZHcj3PVbFHg%40mail.gmail.com.