linux-kernel - Re: [PATCH] Documentation: kunit: Add naming guidelines

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <202008311641.D10607D43@keescook>
Date:   Mon, 31 Aug 2020 16:47:40 -0700
From:   Kees Cook <keescook@...omium.org>
To:     David Gow <davidgow@...gle.com>
Cc:     Marco Elver <elver@...gle.com>,
        Brendan Higgins <brendanhiggins@...gle.com>,
        Jonathan Corbet <corbet@....net>,
        Alan Maguire <alan.maguire@...cle.com>,
        Randy Dunlap <rd.dunlab@...il.com>,
        Theodore Ts'o <tytso@....edu>, Tim Bird <Tim.Bird@...y.com>,
        KUnit Development <kunit-dev@...glegroups.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        "open list:DOCUMENTATION" <linux-doc@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] Documentation: kunit: Add naming guidelines

On Fri, Aug 28, 2020 at 12:17:05AM +0800, David Gow wrote:
> On Thu, Aug 27, 2020 at 9:14 PM Marco Elver <elver@...gle.com> wrote:
> > Just an idea: Maybe the names are also an opportunity to distinguish
> > real _unit_ style tests and then the rarer integration-style tests. I
> > personally prefer using the more generic *-test.c, at least for the
> > integration-style tests I've been working on (KUnit is still incredibly
> > valuable for integration-style tests, because otherwise I'd have to roll
> > my own poor-man's version of KUnit, so thank you!). Using *_kunit.c for
> > such tests is unintuitive, because the word "unit" hints at "unit tests"
> > -- and having descriptive (and not misleading) filenames is still
> > important. So I hope you won't mind if *-test.c are still used where
> > appropriate.

This is a good point, and I guess not one that has really been examined.
I'm not sure what to think of some of the lib/ tests. test_user_copy
seems to be a "unit" test -- it's validating the function family vs
all kinds of arguments and conditions. But test_overflow is less unit
and more integration, which includes "do all of these allocators end up
acting the same way?" etc

I'm not really sure what to make of that -- I don't really have a formal
testing background.

> As Brendan alluded to in the talk, the popularity of KUnit for these
> integration-style tests came as something of a surprise (more due to
> our lack of imagination than anything else, I suspect). It's great
> that it's working, though: I don't think anyone wants the world filled
> with more single-use test "frameworks" than is necessary!
> 
> I guess the interesting thing to note is that we've to date not really
> made a distinction between KUnit the framework and the suite of all
> KUnit tests. Maybe having a separate file/module naming scheme could
> be a way of making that distinction, though it'd really only appear
> when loading tests as modules -- there'd be no indication in e.g.,
> suite names or test results. The more obvious solution to me (at
> least, based on the current proposal) would be to have "integration"
> or similar be part of the suite name (and hence the filename, so
> _integration_kunit.c or similar), though even I admit that that's much
> uglier. Maybe the idea of having the subsystem/suite distinction be
> represented in the code could pave the way to having different suites
> support different suffixes like that.

Heh, yeah, let's not call them "_integration_kunit.c" ;) _behavior.c?
_integration.c?

> Do you know of any cases where something has/would have both
> unit-style tests and integration-style tests built with KUnit where
> the distinction needs to be clear?

This is probably the right question. :)

-- 
Kees Cook