linux-kernel - Re: [PATCH v2 5/9] test_sysctl: add generic script to expand on tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAGXu5jLB9tJq4i0itMXO856iXiWYcMTdH2YYx579a+f5F8WRPA@mail.gmail.com>
Date:   Mon, 13 Feb 2017 12:30:22 -0800
From:   Kees Cook <keescook@...omium.org>
To:     "Luis R. Rodriguez" <mcgrof@...nel.org>
Cc:     Al Viro <viro@...iv.linux.org.uk>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "Eric W. Biederman" <ebiederm@...ssion.com>,
        Arnaldo Carvalho de Melo <acme@...hat.com>,
        Ingo Molnar <mingo@...nel.org>, Mel Gorman <mgorman@...e.de>,
        Subash Abhinov Kasiviswanathan <subashab@...eaurora.org>,
        Jessica Yu <jeyu@...hat.com>,
        Rusty Russell <rusty@...tcorp.com.au>,
        Steven Whitehouse <swhiteho@...hat.com>,
        deepa.kernel@...il.com, Matt Fleming <matt@...eblueprint.co.uk>,
        Alexey Dobriyan <adobriyan@...il.com>,
        Borislav Petkov <bp@...e.de>,
        Dmitry Torokhov <dmitry.torokhov@...il.com>, shuah@...nel.org,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Guenter Roeck <linux@...ck-us.net>,
        linux-kselftest@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v2 5/9] test_sysctl: add generic script to expand on tests

On Fri, Feb 10, 2017 at 4:36 PM, Luis R. Rodriguez <mcgrof@...nel.org> wrote:
> This adds a generic script to let us more easily add more tests
> cases. Since we really have only two types of tests cases just
> fold them into the one file. Each test unit is now identified
> into its separate function:
>
>   # ./sysctl.sh -l
> Test ID list:
>
> TEST_ID x NUM_TEST
> TEST_ID:   Test ID
> NUM_TESTS: Number of recommended times to run the test
>
> 0001 x 1 - tests proc_dointvec_minmax()
> 0002 x 1 - tests proc_dostring()
>
> For now we start off with what we had before, and run only each test once.
> We can now watch a test case until it fails:
>
> ./sysctl.sh -w 0002
>
> We can also run a test case x number of times, say we want to run
> a test case 100 times:
>
> ./sysctl.sh -c 0001 100
>
> To run a test case only once, for example:
>
> ./sysctl.sh -s 0002
>
> The default settings are specified at the top of sysctl.sh.
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@...nel.org>

I'm not a fan of this: it consolidates tests when it's not needed and
creates a test running infrastructure at the wrong level of
abstraction. I'd like to see individual tests that are one-off
runnable. Whatever consumes the tools/testing/selftests/ tree is what
should be doing the -w, -c, etc style options.

-Kees

-- 
Kees Cook
Pixel Security