linux-kernel - Re: [PATCH] kunit/kunit_kernel: Rebuild .config if .kunitconfig is modified

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <b13a1268-1542-ec1c-3316-c194597f7849@kernel.org>
Date:   Fri, 13 Mar 2020 09:45:40 -0600
From:   shuah <shuah@...nel.org>
To:     Russell Currey <ruscur@...sell.cc>,
        Brendan Higgins <brendanhiggins@...gle.com>,
        David Gow <davidgow@...gle.com>
Cc:     SeongJae Park <sj38.park@...il.com>,
        "open list:KERNEL SELFTEST FRAMEWORK" 
        <linux-kselftest@...r.kernel.org>,
        KUnit Development <kunit-dev@...glegroups.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        SeongJae Park <sjpark@...zon.de>,
        Theodore Ts'o <tytso@....edu>,
        Bjorn Helgaas <bhelgaas@...gle.com>, shuah <shuah@...nel.org>
Subject: Re: [PATCH] kunit/kunit_kernel: Rebuild .config if .kunitconfig is
 modified

On 2/5/20 3:09 PM, Russell Currey wrote:
> On Thu, Feb 6, 2020, at 7:00 AM, Brendan Higgins wrote:
>> On Wed, Feb 5, 2020 at 9:58 AM David Gow <davidgow@...gle.com> wrote:
>>>
>>> One thing we'd like to do with kunit_tool is to make its functionality
>>> a bit more independent: in particular, allowing the configuration,
>>> running the kernel, and parsing the results to be done independently.
>>>
>>> If that's the case, it may make sense for "kunit.py run" or similar to
>>> not do anything with the .config, and to relegate that to a separate
>>> "configuration" step, which would allow someone to modify the
>>> configuration themselves (e.g., using make menuconfig) and re-run the
>>> tests, but also allow the config to be explicitly regenerated when
>>> helpful.
>>>
>>> Exactly what that'd end up looking like (and to what extent we'd still
>>> want to support a single command that'd do both) are still up in the
>>> air: but I think a general "separation of concerns" like this is
>>> probably the right path forward for kunit_tool.
>>
>> You and I have talked about splitting up kunit_tool's functionality
>> before. I agree with the idea.
>>
>> I imagine it that we would have
>>
>> - configuration
>> - running tests
>> - dmesg/TAP parsing
>>
>> as separate runnable scripts. I think that would make it a lot easier
>> for people with various test bed setups to reuse our code in their
>> test harness.
>>
>> Nevertheless, I think it would also be nice to have, as Ted has
>> previously suggested, a short easy to remember one line command that
>> just works; it is easily said, and much harder to do, but I think it
>> is at odds with the separation of functionality. I guess one solution
>> might just be to have these three separate tools, and then the classic
>> kunit.py script that combines the functionalities in a single step, or
>> as Ted suggested we could have some sort of default "make kunit"
>> command or something like that. I am not really sure what is best
>> here.
>>
>> It doesn't address the problem of separation of functionality in
>> anyway, but one way we could achieve the idea of having a command that
>> just works, is by putting a line in MAINTAINERS file entries that have
>> a command that a maintainer expects a submitter to run before sending
>> a patch to LKML. That might at least make it possible to hack together
>> a single line KUnit command for every relevant MAINTAINERS entry.
>> (Obviously there is no reason we have to do this particular idea just
>> for KUnit. We could do this for other tests as well.) Russel, I think
>> this was your idea at LCA?
> 
> Hi Brendan, it wasn't me, it was someone in the audience during questions in my
> testing talk.  I don't recall who.
> 
> They were suggesting a script like get_maintainers - i.e. get_tests - that for a
> given file/patch/commit it gives you a suggested set of tests, whether that's
> KUnit you can run there and then, or selftests you can run once it's booted,
> or maybe external test suites that are relevant.
> 

I like this idea of get_tests type script that could be run separately
as well as part of check_patch or get_maintainers will serve as a
reminder or hint to patch submitter.

We have some pieces in the MAINTAINERS file now. Selftest files are
usually listed under subsystem entries. get_tests could leverage
that and we will definitely more information to for a complete set
of tests for a subsystem.

thanks,
-- Shuah