linux-kernel - Re: [RFC v4 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <961494a3-d08c-2720-c59d-7d7008edb288@deltatee.com>
Date:   Thu, 21 Mar 2019 13:29:38 -0600
From:   Logan Gunthorpe <logang@...tatee.com>
To:     Knut Omang <knut.omang@...cle.com>,
        Brendan Higgins <brendanhiggins@...gle.com>
Cc:     Kees Cook <keescook@...gle.com>,
        Luis Chamberlain <mcgrof@...nel.org>, shuah@...nel.org,
        Rob Herring <robh@...nel.org>,
        Kieran Bingham <kieran.bingham@...asonboard.com>,
        Frank Rowand <frowand.list@...il.com>,
        Greg KH <gregkh@...uxfoundation.org>,
        Joel Stanley <joel@....id.au>,
        Michael Ellerman <mpe@...erman.id.au>,
        Joe Perches <joe@...ches.com>, brakmo@...com,
        Steven Rostedt <rostedt@...dmis.org>,
        "Bird, Timothy" <Tim.Bird@...y.com>,
        Kevin Hilman <khilman@...libre.com>,
        Julia Lawall <julia.lawall@...6.fr>,
        linux-kselftest@...r.kernel.org, kunit-dev@...glegroups.com,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Jeff Dike <jdike@...toit.com>,
        Richard Weinberger <richard@....at>,
        linux-um@...ts.infradead.org, Daniel Vetter <daniel@...ll.ch>,
        dri-devel <dri-devel@...ts.freedesktop.org>,
        Dan Williams <dan.j.williams@...el.com>,
        linux-nvdimm <linux-nvdimm@...ts.01.org>,
        devicetree <devicetree@...r.kernel.org>,
        Petr Mladek <pmladek@...e.com>,
        Sasha Levin <Alexander.Levin@...rosoft.com>,
        Amir Goldstein <amir73il@...il.com>,
        Dan Carpenter <dan.carpenter@...cle.com>, wfg@...ux.intel.com,
        Alan Maguire <alan.maguire@...cle.com>
Subject: Re: [RFC v4 00/17] kunit: introduce KUnit, the Linux kernel unit
 testing framework



On 2019-03-21 1:13 p.m., Knut Omang wrote:
>> Nevertheless, I don't really see KTF as a real unit testing framework
>> for a number of different reasons; you pointed out some below, but I
>> think the main one being that it requires booting a real kernel on
>> actual hardware; 
> 
> That depends on what you want to test. If you need hardware (or simulated or
> emulated hardware) for the test, of course you would need to have that hardware,
> but if, lets say, you just wanted to run tests like the skbuff example tests
> (see link above) you wouldn't need anything more than what you need to run KUnit
> tests.

I'm starting to get the same impression: KTF isn't unit testing. When we
are saying "unit tests" we are specifying exactly what we want to test:
small sections of code in isolation. So by definition you should not
need hardware for this.

> I have fulfilled that dream, so I know it is possible (Inifinband driver,
> kernels from 2.6.39 to 4.8.x or so..) I know a lot of projects would benefit
> from support for such workflows, but that's not really the point here - we want
> to achieve both goals!

This is what makes me think we are not talking about testing the same
things. We are not talking about end to end testing of entire drivers
but smaller sections of code. A unit test is far more granular and
despite an infinband driver existing for 2.6.39 through 4.8, the
internal implementation could be drastically different. But unit tests
would be testing internal details which could be very different version
to version and has to evolve with the implementation.

> If your target component under test can be built as a kernel module, or set of
> modules, with KTF your workflow would not involve booting at all (unless you
> happened to crash the system with one of your tests, that is :) )

> You would just unload your module under test and the test module, recompile the
> two and insmod again. My work current work cycle on this is just a few seconds.

Yes, I'm sure we've all done that many a time but it's really beside the
point. Kunit offers a much nicer method for running a lot of unit tests
on existing code.

Logan