Message-ID: <20220406140816.000038ce@Huawei.com>
Date:   Wed, 6 Apr 2022 14:08:16 +0100
From:   Jonathan Cameron <Jonathan.Cameron@...wei.com>
To:     Vincent Whitchurch <vincent.whitchurch@...s.com>
CC:     Jonathan Cameron <jic23@...nel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        kernel <kernel@...s.com>,
        "devicetree@...r.kernel.org" <devicetree@...r.kernel.org>,
        "linux-um@...ts.infradead.org" <linux-um@...ts.infradead.org>,
        "shuah@...nel.org" <shuah@...nel.org>,
        "brendanhiggins@...gle.com" <brendanhiggins@...gle.com>,
        "linux-kselftest@...r.kernel.org" <linux-kselftest@...r.kernel.org>,
        "linux-iio@...r.kernel.org" <linux-iio@...r.kernel.org>,
        "lgirdwood@...il.com" <lgirdwood@...il.com>,
        "broonie@...nel.org" <broonie@...nel.org>,
        "a.zummo@...ertech.it" <a.zummo@...ertech.it>,
        "alexandre.belloni@...tlin.com" <alexandre.belloni@...tlin.com>,
        "linux-rtc@...r.kernel.org" <linux-rtc@...r.kernel.org>,
        "corbet@....net" <corbet@....net>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>
Subject: Re: [RFC v1 08/10] iio: light: vcnl4000: add roadtest

On Tue, 5 Apr 2022 15:48:05 +0200
Vincent Whitchurch <vincent.whitchurch@...s.com> wrote:

> On Sun, Mar 20, 2022 at 06:02:53PM +0100, Jonathan Cameron wrote:
> > Very interesting bit of work. My current approach for similar testing
> > is to write a qemu model for the hardware, but that currently
> > requires carefully crafted tests. Most of the time I'm only doing
> > that to verify refactoring of existing drivers.   
> 
> Thank you for taking a look!
> 
> > One thing that makes me nervous here is the python element though
> > as I've not written significant python in about 20 years.
> > That is going to be a burden for kernel developers and maintainers...
> > Nothing quite like badly written tests to make for a mess in the long run
> > and I suspect my python for example would be very very badly written :)  
> 
> There's a bunch of static checkers to ensure that the code follows some
> basic guidelines, and CI can check that the tests work consistently, and
> also calculate metrics such as test execution time and code coverage, so
> even non-idiomatic Python in the tests wouldn't be entirely broken.
> 
> And unlike driver code, if the tests for a particular driver later do
> turn out to be bad (in what way?), we could just throw those particular
> tests out without breaking anybody's system.

True.  Though CI test triage folk may disagree ;)

> 
> > Cut and paste will of course get us a long way...  
> 
> Isn't some amount of copy/paste followed by modification to be expected
> even if the framework is written in say C (just as there's already
> copy/paste + modification involved when writing drivers)?
> 
> As for the core logic of individual driver tests excluding the framework
> bits, I have a hard time imagining what Python syntax looks like to
> someone with no knowledge of Python, so yes, I guess it's going to be
> harder to review.

I messed around the other day with writing tests for
drivers/staging/iio/cdc/ad7746.c and wasn't "too bad" and was useful for
verifying some refactoring (and identified a possible precision problem
in some integer approximation of floating point calcs)
I'll try and find time to flesh that test set out more in the near future and
post it so you can see how bad my python is. It amused my wife if nothing
else :)

However a future project is to see if I can use this to hook up the SPDM
attestation stack via mctp over i2c - just because I like to live dangerously :)

For IIO use more generally we need a sensible path to SPI (and also platform
drivers).  For my day job I'd like to mess around with doing PCI devices
as well.  The PCI DOE support for example would be nice to run against a
test set that doesn't involve spinning up QEMU.
DOE driver support:
https://lore.kernel.org/all/20220330235920.2800929-1-ira.weiny@intel.com/

Effort wise, it's similar effort to hacking equivalent in QEMU but with the
obvious advantage of being in tree and simpler for CI systems etc to use.

It would be nice to only have to use QEMU for complex system CI tests
like the ones we are doing for CXL.

> 
> > I dream of a world where every driver is testable by people with out hardware
> > but I fear it may be a while yet.  Hopefully this will get us a little
> > closer!
> > 
> > I more or less follow what is going on here (good docs btw in the earlier
> > patch definitely helped).
> > 
> > So far I'm thoroughly in favour of road test subject to actually being
> > able to review the tests or getting sufficient support to do so.
> > It's a 'how to scale it' question really...  
> 
> Would rewriting the framework in C and forcing tests to be written in
> that language mean that maintainers would be able to review tests
> without external support?

I was wondering that.  If we stayed in python I think we'd definitely want
someone to be the 'roadtester/tests' maintainer (or group of maintainers) 
and their Ack to be expected for all tests we upstream.  Idea being they'd
sanity check correct use of framework and just how bad the python code
us C developers are writing is ;)

However, we'd still need a good chunk of that 'framework' use review even
if doing this in C.

Anyhow, very promising bit of work.

Thanks,

Jonathan

Powered by blists - more mailing lists