Message-ID: <20230821141327.1ae35b2e@kernel.org>
Date: Mon, 21 Aug 2023 14:13:27 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: "Michalik, Michal" <michal.michalik@...el.com>
Cc: "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
 "vadim.fedorenko@...ux.dev" <vadim.fedorenko@...ux.dev>, "jiri@...nulli.us"
 <jiri@...nulli.us>, "Kubalewski, Arkadiusz"
 <arkadiusz.kubalewski@...el.com>, "jonathan.lemon@...il.com"
 <jonathan.lemon@...il.com>, "pabeni@...hat.com" <pabeni@...hat.com>, poros
 <poros@...hat.com>, "Olech, Milena" <milena.olech@...el.com>, mschmidt
 <mschmidt@...hat.com>, "linux-clk@...r.kernel.org"
 <linux-clk@...r.kernel.org>, "bvanassche@....org" <bvanassche@....org>
Subject: Re: [PATCH RFC net-next v1 2/2] selftests/dpll: add DPLL system
 integration selftests

On Mon, 21 Aug 2023 09:32:56 +0000 Michalik, Michal wrote:
> On 18 August 2023 11:08 PM CEST, Jakub Kicinski wrote:
> > How fragile do you reckon this setup will be?
> > I mean will it work reliably across distros and various VM setups?
> > I have tried writing tests based on ynl.py and the C codegen, and
> > I can't decide whether the python stuff is easy enough to deploy.
> > Much easier to scp over to the test host a binary based on the 
> > C code. But typing tests in python is generally quicker...
> > What are your thoughts?
> > 
> > Thanks for posting the tests!  
> 
> Hi Jakub,
> 
> First of all - everything I'll write is just my opinion. While having
> quite a bit Python experience, I can't speak confidently about the
> target systems and hardware (architectures) it would be ran against and
> what might be possible problems around that. I need your help here.
> 
> From my point of view, all we need is a Python 3.7 and access to PyPI
> repositories. This version of Python is 5 years old, but if we need
> support of even older versions I can rewrite the code not to use
> dataclasses[1] so we could go with even older version. Do you think it
> is required to support platforms with no Python at all?
> 
> Another requirement is the toolchain for building the module, but I
> assume if Python is not there in some embedded system - the build tools
> are also not there...

I think the module would need to be merged with netdevsim or some such.
The usual process is for module to be part of the normal kernel build
and then require appropriate CONFIG_* options to be set.

I wonder about Python availability. If anyone who works on embedded
knows of Linux platforms which don't have Python - please shout.
Hopefully we're good on that side.

PyPI is a bigger concern, I think pure SW tests are often run in VMs
without external connectivity.

> I've seen other Python pytest code in the kernel repo - that is why I
> thought it might be a good idea to propose something like that. Your
> idea is also cool - binary is always superior. I am a strong advocate of
> taking into consideration not only deployment ease, but also maintenance
> and ease of read which might encourage community to help. I also see a
> benefit of showing the sample implementation (my tested "dummy module").
> 
> My deployment is automatic and does not leave any garbage in the system
> after tests (packages are installed into temporary virtual environment).
> In case any of the requirements are not met - tests are skipped. I've
> tested it both on real HW with some E810T cards installed and on fresh
> VM - seems to work fine. But that of course require further verification.
> Till now only Arek Kubalewski sucessfully gave those tests a shot.

Does it work on a read-only file system? People may mount the kernel
sources over read-only 9p or nfs.

> The biggest concern for me is the requirement of selftests[2]:
>   "Don't take too long;"
> This approach is reloading the modules few times to check few scenarios.
> Also, the DPLL subsystem is being tested against multiple requests - so
> it takes some time to finish (not too long but is definitely not instant).

I think the time constraints are more of a question of practicality.
A developer should be able to run the tests as part of their workflow.

> If you asked me, I would consider those tests to be part of the kernel.
> I am not sure if they should be a part of selftests, though. Maybe a
> reasonable approach would be to have have my tests being a "thorough
> integration tests" and at the same time some limited tests shipped as a
> selftests binary? I can also parametrically limit the scope of my tests
> to run faster in selftests (e.g. only one scenario) and having possibility
> to run extensive tests on demand?
> 
> Thanks,
> M^2
> 
> [1] https://docs.python.org/3/library/dataclasses.html
> [2] https://www.kernel.org/doc/html/v5.0/dev-tools/kselftest.html

Powered by blists - more mailing lists