Date: Thu, 24 Aug 2023 08:59:53 +0000
From: "Michalik, Michal" <michal.michalik@...el.com>
To: Jakub Kicinski <kuba@...nel.org>
CC: "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"vadim.fedorenko@...ux.dev" <vadim.fedorenko@...ux.dev>, "jiri@...nulli.us"
	<jiri@...nulli.us>, "Kubalewski, Arkadiusz" <arkadiusz.kubalewski@...el.com>,
	"jonathan.lemon@...il.com" <jonathan.lemon@...il.com>, "pabeni@...hat.com"
	<pabeni@...hat.com>, poros <poros@...hat.com>, "Olech, Milena"
	<milena.olech@...el.com>, mschmidt <mschmidt@...hat.com>,
	"linux-clk@...r.kernel.org" <linux-clk@...r.kernel.org>, "bvanassche@....org"
	<bvanassche@....org>
Subject: RE: [PATCH RFC net-next v1 2/2] selftests/dpll: add DPLL system
 integration selftests

On 21 August 2023 11:13 PM CEST, Jakub Kicinski wrote:
> 
> On Mon, 21 Aug 2023 09:32:56 +0000 Michalik, Michal wrote:
>> On 18 August 2023 11:08 PM CEST, Jakub Kicinski wrote:
>> > How fragile do you reckon this setup will be?
>> > I mean will it work reliably across distros and various VM setups?
>> > I have tried writing tests based on ynl.py and the C codegen, and
>> > I can't decide whether the python stuff is easy enough to deploy.
>> > Much easier to scp over to the test host a binary based on the 
>> > C code. But typing tests in python is generally quicker...
>> > What are your thoughts?
>> > 
>> > Thanks for posting the tests!  
>> 
>> Hi Jakub,
>> 
>> First of all - everything I'll write is just my opinion. While having
>> quite a bit Python experience, I can't speak confidently about the
>> target systems and hardware (architectures) it would be ran against and
>> what might be possible problems around that. I need your help here.
>> 
>> From my point of view, all we need is a Python 3.7 and access to PyPI
>> repositories. This version of Python is 5 years old, but if we need
>> support of even older versions I can rewrite the code not to use
>> dataclasses[1] so we could go with even older version. Do you think it
>> is required to support platforms with no Python at all?
>> 
>> Another requirement is the toolchain for building the module, but I
>> assume if Python is not there in some embedded system - the build tools
>> are also not there...
> 
> I think the module would need to be merged with netdevsim or some such.
> The usual process is for module to be part of the normal kernel build
> and then require appropriate CONFIG_* options to be set.

That is really interesting idea - I was not aware about netdevsim, I need to
investigate that possibility. It seems more clean. I've seen other modules in
tools/testing like test-cipher.c, nfit.c, ndtest.c, cxl.c or bpf_testmod.c so I
thought it's not too bad to have a separate, clean DPLL module directly in here.

> 
> I wonder about Python availability. If anyone who works on embedded
> knows of Linux platforms which don't have Python - please shout.
> Hopefully we're good on that side.
> 
> PyPI is a bigger concern, I think pure SW tests are often run in VMs
> without external connectivity.

Those tests are the DPLL subsystem tests, they are not meant to be deployed
on every production platform in my humble opinion. Tests which are "available on
most platfroms" are better than no tests at all "on all platforms".

> 
>> I've seen other Python pytest code in the kernel repo - that is why I
>> thought it might be a good idea to propose something like that. Your
>> idea is also cool - binary is always superior. I am a strong advocate of
>> taking into consideration not only deployment ease, but also maintenance
>> and ease of read which might encourage community to help. I also see a
>> benefit of showing the sample implementation (my tested "dummy module").
>> 
>> My deployment is automatic and does not leave any garbage in the system
>> after tests (packages are installed into temporary virtual environment).
>> In case any of the requirements are not met - tests are skipped. I've
>> tested it both on real HW with some E810T cards installed and on fresh
>> VM - seems to work fine. But that of course require further verification.
>> Till now only Arek Kubalewski sucessfully gave those tests a shot.
> 
> Does it work on a read-only file system? People may mount the kernel
> sources over read-only 9p or nfs.
> 

Since it builds a module and creates a temporary virtual environment in /tmp I
don't think it works for read-only file systems.

>> The biggest concern for me is the requirement of selftests[2]:
>>   "Don't take too long;"
>> This approach is reloading the modules few times to check few scenarios.
>> Also, the DPLL subsystem is being tested against multiple requests - so
>> it takes some time to finish (not too long but is definitely not instant).
> 
> I think the time constraints are more of a question of practicality.
> A developer should be able to run the tests as part of their workflow.
> 

That makes sense - agree. So Jakub, if I understand correctly we have a few
different problems to solve here:
1) how to deploy the module:
 - now it's separated, we should consider e.g. netdevsim
2) if we should have those tests a part of selftests
 - I would remove it from selftests and move it to ./tools/testing
3) if we should use Python at all:
 - fast to develop and easy to maintain
 - might be problematic to deploy (no Python, VMs, embedded, no network etc.)
 
Do I understand our current outcome of the discussion correctly?

>> If you asked me, I would consider those tests to be part of the kernel.
>> I am not sure if they should be a part of selftests, though. Maybe a
>> reasonable approach would be to have have my tests being a "thorough
>> integration tests" and at the same time some limited tests shipped as a
>> selftests binary? I can also parametrically limit the scope of my tests
>> to run faster in selftests (e.g. only one scenario) and having possibility
>> to run extensive tests on demand?
>> 
>> Thanks,
>> M^2
>> 
>> [1] https://docs.python.org/3/library/dataclasses.html
>> [2] https://www.kernel.org/doc/html/v5.0/dev-tools/kselftest.html
>

Powered by blists - more mailing lists