netdev - RE: [PATCH RFC net-next v1 2/2] selftests/dpll: add DPLL system integration selftests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CH3PR11MB84141E0EDA588B84F7E10F71E31EA@CH3PR11MB8414.namprd11.prod.outlook.com>
Date: Mon, 21 Aug 2023 09:32:56 +0000
From: "Michalik, Michal" <michal.michalik@...el.com>
To: Jakub Kicinski <kuba@...nel.org>
CC: "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"vadim.fedorenko@...ux.dev" <vadim.fedorenko@...ux.dev>, "jiri@...nulli.us"
	<jiri@...nulli.us>, "Kubalewski, Arkadiusz" <arkadiusz.kubalewski@...el.com>,
	"jonathan.lemon@...il.com" <jonathan.lemon@...il.com>, "pabeni@...hat.com"
	<pabeni@...hat.com>, poros <poros@...hat.com>, "Olech, Milena"
	<milena.olech@...el.com>, mschmidt <mschmidt@...hat.com>,
	"linux-clk@...r.kernel.org" <linux-clk@...r.kernel.org>, "bvanassche@....org"
	<bvanassche@....org>
Subject: RE: [PATCH RFC net-next v1 2/2] selftests/dpll: add DPLL system
 integration selftests

On 18 August 2023 11:08 PM CEST, Jakub Kicinski wrote:
> 
> On Thu, 17 Aug 2023 17:22:09 +0200 Michal Michalik wrote:
>> High level flow of DPLL subsystem integration selftests:
>> (after running run_dpll_tests.sh or 'make -C tools/testing/selftests')
>> 1) check if Python in correct version is installed,
>> 2) create temporary Python virtual environment,
>> 3) install all the required libraries,
>> 4) run the tests,
>> 5) do cleanup.
> 
> How fragile do you reckon this setup will be?
> I mean will it work reliably across distros and various VM setups?
> I have tried writing tests based on ynl.py and the C codegen, and
> I can't decide whether the python stuff is easy enough to deploy.
> Much easier to scp over to the test host a binary based on the 
> C code. But typing tests in python is generally quicker...
> What are your thoughts?
> 
> Thanks for posting the tests!

Hi Jakub,

First of all - everything I'll write is just my opinion. While having
quite a bit Python experience, I can't speak confidently about the
target systems and hardware (architectures) it would be ran against and
what might be possible problems around that. I need your help here.

>From my point of view, all we need is a Python 3.7 and access to PyPI
repositories. This version of Python is 5 years old, but if we need
support of even older versions I can rewrite the code not to use
dataclasses[1] so we could go with even older version. Do you think it
is required to support platforms with no Python at all?

Another requirement is the toolchain for building the module, but I
assume if Python is not there in some embedded system - the build tools
are also not there...

I've seen other Python pytest code in the kernel repo - that is why I
thought it might be a good idea to propose something like that. Your
idea is also cool - binary is always superior. I am a strong advocate of
taking into consideration not only deployment ease, but also maintenance
and ease of read which might encourage community to help. I also see a
benefit of showing the sample implementation (my tested "dummy module").

My deployment is automatic and does not leave any garbage in the system
after tests (packages are installed into temporary virtual environment).
In case any of the requirements are not met - tests are skipped. I've
tested it both on real HW with some E810T cards installed and on fresh
VM - seems to work fine. But that of course require further verification.
Till now only Arek Kubalewski sucessfully gave those tests a shot.

The biggest concern for me is the requirement of selftests[2]:
  "Don't take too long;"
This approach is reloading the modules few times to check few scenarios.
Also, the DPLL subsystem is being tested against multiple requests - so
it takes some time to finish (not too long but is definitely not instant).

If you asked me, I would consider those tests to be part of the kernel.
I am not sure if they should be a part of selftests, though. Maybe a
reasonable approach would be to have have my tests being a "thorough
integration tests" and at the same time some limited tests shipped as a
selftests binary? I can also parametrically limit the scope of my tests
to run faster in selftests (e.g. only one scenario) and having possibility
to run extensive tests on demand?

Thanks,
M^2

[1] https://docs.python.org/3/library/dataclasses.html
[2] https://www.kernel.org/doc/html/v5.0/dev-tools/kselftest.html