netdev - Re: [PATCH RFC net-next v4 2/2] selftests/dpll: add DPLL system integration selftests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Wed, 29 Nov 2023 09:39:51 -0800
From: Jakub Kicinski <kuba@...nel.org>
To: Michal Michalik <michal.michalik@...el.com>
Cc: netdev@...r.kernel.org, vadim.fedorenko@...ux.dev,
 arkadiusz.kubalewski@...el.com, jonathan.lemon@...il.com,
 pabeni@...hat.com, poros@...hat.com, milena.olech@...el.com,
 mschmidt@...hat.com, linux-clk@...r.kernel.org, bvanassche@....org,
 davem@...emloft.net, edumazet@...gle.com
Subject: Re: [PATCH RFC net-next v4 2/2] selftests/dpll: add DPLL system
 integration selftests

On Thu, 23 Nov 2023 05:52:43 -0500 Michal Michalik wrote:
> The tests are written in Python3 (3.7+) and pytest testing framework.
> Framework is basing on the ynl library available in the kernel tree
> at: tools/net/ynl

LGTM!

Somewhat tangential question, a nit, and a comment..
 
> The DPLL system integration tests are meant to be part of selftests, so
> they can be build and run using command:
>   make -C tools/testing/selftests
> 
> Alternatively, they can be run using single command [1]:
>   make kselftest
> 
> If we want to run only DPLL tests, we should set the TARGETS variable:
>   make -C tools/testing/selftests TARGETS=drivers/net/netdevsim/dpll
> 
> They can also be run standalone using starter script:
>   ./run_dpll_tests.sh
> 
> There is a possibliy to set optional PYTEST_PARAMS environment variable
> to set the pytest options, like tests filtering ("-k <filter>") or
> verbose output ("-v").
> 
> [1] https://www.kernel.org/doc/html/v5.0/dev-tools/kselftest.html

nit: s/v5.0/v6.6/ ? Or /v5.0/latest/

Did you try to run it in vmtest or virtme-ng?
https://www.youtube.com/watch?v=NT-325hgXjY
https://lpc.events/event/17/contributions/1506/attachments/1143/2441/virtme-ng.pdf

I'm thinking of using those for continuous testing, curious all 
the Python setup works okay with them.

> +@...est.fixture(scope="class", params=((0,), (1, 0), (0, 1)))

We have both uses of pytest and unittest in the kernel:

$ git grep --files-with-matches '^import .*unittest'
scripts/rust_is_available_test.py
tools/crypto/ccp/test_dbc.py
tools/perf/pmu-events/metric_test.py
tools/testing/kunit/kunit_tool_test.py
tools/testing/selftests/bpf/test_bpftool.py
tools/testing/selftests/tpm2/tpm2.py
tools/testing/selftests/tpm2/tpm2_tests.py

$ git grep --files-with-matches '^import .*pytest'
scripts/kconfig/tests/conftest.py
tools/testing/selftests/drivers/sdsi/sdsi.sh
tools/testing/selftests/drivers/sdsi/sdsi_test.py
tools/testing/selftests/hid/tests/base.py
tools/testing/selftests/hid/tests/conftest.py
tools/testing/selftests/hid/tests/test_gamepad.py
tools/testing/selftests/hid/tests/test_mouse.py
tools/testing/selftests/hid/tests/test_multitouch.py
tools/testing/selftests/hid/tests/test_sony.py
tools/testing/selftests/hid/tests/test_tablet.py
tools/testing/selftests/hid/tests/test_usb_crash.py
tools/testing/selftests/hid/tests/test_wacom_generic.py

unittest seems a bit more popular but pytest does seem like
a better fit indeed.

Did you see what the sdsi test does? It seems to assume everything 
is installed locally, without the venv. I wonder if that may be simpler
to get going with vmtest?