netdev - Re: [PATCH net-next RFC] selftests/net: integrate packetdrill with ksft

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <66d1e32558532_3c08a22949e@willemb.c.googlers.com.notmuch>
Date: Fri, 30 Aug 2024 11:20:05 -0400
From: Willem de Bruijn <willemdebruijn.kernel@...il.com>
To: Jakub Kicinski <kuba@...nel.org>, 
 Willem de Bruijn <willemdebruijn.kernel@...il.com>
Cc: Paolo Abeni <pabeni@...hat.com>, 
 netdev@...r.kernel.org, 
 davem@...emloft.net, 
 edumazet@...gle.com, 
 ncardwell@...gle.com, 
 shuah@...nel.org, 
 linux-kselftest@...r.kernel.org, 
 fw@...len.de, 
 Willem de Bruijn <willemb@...gle.com>, 
 "Matthieu Baerts (NGI0)" <matttbe@...nel.org>, 
 martineau@...nel.org
Subject: Re: [PATCH net-next RFC] selftests/net: integrate packetdrill with
 ksft

Jakub Kicinski wrote:
> On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
> > That could work.
> > 
> > Is reporting one KTAP and exitcode per directory vs per packetdrill
> > invocation good/bad/neither?
> 
> To me "neither", hopefully and unhelpfully I won't tell you anything
> that's not common sense :) The balance is between:
> 
>  - having test cases which don't take too long (assuming debug kernel):
>      <15min is good, 
>      >1h I will start complaining,
>      >1h30m is bad because we can't retry and still make the 3h
>             deadline that NIPA has,
>      >3h the test can't run in NIPA at all.
> 
> vs
> 
>  - flip side is having so many cases it's hard to keep track and render
>    in the UI. JSON is relatively slow to process. If you have 150 cases,
>    that's 300 per branch (debug and non-debug kernels), times 8
>    branches a day => 2.4k results / day
>    I think that's still fine-ish, but on the larger side for sure. For
>    reference net and forwarding have ~100 tests each. FWIW we do have
>    the ability to collect and display nested KTAP so the information is
>    not lost (but it makes queries slower so we don't fetch it by default).
> 
> > Three other issues if this is calling packetdrill directly is
> > - passing the non-trivial IP specific flags
> > - running twice, for IPv4 and IPv6
> > - chdir into the directory of the pkt file
> > 
> > That can be addressed by instead calling a small wrapper shell script.
> > 
> > That would do the test_func_builder part of packetdrill_ksft.py.
> > But without the need to handle netns, popen/cmd, etc, and thus the
> > ksft dependencies.
> 
> Right!

Implemented this, but hit a snag

Kselftest install does not preserve directories.

So all .pkt files are copied into net/packetdrill root. This is messy.
More fundamentally it breaks the includes in the files (e..g, `source
../common/defaults.sh`).

The output of having each test separate is also quite unreadable.

An alternative is to add each subdirectory as a separate TARGET.
But that seems similarly impractical. And does nothing to group
KTAP output.

In practice, most directories have a handful of .pkt scripts, and
each script is fast, so serialization is not a huge issue.

I found tools/testing/selftests/net/kselftest/ktap_helpers.sh,
which has helpful KTAP boilerplate for shell tests, including
KSFT exit codes. Am using that instead of python, to reduce the
external dependencies.