[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <66d1e32558532_3c08a22949e@willemb.c.googlers.com.notmuch>
Date: Fri, 30 Aug 2024 11:20:05 -0400
From: Willem de Bruijn <willemdebruijn.kernel@...il.com>
To: Jakub Kicinski <kuba@...nel.org>,
Willem de Bruijn <willemdebruijn.kernel@...il.com>
Cc: Paolo Abeni <pabeni@...hat.com>,
netdev@...r.kernel.org,
davem@...emloft.net,
edumazet@...gle.com,
ncardwell@...gle.com,
shuah@...nel.org,
linux-kselftest@...r.kernel.org,
fw@...len.de,
Willem de Bruijn <willemb@...gle.com>,
"Matthieu Baerts (NGI0)" <matttbe@...nel.org>,
martineau@...nel.org
Subject: Re: [PATCH net-next RFC] selftests/net: integrate packetdrill with
ksft
Jakub Kicinski wrote:
> On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote:
> > That could work.
> >
> > Is reporting one KTAP and exitcode per directory vs per packetdrill
> > invocation good/bad/neither?
>
> To me "neither", hopefully and unhelpfully I won't tell you anything
> that's not common sense :) The balance is between:
>
> - having test cases which don't take too long (assuming debug kernel):
> <15min is good,
> >1h I will start complaining,
> >1h30m is bad because we can't retry and still make the 3h
> deadline that NIPA has,
> >3h the test can't run in NIPA at all.
>
> vs
>
> - flip side is having so many cases it's hard to keep track and render
> in the UI. JSON is relatively slow to process. If you have 150 cases,
> that's 300 per branch (debug and non-debug kernels), times 8
> branches a day => 2.4k results / day
> I think that's still fine-ish, but on the larger side for sure. For
> reference net and forwarding have ~100 tests each. FWIW we do have
> the ability to collect and display nested KTAP so the information is
> not lost (but it makes queries slower so we don't fetch it by default).
>
> > Three other issues if this is calling packetdrill directly is
> > - passing the non-trivial IP specific flags
> > - running twice, for IPv4 and IPv6
> > - chdir into the directory of the pkt file
> >
> > That can be addressed by instead calling a small wrapper shell script.
> >
> > That would do the test_func_builder part of packetdrill_ksft.py.
> > But without the need to handle netns, popen/cmd, etc, and thus the
> > ksft dependencies.
>
> Right!
Implemented this, but hit a snag
Kselftest install does not preserve directories.
So all .pkt files are copied into net/packetdrill root. This is messy.
More fundamentally it breaks the includes in the files (e..g, `source
../common/defaults.sh`).
The output of having each test separate is also quite unreadable.
An alternative is to add each subdirectory as a separate TARGET.
But that seems similarly impractical. And does nothing to group
KTAP output.
In practice, most directories have a handful of .pkt scripts, and
each script is fast, so serialization is not a huge issue.
I found tools/testing/selftests/net/kselftest/ktap_helpers.sh,
which has helpful KTAP boilerplate for shell tests, including
KSFT exit codes. Am using that instead of python, to reduce the
external dependencies.
Powered by blists - more mailing lists