netdev - Re: [PATCH net-next 3/4] selftests: kselftest

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Wed, 14 Feb 2024 16:25:14 -0800
From: Jakub Kicinski <kuba@...nel.org>
To: Jakub Sitnicki <jakub@...udflare.com>, keescook@...omium.org
Cc: shuah@...nel.org, linux-kselftest@...r.kernel.org,
 netdev@...r.kernel.org
Subject: Re: [PATCH net-next 3/4] selftests: kselftest_harness: support
 using xfail

On Wed, 14 Feb 2024 22:46:46 +0100 Jakub Sitnicki wrote:
> > On second thought, if I can suggest a follow up change so this:
> >
> > ok 17 # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> >
> > ... becomes this
> >
> > ok 17 ip_local_port_range.ip4_stcp.late_bind # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> >
> > You see, we parse test results if they are in TAP format. Lack of test
> > name for xfail'ed and skip'ed tests makes it difficult to report in CI
> > which subtest was it. Happy to contribute it, once this series gets
> > applied.  
> 
> Should have said "harder", not "difficult". That was an overstatement.
> 
> Test name can be extracted from diagnostic lines preceeding the status.
> 
> #  RUN           ip_local_port_range.ip4_stcp.late_bind ...
> #      XFAIL      SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> #            OK  ip_local_port_range.ip4_stcp.late_bind
> ok 17 ip_local_port_range.ip4_stcp.late_bind # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
> 
> It just makes the TAP parser easier if the test name is included on the
> status line. That would be the motivation here. Let me know what you
> think.

Good catch, I just copied what we do for skip and completely missed
this. As you said we'd report:

ok 17 # XFAIL SCTP doesn't support IP_BIND_ADDRESS_NO_PORT

and I think that's sort of closer to valid TAP than to valid KTAP
which always mentions test/test_case_name:

https://docs.kernel.org/dev-tools/ktap.html

We currently do the same thing for SKIP, e.g.:

#  RUN           ip_local_port_range.ip4_stcp.late_bind ...
#      SKIP      SCTP doesn't support IP_BIND_ADDRESS_NO_PORT
#            OK  ip_local_port_range.ip4_stcp.late_bind
ok 17 # SKIP SCTP doesn't support IP_BIND_ADDRESS_NO_PORT

I'm not sure if we can realistically do surgery on the existing print
helpers to add the test_name, because:

$ git grep 'ksft_test_result_*' | wc -l
915

That'd be a cruel patch to send.

But I do agree that adding the test_name to the prototype is a good
move, to avoid others making the same mistake. Should we introduce
a new set of helpers which take the extra arg and call them
ksft_test_report_*() instead of ksft_test_result_*() ?

Maybe we're overthinking and a local fix in the harness is enough.

Kees, WDYT?