linux-kernel - Re: [PATCH v5 1/2] selftests: net: Create veth pair for testing in networkless kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20240809211911.1445c965@kernel.org>
Date: Fri, 9 Aug 2024 21:19:11 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: Abhinav Jain <jain.abhinav177@...il.com>
Cc: davem@...emloft.net, edumazet@...gle.com,
 javier.carrasco.cruz@...il.com, linux-kernel@...r.kernel.org,
 linux-kselftest@...r.kernel.org, netdev@...r.kernel.org, pabeni@...hat.com,
 shuah@...nel.org, skhan@...uxfoundation.org
Subject: Re: [PATCH v5 1/2] selftests: net: Create veth pair for testing in
 networkless kernel

On Fri,  9 Aug 2024 16:53:26 +0000 Abhinav Jain wrote:
> On Thu, 8 Aug 2024 09:23:09 -0700, Jakub Kicinski wrote:
> > A number of checks now return SKIP because veth doesn't support all
> > ethtool APIs.
> >
> > In netdev selftests we try to make sure SKIP is only used when test
> > cannot be performed because of limitations of the environment.
> > For example some tool is not installed, kernel doesn't have a config.
> > Something that the person running the test is able to fix by fixing
> > how the test is run.
> >
> > Running this test on veth will always SKIP, nothing CI system can do.
> > Please make the test use the keyword XFAIL instead of SKIP when
> > functionality is not supported by the underlying driver.  
> 
> Ack, understood. I will do that, one clarification though.
> Currently, the tests are using either PASS or FAIL and no SKIP. Based on
> the above suggestion, it seems that I have replace FAIL with XFAIL for all
> the tests that fail due to functionality not being supported by the
> underlying driver.

Right, sorry for lack of clarity.

Our CI doesn't fully trust the exit codes, so even though the test
exits with zero the CI parses the output and finds the "SKIP: ..."
lines. You need to replace those "SKIP"s in the output with "XFAIL".