netdev - Re: [PATCH v5 1/2] selftests: net: Create veth pair for testing in networkless kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <20240810175509.404094-1-jain.abhinav177@gmail.com>
Date: Sat, 10 Aug 2024 17:55:09 +0000
From: Abhinav Jain <jain.abhinav177@...il.com>
To: kuba@...nel.org
Cc: davem@...emloft.net,
	edumazet@...gle.com,
	jain.abhinav177@...il.com,
	javier.carrasco.cruz@...il.com,
	linux-kernel@...r.kernel.org,
	linux-kselftest@...r.kernel.org,
	netdev@...r.kernel.org,
	pabeni@...hat.com,
	shuah@...nel.org,
	skhan@...uxfoundation.org
Subject: Re: [PATCH v5 1/2] selftests: net: Create veth pair for testing in networkless kernel

On Fri, 9 Aug 2024 21:19:11 -0700, Jakub Kicinski wrote:

> > On Thu, 8 Aug 2024 09:23:09 -0700, Jakub Kicinski wrote:
> > > A number of checks now return SKIP because veth doesn't support all
> > > ethtool APIs.
> > >
> > > In netdev selftests we try to make sure SKIP is only used when test
> > > cannot be performed because of limitations of the environment.
> > > For example some tool is not installed, kernel doesn't have a config.
> > > Something that the person running the test is able to fix by fixing
> > > how the test is run.
> > >
> > > Running this test on veth will always SKIP, nothing CI system can do.
> > > Please make the test use the keyword XFAIL instead of SKIP when
> > > functionality is not supported by the underlying driver.  
> > 
> > Ack, understood. I will do that, one clarification though.
> > Currently, the tests are using either PASS or FAIL and no SKIP. Based on
> > the above suggestion, it seems that I have replace FAIL with XFAIL for all
> > the tests that fail due to functionality not being supported by the
> > underlying driver.
>
> Right, sorry for lack of clarity.
>
> Our CI doesn't fully trust the exit codes, so even though the test
> exits with zero the CI parses the output and finds the "SKIP: ..."
> lines. You need to replace those "SKIP"s in the output with "XFAIL".

I re-tested and found that currently only two APIs are tested, "dump" and
"stats". For veth pair, the only test that fails currently with a SKIP is
the dump operation.

```
# Cannot get register dump: Operation not supported
# SKIP: veth1: ethtool dump not supported
```

This is present in kci_netdev_ethtool_test function, please confirm if this
is the one that I need to change to XFAIL. The logic to incorporate
the failure exit code for dump operation (74) is already in place.

```
kci_netdev_ethtool_test 74 'dump' "ethtool -d $netdev"
```

I just need to change the SKIP in this function to XFAIL. In case, if you
were referring to any other tests that are failing (features on/off),
please let me know. Thank you for prompt feedbacks, really appreciated.