netdev - Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aRV1VZ6Z-tzbDlLH@fedora>
Date: Thu, 13 Nov 2025 06:06:13 +0000
From: Hangbin Liu <liuhangbin@...il.com>
To: Donald Hunter <donald.hunter@...il.com>
Cc: netdev@...r.kernel.org, Jakub Kicinski <kuba@...nel.org>,
	"David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>,
	Simon Horman <horms@...nel.org>, Jan Stancek <jstancek@...hat.com>,
	"Matthieu Baerts (NGI0)" <matttbe@...nel.org>,
	Asbjørn Sloth Tønnesen <ast@...erby.net>,
	Stanislav Fomichev <sdf@...ichev.me>,
	Ido Schimmel <idosch@...dia.com>,
	Guillaume Nault <gnault@...hat.com>,
	Sabrina Dubroca <sd@...asysnail.net>,
	Petr Machata <petrm@...dia.com>
Subject: Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

On Tue, Nov 11, 2025 at 11:51:38AM +0000, Donald Hunter wrote:
> > diff --git a/tools/net/ynl/tests/Makefile b/tools/net/ynl/tests/Makefile
> > new file mode 100644
> > index 000000000000..4d527f9c3de9
> > --- /dev/null
> > +++ b/tools/net/ynl/tests/Makefile
> > @@ -0,0 +1,38 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Makefile for YNL tests
> > +
> > +TESTS := \
> > +	test_ynl_cli.sh \
> > +	test_ynl_ethtool.sh \
> > +# end of TESTS
> > +
> > +all: $(TESTS)
> > +
> > +run_tests:
> > +	@echo "Running YNL tests..."
> > +	@failed=0; \
> > +	echo "Running test_ynl_cli.sh..."; \
> > +	./test_ynl_cli.sh || failed=$$(($$failed + 1)); \
> > +	echo "Running test_ynl_ethtool.sh..."; \
> > +	./test_ynl_ethtool.sh || failed=$$(($$failed + 1)); \
> 
> This could iterate through $(TESTS) instead of being hard coded.
> 
> > +	if [ $$failed -eq 0 ]; then \
> > +		echo "All tests passed!"; \
> > +	else \
> > +		echo "$$failed test(s) failed!"; \
> 
> AFAICS this will never be reported since the scripts only ever exit 0.
> The message is also a bit misleading since it would be the count of
> scripts that failed, not individual tests.
> 
> It would be great if the scripts exited with the number of test failures
> so the make file could report a total.

Oh, BTW, do you think if we should exit with the failed test number or just
report failed number in test and exit 1. i.e.

Option 1:

test 1 (pass)
test 2 (fail)
test 3 (fail)
exit 2

Option 2:

test 1 (pass)
test 2 (fail)
test 3 (fail)
echo "2 test failed"
exit 1

With Option 1, we will summary all the failure number in the Makefile.
With Option 2, we will report failure number in each test, and report failed
*test script* number in Makefile.

Thanks
Hangbin