netdev - Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <e63b88ca-ba6b-4a6f-9a57-8d3b2e8c5de2@kernel.org>
Date: Thu, 13 Nov 2025 10:21:41 +0100
From: Matthieu Baerts <matttbe@...nel.org>
To: Hangbin Liu <liuhangbin@...il.com>,
 Donald Hunter <donald.hunter@...il.com>
Cc: netdev@...r.kernel.org, Jakub Kicinski <kuba@...nel.org>,
 "David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
 Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>,
 Jan Stancek <jstancek@...hat.com>, Asbjørn Sloth Tønnesen <ast@...erby.net>, Stanislav Fomichev <sdf@...ichev.me>,
 Ido Schimmel <idosch@...dia.com>, Guillaume Nault <gnault@...hat.com>,
 Sabrina Dubroca <sd@...asysnail.net>, Petr Machata <petrm@...dia.com>
Subject: Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

Hi Hangbin,

On 13/11/2025 07:06, Hangbin Liu wrote:
> On Tue, Nov 11, 2025 at 11:51:38AM +0000, Donald Hunter wrote:
>>> diff --git a/tools/net/ynl/tests/Makefile b/tools/net/ynl/tests/Makefile
>>> new file mode 100644
>>> index 000000000000..4d527f9c3de9
>>> --- /dev/null
>>> +++ b/tools/net/ynl/tests/Makefile
>>> @@ -0,0 +1,38 @@
>>> +# SPDX-License-Identifier: GPL-2.0
>>> +# Makefile for YNL tests
>>> +
>>> +TESTS := \
>>> +	test_ynl_cli.sh \
>>> +	test_ynl_ethtool.sh \
>>> +# end of TESTS
>>> +
>>> +all: $(TESTS)
>>> +
>>> +run_tests:
>>> +	@echo "Running YNL tests..."
>>> +	@failed=0; \
>>> +	echo "Running test_ynl_cli.sh..."; \
>>> +	./test_ynl_cli.sh || failed=$$(($$failed + 1)); \
>>> +	echo "Running test_ynl_ethtool.sh..."; \
>>> +	./test_ynl_ethtool.sh || failed=$$(($$failed + 1)); \
>>
>> This could iterate through $(TESTS) instead of being hard coded.
>>
>>> +	if [ $$failed -eq 0 ]; then \
>>> +		echo "All tests passed!"; \
>>> +	else \
>>> +		echo "$$failed test(s) failed!"; \
>>
>> AFAICS this will never be reported since the scripts only ever exit 0.
>> The message is also a bit misleading since it would be the count of
>> scripts that failed, not individual tests.
>>
>> It would be great if the scripts exited with the number of test failures
>> so the make file could report a total.
> 
> Oh, BTW, do you think if we should exit with the failed test number or just

I know these new tests are not in the selftests, but maybe "safer" to
keep the same exit code to avoid being misinterpreted?

  KSFT_PASS=0
  KSFT_FAIL=1
  KSFT_XFAIL=2
  KSFT_XPASS=3
  KSFT_SKIP=4

If there is a need to know which tests have failed, why not using (K)TAP
format for the output?

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.