netdev - Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <e5c95174-2f6f-439d-b557-6e223f982de5@kernel.org>
Date: Thu, 13 Nov 2025 10:59:49 +0100
From: Matthieu Baerts <matttbe@...nel.org>
To: Hangbin Liu <liuhangbin@...il.com>
Cc: Donald Hunter <donald.hunter@...il.com>, netdev@...r.kernel.org,
 Jakub Kicinski <kuba@...nel.org>, "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>,
 Simon Horman <horms@...nel.org>, Jan Stancek <jstancek@...hat.com>,
 Asbjørn Sloth Tønnesen <ast@...erby.net>,
 Stanislav Fomichev <sdf@...ichev.me>, Ido Schimmel <idosch@...dia.com>,
 Guillaume Nault <gnault@...hat.com>, Sabrina Dubroca <sd@...asysnail.net>,
 Petr Machata <petrm@...dia.com>
Subject: Re: [PATCHv3 net-next 3/3] tools: ynl: add YNL test framework

On 13/11/2025 10:51, Hangbin Liu wrote:
> On Thu, Nov 13, 2025 at 10:21:41AM +0100, Matthieu Baerts wrote:
>>>>> +	if [ $$failed -eq 0 ]; then \
>>>>> +		echo "All tests passed!"; \
>>>>> +	else \
>>>>> +		echo "$$failed test(s) failed!"; \
>>>>
>>>> AFAICS this will never be reported since the scripts only ever exit 0.
>>>> The message is also a bit misleading since it would be the count of
>>>> scripts that failed, not individual tests.
>>>>
>>>> It would be great if the scripts exited with the number of test failures
>>>> so the make file could report a total.
>>>
>>> Oh, BTW, do you think if we should exit with the failed test number or just
>>
>> I know these new tests are not in the selftests, but maybe "safer" to
>> keep the same exit code to avoid being misinterpreted?
>>
>>   KSFT_PASS=0
>>   KSFT_FAIL=1
>>   KSFT_XFAIL=2
>>   KSFT_XPASS=3
>>   KSFT_SKIP=4
> 
> Yes, that's why I ask about the return code. I also prefer use the same exit
> code with selftest.

Should you then exit with rc=4 instead of 0 in case of SKIP?

>> If there is a need to know which tests have failed, why not using (K)TAP
>> format for the output?
> 
> I feel it's too heavy to copy the (K)TAP format here. I would just using the
> exit code unless Jakub ask to using the specific output format

OK, I thought it was just a question of changing your 'echo' from "PASS"
and "FAIL" to "(not )ok ${COUNTER} (...)" + print the header and the
number of tests. But sure, if this format is not needed, no need to bother.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.