linux-kernel - Re: [PATCH] selftests: Add information about TAP conformance in tests

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <7588c2a8-27c3-4b1c-bebd-9c3b099c8b80@collabora.com>
Date: Wed, 12 Jun 2024 12:27:10 +0500
From: Muhammad Usama Anjum <usama.anjum@...labora.com>
To: Shuah Khan <skhan@...uxfoundation.org>, Shuah Khan <shuah@...nel.org>,
 Jonathan Corbet <corbet@....net>
Cc: Muhammad Usama Anjum <usama.anjum@...labora.com>, kernel@...labora.com,
 linux-kselftest@...r.kernel.org, linux-doc@...r.kernel.org,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH] selftests: Add information about TAP conformance in tests

On 6/12/24 1:27 AM, Shuah Khan wrote:
> On 6/10/24 02:32, Muhammad Usama Anjum wrote:
>> Although "TAP" word is being used already in documentation, but it hasn't
>> been defined in informative way for developers that how to write TAP
>> conformant tests and what are the benefits. Write a short brief about it.
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@...labora.com>
>> ---
>>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst
>> b/Documentation/dev-tools/kselftest.rst
>> index dcf634e411bd9..b579f491f3e97 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -228,6 +228,14 @@ In general, the rules for selftests are
>>    * Don't cause the top-level "make run_tests" to fail if your feature is
>>      unconfigured.
>>   + * The output of tests must conform to the TAP standard to ensure high
>> +   testing quality and to capture failures/errors with specific details.
>> +   The kselftest.h and kselftest_harness.h headers provide wrappers for
>> +   outputting test results such as pass, fail, or skip etc. These wrappers
>> +   should be used instead of reinventing the wheel or using raw printf and
> 
> When there are multiple tests supported by a test suite, it doesn't
> make sense to convert every single print statement. That is the reason
> why we have the run_tests wrapper that prints the summary in TAP format.
> 
> I would rephrase this to say "should be used for pass, fail, exit, and
> skip messages".
Makes sense. I'll update.

> 
> This rule is applicable to only the tests that are in the selftests Makefile
> and included in the default run.
> 
> There is no need to convert every single print statement in tests.
> 
>> +   exit statements. CI systems can easily parse TAP output messages to
>> +   detect test failures.
> 
> I would rather see people spending energy writing new tests and fixing
> warnings. TAP conversions need to be focused if they add value.
Some tests don't print pass/fail and just return status by exit value.
Adding pass/fail statements there really add value by exposing what went
wrong. So it can be reproduced in debug environment to fix it.

> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum