linux-kernel - Re: [PATCH v2] selftests: openat2: don't print total number of tests and then skip

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2e1509f5-2a10-4876-8619-6a6b9c29d269@linuxfoundation.org>
Date: Thu, 1 Aug 2024 11:27:33 -0600
From: Shuah Khan <skhan@...uxfoundation.org>
To: Muhammad Usama Anjum <usama.anjum@...labora.com>,
 Shuah Khan <shuah@...nel.org>
Cc: Aleksa Sarai <cyphar@...har.com>, kernel@...labora.com,
 linux-kselftest@...r.kernel.org, linux-kernel@...r.kernel.org,
 Shuah Khan <skhan@...uxfoundation.org>
Subject: Re: [PATCH v2] selftests: openat2: don't print total number of tests
 and then skip

On 8/1/24 10:27, Shuah Khan wrote:
> On 8/1/24 02:42, Muhammad Usama Anjum wrote:
>> On 7/31/24 9:57 PM, Shuah Khan wrote:
>>> On 7/31/24 07:39, Muhammad Usama Anjum wrote:
>>>> Don't print that 88 sub-tests are going to be executed, but then skip.
>>>> This is against TAP compliance. Instead check pre-requisites first
>>>> before printing total number of tests.
>>>
>>> Does TAP clearly mention this?
>> Yes from https://testanything.org/tap-version-13-specification.html
>>
>> Skipping everything
>> This listing shows that the entire listing is a skip. No tests were run.
>>
>> TAP version 13
>> 1..0 # skip because English-to-French translator isn't installed
> 

One more thing on TAP compliance - we don't strive to be TAP compliant
as it doesn't meet our special needs.

It is important to keep the how many tests could be run to improve testing
coverage.

Refer to: https://www.kernel.org/doc/html/latest/dev-tools/ktap.html

"The Linux Kernel largely uses TAP output for test results. However, Kernel
testing frameworks have special needs for test results which don’t align with
the original TAP specification. Thus, a “Kernel TAP” (KTAP) format is specified
to extend and alter TAP to support these use-cases. This specification describes
the generally accepted format of KTAP as it is currently used in the kernel."

I appreciate the effort you are putting into improving the reports. I request
that you refer to the above document and also keep in mind what would help us
improve our testing over focusing just on reports and compliance.

thanks,
-- Shuah