[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <c66ab6ff-c9e1-4935-95a5-67e8b83962a3@collabora.com>
Date: Fri, 2 Aug 2024 10:40:01 +0500
From: Muhammad Usama Anjum <usama.anjum@...labora.com>
To: Shuah Khan <skhan@...uxfoundation.org>, Shuah Khan <shuah@...nel.org>
Cc: Muhammad Usama Anjum <usama.anjum@...labora.com>,
Aleksa Sarai <cyphar@...har.com>, kernel@...labora.com,
linux-kselftest@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2] selftests: openat2: don't print total number of tests
and then skip
On 8/1/24 10:27 PM, Shuah Khan wrote:
> On 8/1/24 10:27, Shuah Khan wrote:
>> On 8/1/24 02:42, Muhammad Usama Anjum wrote:
>>> On 7/31/24 9:57 PM, Shuah Khan wrote:
>>>> On 7/31/24 07:39, Muhammad Usama Anjum wrote:
>>>>> Don't print that 88 sub-tests are going to be executed, but then skip.
>>>>> This is against TAP compliance. Instead check pre-requisites first
>>>>> before printing total number of tests.
>>>>
>>>> Does TAP clearly mention this?
>>> Yes from https://testanything.org/tap-version-13-specification.html
>>>
>>> Skipping everything
>>> This listing shows that the entire listing is a skip. No tests were run.
>>>
>>> TAP version 13
>>> 1..0 # skip because English-to-French translator isn't installed
>>
>
> One more thing on TAP compliance - we don't strive to be TAP compliant
> as it doesn't meet our special needs.
>
> It is important to keep the how many tests could be run to improve testing
> coverage.
>
> Refer to: https://www.kernel.org/doc/html/latest/dev-tools/ktap.html
>
> "The Linux Kernel largely uses TAP output for test results. However, Kernel
> testing frameworks have special needs for test results which don’t align with
> the original TAP specification. Thus, a “Kernel TAP” (KTAP) format is
> specified
> to extend and alter TAP to support these use-cases. This specification
> describes
> the generally accepted format of KTAP as it is currently used in the kernel."
>
> I appreciate the effort you are putting into improving the reports. I request
> that you refer to the above document and also keep in mind what would help us
> improve our testing over focusing just on reports and compliance.
That makes sense. I'll work on improving the testing going forward.
>
> thanks,
> -- Shuah
>
--
BR,
Muhammad Usama Anjum
Powered by blists - more mailing lists