Message-ID: <b15906c4-3cd0-481f-8f8b-3dc3e581d817@gmail.com>
Date: Wed, 23 Apr 2025 11:32:47 +0530
From: "Nirjhar Roy (IBM)" <nirjhar.roy.lists@...il.com>
To: Dave Chinner <david@...morbit.com>
Cc: Theodore Ts'o <tytso@....edu>, fstests@...r.kernel.org,
 linux-ext4@...r.kernel.org, linux-xfs@...r.kernel.org,
 ritesh.list@...il.com, ojaswin@...ux.ibm.com, djwong@...nel.org,
 zlang@...nel.org
Subject: Re: [PATCH v2 2/3] check: Add -q <n> option to support unconditional
 looping.


On 4/16/25 04:58, Dave Chinner wrote:
> On Tue, Apr 15, 2025 at 01:02:49PM +0530, Nirjhar Roy (IBM) wrote:
>> On 4/14/25 03:18, Theodore Ts'o wrote:
>>> On Thu, Apr 03, 2025 at 08:58:19AM +0000, Nirjhar Roy (IBM) wrote:
>>>> This patch adds -q <n> option through which one can run a given test <n>
>>>> times unconditionally. It also prints pass/fail metrics at the end.
>>>>
>>>> The advantage of this over -L <n> and -i/-I <n> is that:
>>>>       a. -L <n> will not re-run a flakey test if the test passes for the first time.
>>>>       b. -I/-i <n> sets up devices during each iteration and hence slower.
>>>> Note -q <n> will override -L <n>.
>>> I'm wondering if we need to keep the current behavior of -I/-i.  The
>>> primary difference between them and how your proposed -q works is that
>>> instead of iterating over the section, your proposed option iterates
>>> over each test.  So for example, if a section contains generic/001 and
>>> generic/002, iterating using -i 3 will do this:
>> Yes, the motivation to introduce -q was to:
>>
>> 1. Make the re-run faster and not re-format the device. -i re-formats the
>> device and hence is slightly slower.
> Why does -i reformat the test device on every run in your setup?
> i.e. if the FSTYP is not changing from iteration to iteration, then
> each iteration should not reformat the test device at all. Unless, of
> course, you have told it to do so via the RECREATE_TEST_DEV env
> variable....

No, it doesn't re-format the test device. It re-formats the scratch 
device. With -q, there will be no re-formatting of the scratch device too.

>
> Hence it seems to me like this is working around some other setup or
> section iteration problem here...
>
>> 2. To unconditionally loop a test - useful for scenarios when a flaky test
>> doesn't fail for the first time (something that -L) does.
> That's what -i does. it will unconditionally loop over the specified
> tests N times regardless of success or failure.
>
> OTOH, -I will abort on first failure. i.e. to enable flakey tests
> to be run until it eventually fails and leave the corpse behind for
> debugging.
>
>> So, are saying that re-formatting a disk on every run, something that -i
>> does, doesn't have much value and can be removed?
> -i does not imply that the test device should be reformatted on
> every loop. If that is happening, that is likely a result of test
> config or environment conditions.
>
> Can you tell us why the test device is getting reformatted on every
> iteration in your setup?

As mentioned above, -i isn't reformatting our test device. It is 
re-formatting scratch device and we introduced -q to unconditionally 
loop without even reformatting the scratch device, and hence making the 
re-runs faster.

--NR

>
>>> generic/001
>>> generic/002
>>> generic/001
>>> generic/002
>>> generic/001
>>> generic/002
>>>
>>> While generic -q 3 would do this instead:
>>>
>>> generic/001
>>> generic/001
>>> generic/001
>>> generic/002
>>> generic/002
>>> generic/002
> There are arguments both for and against the different iteration
> orders. However, if there is no overriding reason to change the
> existing order of test execution, then we should not change the
> order or test execution....
>
> -Dave.

-- 
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore

Powered by blists - more mailing lists