Message-Id: <03C8F5B9-B2E1-4C78-A043-B4F9422508B7@zazolabs.com>
Date: Fri, 23 Jan 2026 17:00:33 +0200
From: Alexander Atanasov <alex@...olabs.com>
To: Ming Lei <ming.lei@...hat.com>
Cc: Shuah Khan <shuah@...nel.org>,
 linux-block@...r.kernel.org,
 linux-kselftest@...r.kernel.org,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH] selftests: ublk: use tmpdir for scratch files and improve
 relative paths use



> On 23 Jan 2026, at 16:33, Ming Lei <ming.lei@...hat.com> wrote:
> 
> On Fri, Jan 23, 2026 at 03:59:31PM +0200, Alexander Atanasov wrote:
>> On 23 Jan 2026, at 15:33, Ming Lei <ming.lei@...hat.com> wrote:
>>> 
>>> On Fri, Jan 23, 2026 at 11:20:36AM +0000, Alexander Atanasov wrote:
>>>> Create a temp dir for temporary files and use it instead of
>>>> placing them inside source tree.
>>> 
>>> Many temporary files are backing files of file storage target, so far
>>> the code requires O_DIRECT, or the size could be a bit big.
>>> 
>>> In case of ramfs/tmpfs of temp dir, it may cause problem for tests.
>>> 
>> 
>> I am aware of O_DIRECT problem but you can export different TMPDIR that has working O_DIRECT.
> 
> Can you share how to export TMPDIR capable of O_DIRECT?


+export TMPDIR=$(mktemp -d ${TMPDIR:-/tmp}/ublktest-dir.XXXXXX)

I made the tests to run in own TMPDIR. Which is under already set TMPDIR or
if TMPDIR is not set it is defaults to  /tmp.

export TMPDIR=/path/to/odirect/capable
Before running and tests will run in:
/path/to/odirect/capable/ublktest-dir.XXXXXX


> 
>> 
>> I use sshfs mount of the build to run the tests and that is a problem sshfs/fuse does not
>> do O_DIRECT too.
>> 
>> I think test_generic_06.sh is the only one that fails due to this(thou I still have to investigate).
>> 
>> If O_DIRECT is required by the tests it may be possible to go thru a RAM disk which does support it,
>> so it works eveerywhere
>> 
>> Other option is to preserve working in source tree as it is now, and just add a variable to specify working directory -
>> UBLK_TMPDIR or something.
>> 
>> 
>> I get a lot of out of order io - between 0 and 10 on average on my test setup:
>> tools/testing/selftests/ublk/test_generic_01.sh 
>> Attached 3 probes
>> io_out_of_order: exp 564688 actual 564648
>> io_out_of_order: exp 564648 actual 565584
>> io_out_of_order: exp 565584 actual 564688
>> io_out_of_order: exp 565592 actual 564688
>> io_out_of_order: exp 566328 actual 565592
>> io_out_of_order: exp 882256 actual 882248
>> io_out_of_order: exp 883032 actual 882912
>> io_out_of_order: exp 882912 actual 883040
>> io_out_of_order: exp 883040 actual 883032
>> 
>> 
>> generic_01 : [FAIL]
>> 
>> All rq-s are there just reordered , AFAIK blk-mq does not guarantee that requests will be completed in order, what’s the idea to catch this and
> 
> If there is just 0 ~ 10, it could be fine. But if all are reorderd,
> something must be wrong. One improvement could be check if there is too
> many reorder...
> 
> Actually what I am trying to test is to make sure same order is observed
> from both ublk driver dispatch code path and ublk target io handling code
> path, because io_uring task work schedule uses llist, which may introduce io
> reorder.

There are for sure other places where a reordering can be introduced, so the code should be ready and expecting 
It. (For my case see bellow) Is preserving the order required for some reason for ublk?

> 
> However, that involves ublk kprobe/kfunc trace, which may not be stable,
> so I simply check the end-to-end IO order. Sometimes blk-mq IO queue/dispatch
> may re-order IO.
> 
> I guess the following change may avoid the re-order, but batch IO case may
> not be covered:
> 
> diff --git a/tools/testing/selftests/ublk/test_generic_01.sh b/tools/testing/selftests/ublk/test_generic_01.sh
> index 21a31cd5491a..5805da4c84c5 100755
> --- a/tools/testing/selftests/ublk/test_generic_01.sh
> +++ b/tools/testing/selftests/ublk/test_generic_01.sh
> @@ -29,14 +29,8 @@ if ! kill -0 "$btrace_pid" > /dev/null 2>&1; then
>        exit "$UBLK_SKIP_CODE"
> fi
> 
> -# run fio over this ublk disk
> -fio --name=write_seq \
> -    --filename=/dev/ublkb"${dev_id}" \
> -    --ioengine=libaio --iodepth=16 \
> -    --rw=write \
> -    --size=512M \
> -    --direct=1 \
> -    --bs=4k > /dev/null 2>&1
> +taskset -c 0 dd if=/dev/zero of=/dev/ublkb"${dev_id}" bs=1M count=256 oflag=direct > /dev/null 2>&1
> +
> 
> 
>> consider it an error? (Latest tree with batch io and batch io fixes on top of if that matters)
> 
> Never observe generic_01 failure in my test VM and hardware.
> 
> My kernel config is based on Fedora, maybe scheduler config option makes the difference.

Fedora 43 default config with some debugging options enabled, but no changes in schedulers.
Test VM storage is on a networked NAS over iSCSI - both boxes VM host and NAS have two NICs,
I get the errors when I load the network. So I believe the requests really complete out of 
order due to the network in my case. All tests that have the bpftrace check fail on occasion.


Regards,
Alexander Atanasov

Powered by blists - more mailing lists