| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <c517e1ca-0e2b-4143-b0f6-07a3d0c3abcf@zazolabs.com>
Date: Sun, 25 Jan 2026 20:35:20 +0200
From: Alexander Atanasov <alex@...olabs.com>
To: Ming Lei <ming.lei@...hat.com>
Cc: Shuah Khan <shuah@...nel.org>, linux-block@...r.kernel.org,
linux-kselftest@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] selftests: ublk: io-reorder triggered in
test_generic_01.sh
On 25.01.26 17:28, Ming Lei wrote:
> On Fri, Jan 23, 2026 at 05:00:33PM +0200, Alexander Atanasov wrote:
>>
>>
[snip]
> Can you test the following patch and see if re-order still can happen?
>
>
> diff --git a/tools/testing/selftests/ublk/test_generic_01.sh b/tools/testing/selftests/ublk/test_generic_01.sh
> index 26cf3c7ceeb5..26d5e52ece29 100755
> --- a/tools/testing/selftests/ublk/test_generic_01.sh
> +++ b/tools/testing/selftests/ublk/test_generic_01.sh
> @@ -13,7 +13,7 @@ if ! _have_program fio; then
> exit "$UBLK_SKIP_CODE"
> fi
>
> -_prep_test "null" "sequential io order"
> +_prep_test "null" "ublk dispatch won't reorder IO"
>
> dev_id=$(_add_ublk_dev -t null)
> _check_add_dev $TID $?
> @@ -39,9 +39,13 @@ fio --name=write_seq \
> ERR_CODE=$?
> kill "$btrace_pid"
> wait
> -if grep -q "io_out_of_order" "$UBLK_TMP"; then
> - cat "$UBLK_TMP"
> +
> +# Check for out-of-order completions detected by bpftrace
> +if grep -q "^out_of_order:" "$UBLK_TMP"; then
> + echo "I/O reordering detected:"
> + grep "^out_of_order:" "$UBLK_TMP"
> ERR_CODE=255
> fi
> +
> _cleanup_test "null"
> _show_result $TID $ERR_CODE
> diff --git a/tools/testing/selftests/ublk/trace/seq_io.bt b/tools/testing/selftests/ublk/trace/seq_io.bt
> index b2f60a92b118..60ac40e66606 100644
> --- a/tools/testing/selftests/ublk/trace/seq_io.bt
> +++ b/tools/testing/selftests/ublk/trace/seq_io.bt
> @@ -2,23 +2,45 @@
> $1: dev_t
> $2: RWBS
> $3: strlen($2)
> +
> + Track request order between block_io_start and block_rq_complete.
> + For each request, record its start sequence number and verify
> + completions happen in the same order.
> */
> +
> BEGIN {
> - @last_rw[$1, str($2)] = (uint64)0;
> + @start_seq = (uint64)0;
> + @complete_seq = (uint64)0;
> + @out_of_order = (uint64)0;
> +}
> +
> +tracepoint:block:block_io_start
> +{
> + if ((int64)args.dev == $1 && !strncmp(args.rwbs, str($2), $3)) {
> + @start_order[args.sector] = @start_seq;
> + @start_seq = @start_seq + 1;
> + }
> }
> +
> tracepoint:block:block_rq_complete
> {
> - $dev = $1;
> if ((int64)args.dev == $1 && !strncmp(args.rwbs, str($2), $3)) {
> - $last = @last_rw[$dev, str($2)];
> - if ((uint64)args.sector != $last) {
> - printf("io_out_of_order: exp %llu actual %llu\n",
> - args.sector, $last);
> + $expected_order = @start_order[args.sector];
> + if ($expected_order != @complete_seq) {
> + printf("out_of_order: sector %llu started at seq %llu but completed at seq %llu\n",
> + args.sector, $expected_order, @complete_seq);
> + @out_of_order = @out_of_order + 1;
> }
> - @last_rw[$dev, str($2)] = (args.sector + args.nr_sector);
> + delete(@start_order[args.sector]);
> + @complete_seq = @complete_seq + 1;
> }
> }
>
> END {
> - clear(@last_rw);
> + printf("total_start: %llu total_complete: %llu out_of_order: %llu\n",
> + @start_seq, @complete_seq, @out_of_order);
> + clear(@start_order);
> + clear(@start_seq);
> + clear(@complete_seq);
> + clear(@out_of_order);
> }
First run after reboot always fails:
./test_generic_01.sh
ublk selftest: generic_01 starting at 2026-01-25T19:08:47+02:00
I/O reordering detected:
out_of_order: sector 112040 started at seq 88 but completed at seq 0
out_of_order: sector 112048 started at seq 89 but completed at seq 1
out_of_order: sector 112056 started at seq 90 but completed at seq 2
out_of_order: sector 112064 started at seq 91 but completed at seq 3
out_of_order: sector 112072 started at seq 92 but completed at seq 4
out_of_order: sector 112080 started at seq 93 but completed at seq 5
out_of_order: sector 112088 started at seq 94 but completed at seq 6
out_of_order: sector 112096 started at seq 95 but completed at seq 7
out_of_order: sector 112104 started at seq 96 but completed at seq 8
out_of_order: sector 112112 started at seq 97 but completed at seq 9
out_of_order: sector 112120 started at seq 98 but completed at seq 10
out_of_order: sector 112128 started at seq 99 but completed at seq 11
[snip]
out_of_order: sector 1048528 started at seq 117149 but completed at seq
117061
out_of_order: sector 1048536 started at seq 117150 but completed at seq
117062
out_of_order: sector 1048544 started at seq 117151 but completed at seq
117063
out_of_order: sector 1048552 started at seq 117152 but completed at seq
117064
out_of_order: sector 1048560 started at seq 117153 but completed at seq
117065
out_of_order: sector 1048568 started at seq 117154 but completed at seq
117066
ublk selftest: generic_01 done at 2026-01-25T19:08:54+02:00
generic_01 : [FAIL]
consecutive runs are okay
./test_generic_01.sh
ublk selftest: generic_01 starting at 2026-01-25T20:21:31+02:00
ublk selftest: generic_01 done at 2026-01-25T20:21:38+02:00
generic_01 : [PASS]
I went thru the reboot cycle several times - it repeats.
Tried with and without setting the affinity - and i observed no difference.
I will do some more testing but it was easier to reproduce before that
patch.
--
have fun,
alex
Powered by blists - more mailing lists