linux-kernel - Re: [PATCH RFC v2 0/5] blk-mq-sched: support request batch dispatching for sq elevator

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <16fd5432-36b0-0a92-0caa-7374ce1464a5@huaweicloud.com>
Date: Mon, 16 Jun 2025 15:22:10 +0800
From: Yu Kuai <yukuai1@...weicloud.com>
To: Damien Le Moal <dlemoal@...nel.org>, Yu Kuai <yukuai1@...weicloud.com>,
 ming.lei@...hat.com, tj@...nel.org, josef@...icpanda.com, axboe@...nel.dk
Cc: linux-block@...r.kernel.org, cgroups@...r.kernel.org,
 linux-kernel@...r.kernel.org, yi.zhang@...wei.com, yangerkun@...wei.com,
 johnny.chenyi@...wei.com, "yukuai (C)" <yukuai3@...wei.com>
Subject: Re: [PATCH RFC v2 0/5] blk-mq-sched: support request batch
 dispatching for sq elevator

Hi,

在 2025/06/16 12:03, Damien Le Moal 写道:
> On 6/14/25 18:25, Yu Kuai wrote:
>> From: Yu Kuai <yukuai3@...wei.com>
>>
>> Before this patch, each dispatch context will hold a global lock to
>> dispatch one request at a time, which introduce intense lock competition:
>>
>> lock
>> ops.dispatch_request
>> unlock
>>
>> Hence support dispatch a batch of requests while holding the lock to
>> reduce lock contention.
>>
>> nullblk setup:
>> modprobe null_blk nr_devices=0 &&
>>      udevadm settle &&
>>      cd /sys/kernel/config/nullb &&
>>      mkdir nullb0 &&
>>      cd nullb0 &&
>>      echo 0 > completion_nsec &&
>>      echo 512 > blocksize &&
>>      echo 0 > home_node &&
>>      echo 0 > irqmode &&
>>      echo 128 > submit_queues &&
>>      echo 1024 > hw_queue_depth &&
>>      echo 1024 > size &&
>>      echo 0 > memory_backed &&
>>      echo 2 > queue_mode &&
>>      echo 1 > power ||
>>      exit $?
>>
>> Test script:
>> fio -filename=/dev/$disk -name=test -rw=randwrite -bs=4k -iodepth=32 \
>>    -numjobs=16 --iodepth_batch_submit=8 --iodepth_batch_complete=8 \
>>    -direct=1 -ioengine=io_uring -group_reporting -time_based -runtime=30
>>
>> Test result(elevator is deadline): iops
>> |                 | null_blk | scsi hdd |
>> | --------------- | -------- | -------- |
>> | before this set | 263k     | 24       |
>> | after this set  | 475k     | 272      |
> 
> For the HDD, these numbers are very low, and I do not understand how you can get
> any improvement from reducing lock contention, since contention should not be an
> issue with this kind of performance. What HW did you use for testing ? Was this
> a VM ?
> 

Thanks for reviewing this RFC set! I'm curious why there are improvement
as well, I didn't have the answer when I sent this set.

I'm testing on 256-core Kunpeng-920 server, with MG04ACA600E, 5TB HDD,
attched to hisi_sas_v3, and the disk have beed used for testing for more
than 5 years, perhaps this is why randwrite numbers are so low.

> I tested this null_blk setup and your fio command on a bare-metal 16-cores Xeon
> machine. For the scsi disk, I used a 26TB SATA HDD connected to an AHCI port).
> With this setup, results are like this:
> 
> |                 | null_blk | hdd (write) | hdd (read) |
> | --------------- | -------- | ----------- | ---------- |
> | before this set | 613k     | 1088        | 211        |
> | after this set  | 940k     | 1093        | 212        |
> 
> So not surprisingly, there is no improvement for the SATA HDD because of the low
> max IOPS these devices can achieve: lock contention is not really an issue when
> you are dealing with a slow device. And a SAS HDD will be the same. Gains may
> likely be more significant with a fast SAS/FC RAID array but I do not have
> access to that.
> 
> But the improvement for a fast device like null_blk is indeed excellent (+53%).
> 
> With LOCKDEP & KASAN disabled, the results are like this:
> 
> |                 | null_blk | hdd (write) | hdd (read) |
> | --------------- | -------- | ----------- | ---------- |
> | before this set | 625k     | 1092        | 213        |
> | after this set  | 984k     | 1095        | 215        |
> 
> No real changes for the HDD, as expected, and the improvement for null_blk is
> still good.

I agree that lock contention here will not affect HDD performance.
What I suspect the difference in my environment is that the order of rqs
might be changed from elevator dispatching them and the disk handling
them.

For example, the order can be easily revised if more than one context
dispatch one request at a time:

t1:

lock
rq1 = dd_dispatch_request
unlock
			t2:
			lock
			rq2 = dd_dispatch_request
			unlock

lock
rq3 = dd_dispatch_request
unlock

			lock
			rq4 = dd_dispatch_request
			unlock

//rq1,rq3 issue to disk
			// rq2, rq4 issue to disk

In this case, the elevator dispatch order is rq 1-2-3-4, however,
such order in disk is rq 1-3-2-4.

And with batch requests dispatch, will this less likely to happen?
> 
> So maybe drop the RFC tag on these patches and repost after cleaning things up ?

Sure, thanks again for reviewing this RFC set.
Kuai

> 
>