Message-ID: <Z4YIAqonpBNuB9nJ@xsang-OptiPlex-9020>
Date: Tue, 14 Jan 2025 14:45:22 +0800
From: Oliver Sang <oliver.sang@...el.com>
To: Niklas Cassel <cassel@...nel.org>
CC: Christoph Hellwig <hch@....de>, <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
	<linux-kernel@...r.kernel.org>, Jens Axboe <axboe@...nel.dk>,
	<linux-block@...r.kernel.org>, <virtualization@...ts.linux.dev>,
	<linux-nvme@...ts.infradead.org>, Damien Le Moal <dlemoal@...nel.org>,
	<linux-btrfs@...r.kernel.org>, <linux-aio@...ck.org>, <oliver.sang@...el.com>
Subject: Re: [linus:master] [block]  e70c301fae: stress-ng.aiol.ops_per_sec
 49.6% regression

hi, Niklas,

On Wed, Jan 08, 2025 at 11:39:28AM +0100, Niklas Cassel wrote:
> On Tue, Jan 07, 2025 at 04:27:44PM +0800, Oliver Sang wrote:
> > hi, Niklas,
> > 
> > On Fri, Jan 03, 2025 at 10:09:14AM +0100, Niklas Cassel wrote:
> > > On Fri, Jan 03, 2025 at 07:49:25AM +0100, Christoph Hellwig wrote:
> > > > On Thu, Jan 02, 2025 at 10:49:41AM +0100, Niklas Cassel wrote:
> > > > > > > from below information, it seems an 'ahci' to me. but since I have limited
> > > > > > > knowledge about storage driver, maybe I'm wrong. if you want more information,
> > > > > > > please let us know. thanks a lot!
> > > > > > 
> > > > > > Yes, this looks like ahci.  Thanks a lot!
> > > > > 
> > > > > Did this ever get resolved?
> > > > > 
> > > > > I haven't seen a patch that seems to address this.
> > > > > 
> > > > > AHCI (ata_scsi_queuecmd()) only issues a single command, so if there is any
> > > > > reordering when issuing a batch of commands, my guess is that the problem
> > > > > also affects SCSI / the problem is in upper layers above AHCI, i.e. SCSI lib
> > > > > or block layer.
> > > > 
> > > > I started looking into this before the holidays.  blktrace shows perfectly
> > > > sequential writes without any reordering using ahci, directly on the
> > > > block device or using xfs and btrfs when using dd.  I also started
> > > > looking into what the test does and got as far as checking out the
> > > > stress-ng source tree and looking at stress-aiol.c.  AFAICS the default
> > > > submission does simple reads and writes using increasing offsets.
> > > > So if the test result isn't a fluke either the aio code does some
> > > > weird reordering or btrfs does.
> > > > 
> > > > Oliver, did the test also show any interesting results on non-btrfs
> > > > setups?
> > > > 
> > > 
> > > One thing that came to mind.
> > > Some distros (e.g. Fedora and openSUSE) ship with an udev rule that sets
> > > the I/O scheduler to BFQ for single-queue HDDs.
> > > 
> > > It could very well be the I/O scheduler that reorders.
> > > 
> > > Oliver, which I/O scheduler are you using?
> > > $ cat /sys/block/sdb/queue/scheduler 
> > > none mq-deadline kyber [bfq]
> > 
> > while our test running:
> > 
> > # cat /sys/block/sdb/queue/scheduler
> > none [mq-deadline] kyber bfq
> 
> The stddev numbers you showed is all over the place, so are we certain
> if this is a regression caused by commit e70c301faece ("block:
> don't reorder requests in blk_add_rq_to_plug") ?
> 
> Do you know if the stddev has such big variation for this test even before
> the commit?
> 
> 
> If it is not too much to ask... It might be interesting to know if we see
> a regression when comparing before/after e70c301faece with scheduler none
> instead of mq-deadline.

we also finished the test for scheduler none, run v6.12-rc4, before/after
e70c301faece 15 times for each. it seems the data is not stable enough under
scheduler none, only can say there is a regression trend if comparing
e70c301faece to its parent.

=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  gcc-12/performance/none-scheduler/1HDD/btrfs/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/aiol/stress-ng/60s

commit:
  v6.12-rc4
  a3396b9999 ("block: add a rq_list type")
  e70c301fae ("block: don't reorder requests in blk_add_rq_to_plug")

       v6.12-rc4 a3396b99990d8b4e5797e7b16fd e70c301faece15b618e54b613b1
---------------- --------------------------- ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
    114.62 ± 19%      -1.9%     112.49 ± 17%     -32.4%      77.47 ± 21%  stress-ng.aiol.ops_per_sec


raw data is as below:

v6.12-rc4/matrix.json:  "stress-ng.aiol.ops_per_sec": [
v6.12-rc4/matrix.json-    108.03,
v6.12-rc4/matrix.json-    108.4,
v6.12-rc4/matrix.json-    109.11,
v6.12-rc4/matrix.json-    109.58,
v6.12-rc4/matrix.json-    194.21,
v6.12-rc4/matrix.json-    111.53,
v6.12-rc4/matrix.json-    107.99,
v6.12-rc4/matrix.json-    115.29,
v6.12-rc4/matrix.json-    105.75,
v6.12-rc4/matrix.json-    113.62,
v6.12-rc4/matrix.json-    96.51,
v6.12-rc4/matrix.json-    110.53,
v6.12-rc4/matrix.json-    108.71,
v6.12-rc4/matrix.json-    98.06,
v6.12-rc4/matrix.json-    121.95
v6.12-rc4/matrix.json-  ],


a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json:  "stress-ng.aiol.ops_per_sec": [
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    116.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    106.51,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    119.23,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    108.91,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    111.79,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    111.81,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    114.94,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    99.49,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    106.13,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    124.99,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    174.15,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    92.65,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    113.05,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    75.97,
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-    111.05
a3396b99990d8b4e5797e7b16fdeb64c15ae97bb/matrix.json-  ],


e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json:  "stress-ng.aiol.ops_per_sec": [
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    85.2,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    72.6,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    73.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    69.03,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    66.9,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    90.24,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    66.88,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    71.53,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    56.86,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    63.49,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    97.99,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    69.28,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    58.52,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    114.23,
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-    105.79
e70c301faece15b618e54b613b1fd6ece3dd05b4/matrix.json-  ],


since not sure if it's valuable, I didn't list the full comparison table here.
if you want it, please let us know. thanks!

> 
> 
> Kind regards,
> Niklas
> 

Powered by blists - more mailing lists