linux-ext4 - [RFC] performance regression with "ext4: Allow parallel DIO reads"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 19 Jul 2019 17:22:24 +0800
From:   Joseph Qi <joseph.qi@...ux.alibaba.com>
To:     Theodore Ts'o <tytso@....edu>, Jan Kara <jack@...e.cz>
Cc:     linux-ext4@...r.kernel.org,
        Xiaoguang Wang <xiaoguang.wang@...ux.alibaba.com>
Subject: [RFC] performance regression with "ext4: Allow parallel DIO reads"

Hi Ted & Jan,
I've observed an significant performance regression with the following
commit in my Intel P3600 NVMe SSD.
16c54688592c ext4: Allow parallel DIO reads

>From my initial investigation, it may be because of the
inode_lock_shared (down_read) consumes more than inode_lock (down_write)
in mixed random read write workload.

Here is my test result.

ioengine=psync
direct=1
rw=randrw
iodepth=1
numjobs=8
size=20G
runtime=600

w/ parallel dio reads : kernel 5.2.0
w/o parallel dio reads: kernel 5.2.0, then revert the following commits:
  1d39834fba99 ext4: remove EXT4_STATE_DIOREAD_LOCK flag (related)
  e5465795cac4 ext4: fix off-by-one error when writing back pages before dio read (related)
  16c54688592c ext4: Allow parallel DIO reads

bs=4k:
-------------------------------------------------------------------------------------------
w/ parallel dio reads | READ 30898KB/s, 7724, 555.00us   | WRITE 30875KB/s, 7718, 479.70us
-------------------------------------------------------------------------------------------
w/o parallel dio reads| READ 117915KB/s, 29478, 248.18us | WRITE 117854KB/s，29463, 21.91us
-------------------------------------------------------------------------------------------

bs=16k:
-------------------------------------------------------------------------------------------
w/ parallel dio reads | READ 58961KB/s, 3685, 835.28us   | WRITE 58877KB/s, 3679, 1335.98us
-------------------------------------------------------------------------------------------
w/o parallel dio reads| READ 218409KB/s, 13650, 554.46us | WRITE 218257KB/s，13641, 29.22us
-------------------------------------------------------------------------------------------

bs=64k:
-------------------------------------------------------------------------------------------
w/ parallel dio reads | READ 119396KB/s, 1865, 1759.38us | WRITE 119159KB/s, 1861, 2532.26us
-------------------------------------------------------------------------------------------
w/o parallel dio reads| READ 422815KB/s, 6606, 1146.05us | WRITE 421619KB/s, 6587, 60.72us
-------------------------------------------------------------------------------------------

bs=512k:
-------------------------------------------------------------------------------------------
w/ parallel dio reads | READ 392973KB/s, 767, 5046.35us  | WRITE 393165KB/s, 767, 5359.86us
-------------------------------------------------------------------------------------------
w/o parallel dio reads| READ 590266KB/s, 1152, 4312.01us | WRITE 590554KB/s, 1153, 2606.82us
-------------------------------------------------------------------------------------------

bs=1M:
-------------------------------------------------------------------------------------------
w/ parallel dio reads | READ 487779KB/s, 476, 8058.55us  | WRITE 485592KB/s, 474, 8630.51us
-------------------------------------------------------------------------------------------
w/o parallel dio reads| READ 593927KB/s, 580, 7623.63us  | WRITE 591265KB/s, 577, 6163.42us
-------------------------------------------------------------------------------------------

Thanks,
Joseph