linux-ext4 - [PATCH RFC 0/2] ext4: Do endio process under irq context for DIO overwrites

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date: Thu, 29 Feb 2024 19:38:47 +0800
From: Zhihao Cheng <chengzhihao1@...wei.com>
To: <brauner@...nel.org>, <david@...morbit.com>, <djwong@...nel.org>,
	<jack@...e.cz>, <tytso@....edu>
CC: <linux-xfs@...r.kernel.org>, <linux-fsdevel@...r.kernel.org>,
	<linux-kernel@...r.kernel.org>, <linux-ext4@...r.kernel.org>,
	<yi.zhang@...wei.com>
Subject: [PATCH RFC 0/2] ext4: Do endio process under irq context for DIO overwrites

Recently we found an ext4 performance regression problem between 4.18
and 5.10 by following test command on a x86 physical machine with nvme:
  fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k
      -size=2G -numjobs=1 -time_based -runtime=60 -group_reporting
      -filename=/test/test -name=Rand_write_Testing --cpus_allowed=1
4.18: 288k IOPS    5.10: 234k IOPS

After anlayzing the changes between above two versions, we found that
the endio context changed since commit 378f32bab3714("ext4: introduce
direct I/O write using iomap infrastructure"), in aio DIO overwriting
case. And the problem still exist in latest mainline.
4.18: endio is processed under irq context
 dio_bio_end_aio
  defer_completion = dio->defer_completion || ... // defer_completion is
                                                  // false for overwrite
  if (defer_completion) {
    queue_work
  } else {
    dio_complete                          // endio in irq context
  }
mainline: endio is processed in kworker
 iomap_dio_bio_end_io
  if (dio->flags & IOMAP_DIO_INLINE_COMP) // false, only for read
  if (dio->flags & IOMAP_DIO_CALLER_COMP) // false, only for io_uring
  queue_work                              // endio in kworker context

Assume that nvme irq registers on cpu 1, and fio runs on cpu 1, it could
be possible(Actually we did reproduce it easily) that fio, nvme irq and
kworker all run on cpu 1. Firstly, compared with 4.18, there are extra
operations(eg. queue work and wake up kworker) while processing endio,
which is slower than processing endio under nvme irq context. Secondly,
fio and kworker will race to get cpu resource, which will slow down fio.

There is no need to do
ext4_convert_unwritten_extents/ext4_handle_inode_extension under
ext4_dio_write_end_io in overwriting case, so we can put ext4 dio endio
under irq context.

It is worth noting that iomap_dio_complete shouldn't be executed under
irq context if datasync is required(IOMAP_DIO_NEED_SYNC), so overwriting
endio is still done in kworker for this situation.

Zhihao Cheng (2):
  iomap: Add a IOMAP_DIO_MAY_INLINE_COMP flag
  ext4: Optimize endio process for DIO overwrites

 fs/ext4/file.c        |  8 ++++++--
 fs/iomap/direct-io.c  | 10 ++++++++--
 include/linux/iomap.h |  6 ++++++
 3 files changed, 20 insertions(+), 4 deletions(-)

-- 
2.39.2