lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Wed, 23 Jun 2021 12:04:12 -0400
From:   Gabriel Krisman Bertazi <krisman@...labora.com>
To:     "Theodore Ts'o" <tytso@....edu>
Cc:     kernel@...labora.com, linux-ext4@...r.kernel.org
Subject: Re: Potential regression with iomap DIO for 4k writes

"Theodore Ts'o" <tytso@....edu> writes:

> On Tue, Jun 15, 2021 at 08:17:55PM -0400, Gabriel Krisman Bertazi wrote:

> Apologies for the delay in responding; somehow I missed your initial
> e-mail on the subject on June 2nd, although I haven't found it in the
> mailing list archives[1].  I don't know if it got caught in a spam
> trap, or was accidentally deleted from my inbox.
>
> [1] https://lore.kernel.org/linux-ext4/87lf7rkffv.fsf@collabora.com/
>
> I didn't do any bs=4k benchmarks before we landed the DIO iomap
> changes, and it's interesting that it largely goes away with a 16k
> block size[2]
>
> [2] https://people.collabora.com/~krisman/dio/week21/bench.png
>
> Looking at your flame graphs[3][4]
>
> [3] https://people.collabora.com/~krisman/dio/week23/clean_flames/5.4.0-dio_original-dio-ext4-write-4k.svg
> [4] https://people.collabora.com/~krisman/dio/week23/clean_flames/5.5.0-dio_old-iomap-ext4-write-4k.svg
>
> ... nothing immediately jumps out at me.
>
> Have you compared the output of /proc/lock_stat for the two kernels?
> And is there anything obvious in the blktrace of the two kernels?

Hi Ted,

I updated and rerun my test script [1] to collect lock_stat and blktrace.
I particularly don't see anything standing out on lock-stat, but I'm
sharing the raw data [2], in case you have a better insight.

blktrace, on the other hand, shows something interesting.  Looks like
the original DIO is always keeping a lower device queue depth than iomap
[3], although I'd expect the opposite if there is FS contention,
correct?  Iff ext4 is taking longer to submit bios, it would be expected
for the device queue depth to be more empty instead of more full, right?

[1] https://people.collabora.com/~krisman/dio/week24/bench-regression.sh
[2] https://people.collabora.com/~krisman/dio/week24/
[3] (LARGE SVG!) https://people.collabora.com/~krisman/dio/week24/iowatcher-charts.svg

-- 
Gabriel Krisman Bertazi

Powered by blists - more mailing lists