linux-ext4 - Re: ext4 hang and per-memcg dirty throttling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Wed, 12 Sep 2018 12:22:35 -0700
From:   Liu Bo <bo.liu@...ux.alibaba.com>
To:     "Theodore Y. Ts'o" <tytso@....edu>
Cc:     Jan Kara <jack@...e.cz>, linux-ext4@...r.kernel.org,
        fengguang.wu@...el.com, tj@...nel.org, cgroups@...r.kernel.org,
        gthelen@...gle.com, linux-mm@...ck.org, yang.shi@...ux.alibaba.com
Subject: Re: ext4 hang and per-memcg dirty throttling

On Wed, Sep 12, 2018 at 11:07:17AM -0400, Theodore Y. Ts'o wrote:
> On Wed, Sep 12, 2018 at 02:11:30PM +0200, Jan Kara wrote:
> > 
> > Yes, I guess you're speaking about the one Chris Mason mentioned [1].
> > Essentially it's a priority inversion where jbd2 thread gets blocked behind
> > writeback done on behalf of a heavily restricted process. It actually is
> > not related to dirty throttling or anything like that. And the solution for
> > this priority inversion is to use unwritten extents for writeback
> > unconditionally as I wrote in that thread. The core of this is implemented
> > and hidden behind dioread_nolock mount option but it needs some serious
> > polishing work and testing...
> > 
> > [1] https://marc.info/?l=linux-fsdevel&m=151688776319077
> 
> I've actually be considering making dioread_nolock the default when
> page_size == block_size.
> 
> Arguments in favor:
> 
> 1)  Improves AIO latency in some circumstances
> 2)  Improves parallel DIO read performance
> 3)  Should address the block-cg throttling priority inversion problem
> 
> Arguments against:
> 
> 1)  Hasn't seen much usage outside of Google (where it makes a big
>     difference for fast flash workloads; see (1) and (2) above)
> 2)  Dioread_nolock only works when page_size == block_size; so this
>     implies we would be using a different codepath depending on
>     the block size.
> 3)  generic/500 (dm-thin ENOSPC hitter with concurrent discards)
>     fails with dioread_nolock, but not in the 4k workload
> 
> Liu, can you try out mount -o dioread_nolock and see if this address
> your problem, if so, maybe this is the development cycle where we
> finally change the default.
> 

I've confirmed that "mount -o dioread_nolock" fixed the hang, I can do
further testing (maybe in production environment) if that's needed.

Many thanks to both you and Jan.

thanks,
-liubo