linux-ext4 - Re: ext4 hang and per-memcg dirty throttling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180912121130.GF7782@quack2.suse.cz>
Date:   Wed, 12 Sep 2018 14:11:30 +0200
From:   Jan Kara <jack@...e.cz>
To:     Liu Bo <bo.liu@...ux.alibaba.com>
Cc:     linux-ext4@...r.kernel.org, fengguang.wu@...el.com, tj@...nel.org,
        jack@...e.cz, cgroups@...r.kernel.org, gthelen@...gle.com,
        linux-mm@...ck.org, yang.shi@...ux.alibaba.com
Subject: Re: ext4 hang and per-memcg dirty throttling

Hi!

On Tue 11-09-18 17:10:55, Liu Bo wrote:
> With ext4's data=ordered mode and the underlying blk throttle setting, we
> can easily run to hang,
> 
> 1.
> mount /dev/sdc /mnt -odata=ordered
> 2.
> mkdir /sys/fs/cgroup/unified/cg
> 3.
> echo "+io" > /sys/fs/cgroup/unified/cgroup.subtree_control
> 4.
> echo "`cat /sys/block/sdc/dev` wbps=$((1 << 20))" > /sys/fs/cgroup/unified/cg/io.max
> 5.
> echo $$ >  /sys/fs/cgroup/unified/cg/cgroup.procs
> 6.
> // background dirtier
> xfs_io -f -c "pwrite 0 1G" $M/dummy &
> 7.
> echo $$ > /sys/fs/cgroup/unified/cgroup.procs
> 8.
> // issue synchronous IO
> for i in `seq 1 100`;
> do
>     xfs_io -f -s -c "pwrite 0 4k" $M/foo > /dev/null
> done
> 
> 
> And the hang is like
> 
>       [jbd2-sdc]
> jbd2_journal_commit_transaction                              
>   journal_submit_data_buffers
>     # file 'dummy' has been written by writeback kthread
>   journal_finish_inode_data_buffers
>     # wait on page's writeback

Yes, I guess you're speaking about the one Chris Mason mentioned [1].
Essentially it's a priority inversion where jbd2 thread gets blocked behind
writeback done on behalf of a heavily restricted process. It actually is
not related to dirty throttling or anything like that. And the solution for
this priority inversion is to use unwritten extents for writeback
unconditionally as I wrote in that thread. The core of this is implemented
and hidden behind dioread_nolock mount option but it needs some serious
polishing work and testing...

[1] https://marc.info/?l=linux-fsdevel&m=151688776319077

								Honza
-- 
Jan Kara <jack@...e.com>
SUSE Labs, CR