lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Wed, 23 Sep 2015 12:13:59 -0400
From:	Chris Mason <>
To:	Dexuan Cui <>
CC:	"Theodore Ts'o" <>,
	Andreas Dilger <>,
	Tejun Heo <>,
	"" <>,
	"" <>,
	"" <>
Subject: Re: ext4: performance regression introduced by the cgroup writeback

On Wed, Sep 23, 2015 at 01:49:31PM +0000, Dexuan Cui wrote:
> Hi all,
> Since some point between July and Sep, I have been suffered from a strange "very slow write" issue and on Sep 9 I reported it to LKML (but got no reply):
> The issue is: under high CPU and disk I/O pressure, *some* processes can suffer from a very slow write speed (e.g., <1MB/s or even only 20KB/s), while the normal write speed should be at least dozens of MB/s.
> I think I identified the commit which introduced the regression:
> ext4: implement cgroup writeback support (
> This commit is already in the mainline tree, so I can reproduce the issue there too:
> With the latest mainline,  I can reproduce the issue; after I revert the patch, I can't reproduce the issue.
> When the issue happens:
> 1. the read speed is pretty normal, e.g.. it's still >100MB/s.
> 2. 'top' shows both the 'user' and 'sys' utilization is about 0%, but the IO-wait is always about 100%.
> 3. 'iotop' shows the read speed is 0 (this is correct because there is indeed no read request)  and the write speed is pretty slow (the average is <1MB/s or even 20KB/s).
> 4. when the issue happens, sometimes any new process suffers from the slow write issue, but sometimes it looks not all the new processes suffers from the issue.
> 5. The " WARNING: CPU: 7 PID: 6782 at fs/inode.c:390 ihold+0x30/0x40() " in my Sep-9 mail may be another different issue.
> 6. To reproduce the issue, I need to run my workload for enough long time (see the below).
> My workload is simple: I just repeatedly build the kernel source ("make clean; make -j16"). My kernel config is attached FYI.
> I can reproduce the issue on a physical machine: e.g., in my kernel building test with my .config, it took only ~5 minutes in the first 176 runs, but since the 177th run, it could take from 10 hours to 5 minutes - very unstable.
> It looks it's easier to reproduce the issue in a Hyper-V VM: usually I can reproduce the issue within the first 10 or 20 runs.
> Any idea?

Are you using cgroups?  That patch really shouldn't impact load unless
there are actual IO controls in place.


To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to
More majordomo info at

Powered by blists - more mailing lists