linux-ext4 - RE: ext4: performance regression introduced by the cgroup writeback support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d9d2300ea9fd4dd29a2e02c66e68a2ec@SIXPR30MB031.064d.mgd.msft.net>
Date:	Thu, 24 Sep 2015 00:12:30 +0000
From:	Dexuan Cui <decui@...rosoft.com>
To:	Chris Mason <clm@...com>
CC:	Theodore Ts'o <tytso@....edu>,
	Andreas Dilger <adilger.kernel@...ger.ca>,
	Tejun Heo <tj@...nel.org>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: ext4: performance regression introduced by the cgroup writeback
 support

> -----Original Message-----
> From: Chris Mason [mailto:clm@...com]
> Sent: Thursday, September 24, 2015 0:14
> To: Dexuan Cui <decui@...rosoft.com>
> Cc: Theodore Ts'o <tytso@....edu>; Andreas Dilger <adilger.kernel@...ger.ca>;
> Tejun Heo <tj@...nel.org>; linux-ext4@...r.kernel.org; linux-
> fsdevel@...r.kernel.org; linux-kernel@...r.kernel.org
> Subject: Re: ext4: performance regression introduced by the cgroup writeback
> support
> 
> On Wed, Sep 23, 2015 at 01:49:31PM +0000, Dexuan Cui wrote:
> > Hi all,
> > Since some point between July and Sep, I have been suffered from a strange
> "very slow write" issue and on Sep 9 I reported it to LKML (but got no reply):
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2flkml.org%
> 2flkml%2f2015%2f9%2f9%2f290&data=01%7c01%7cdecui%40064d.mgd.micros
> oft.com%7c8001aa10249f41a0363608d2c432042d%7c72f988bf86f141af91ab2
> d7cd011db47%7c1&sdata=oJBsP55jdg86TNt2X71s0gfPlwbMTzaJN9QIcsXsSmA%
> 3d
> >
> > The issue is: under high CPU and disk I/O pressure, *some* processes can
> suffer from a very slow write speed (e.g., <1MB/s or even only 20KB/s), while
> the normal write speed should be at least dozens of MB/s.
> >
> > I think I identified the commit which introduced the regression:
> > ext4: implement cgroup writeback support
> (https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fgit.kernel.
> org%2fcgit%2flinux%2fkernel%2fgit%2fnext%2flinux-
> next.git%2fcommit%2f%3fid%3d001e4a8775f6e8ad52a89e0072f09aee47d5d25
> 2&data=01%7c01%7cdecui%40064d.mgd.microsoft.com%7c8001aa10249f41a0
> 363608d2c432042d%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=QIcX
> R%2flZMqkK2afIxV%2fYxZDug26vj5yx%2bkoh6ugJB2A%3d)
> >
> > This commit is already in the mainline tree, so I can reproduce the issue there
> too:
> > With the latest mainline,  I can reproduce the issue; after I revert the patch, I
> can't reproduce the issue.
> >
> > When the issue happens:
> > 1. the read speed is pretty normal, e.g.. it's still >100MB/s.
> > 2. 'top' shows both the 'user' and 'sys' utilization is about 0%, but the IO-wait is
> always about 100%.
> > 3. 'iotop' shows the read speed is 0 (this is correct because there is indeed no
> read request)  and the write speed is pretty slow (the average is <1MB/s or even
> 20KB/s).
> > 4. when the issue happens, sometimes any new process suffers from the slow
> write issue, but sometimes it looks not all the new processes suffers from the
> issue.
> > 5. The " WARNING: CPU: 7 PID: 6782 at fs/inode.c:390 ihold+0x30/0x40() " in
> my Sep-9 mail may be another different issue.
> > 6. To reproduce the issue, I need to run my workload for enough long time
> (see the below).
> >
> > My workload is simple: I just repeatedly build the kernel source ("make clean;
> make -j16"). My kernel config is attached FYI.
> >
> > I can reproduce the issue on a physical machine: e.g., in my kernel building test
> with my .config, it took only ~5 minutes in the first 176 runs, but since the 177th
> run, it could take from 10 hours to 5 minutes - very unstable.
> >
> > It looks it's easier to reproduce the issue in a Hyper-V VM: usually I can
> reproduce the issue within the first 10 or 20 runs.
> >
> > Any idea?
> 
> Are you using cgroups?  That patch really shouldn't impact load unless
> there are actual IO controls in place.
> 
> -chris

I'm not using cgroups here.

Tejun just now found the root cause: "Separate wb domains
shouldn't have been enabled on traditional hierarchies " and supplied a fix.

Thanks,
-- Dexuan
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html