linux-kernel - Re: [REGRESSION] 2.6.19/2.6.20-rc3 buffered write slowdown

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20070111003158.GT33919298@melbourne.sgi.com>
Date:	Thu, 11 Jan 2007 11:31:58 +1100
From:	David Chinner <dgc@....com>
To:	Nick Piggin <nickpiggin@...oo.com.au>
Cc:	David Chinner <dgc@....com>, Christoph Lameter <clameter@....com>,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: Re: [REGRESSION] 2.6.19/2.6.20-rc3 buffered write slowdown

On Thu, Jan 11, 2007 at 10:13:55AM +1100, Nick Piggin wrote:
> David Chinner wrote:
> >On Wed, Jan 10, 2007 at 03:04:15PM -0800, Christoph Lameter wrote:
> >
> >>On Thu, 11 Jan 2007, David Chinner wrote:
> >>
> >>
> >>>The performance and smoothness is fully restored on 2.6.20-rc3
> >>>by setting dirty_ratio down to 10 (from the default 40), so
> >>>something in the VM is not working as well as it used to....
> >>
> >>dirty_background_ratio is left as is at 10?
> >
> >
> >Yes.
> >
> >
> >>So you gain performance by switching off background writes via pdflush?
> >
> >
> >Well, pdflush appears to be doing very little on both 2.6.18 and
> >2.6.20-rc3. In both cases kswapd is consuming 10-20% of a CPU and
> >all of the pdflush threads combined (I've seen up to 7 active at
> >once) use maybe 1-2% of cpu time. This occurs regardless of the
> >dirty_ratio setting.
> 
> Hi David,
> 
> Could you get /proc/vmstat deltas for each kernel, to start with?

Sure, but that doesn't really show the how erratic the per-filesystem
throughput is because the test I'm running is PCI-X bus limited in
it's throughput at about 750MB/s. Each dm device is capable of about
340MB/s write, so when one slows down, the others will typically
speed up.

So, what I've attached is three files which have both
'vmstat 5' output and 'iostat 5 |grep dm-' output in them.

	- 2.6.18.out - 2.6.18 behaviour near start of writes.
	  Behaviour does not change over the couse of the test,
	  just gets a bit slower as the test moves from the outer
	  edge of the disk to the inner. erractic behaviour is
	  highlighted.

	- 2.6.20-rc3.out - 2.6.20-rc3 behaviour near start of writes.
	  Somewhat more erratic than 2.6.18, but about 100-150GB into
	  the write test, things change with dirty_ratio=40. erractic
	  behaviour is highlighted.

	- 2.6.20-rc3-worse.out - 2.6.20-rc3 behavour when things go
	  bad. We're not keeping the disks or the PCI-X bus fully
	  utilised (each dm device can do about 300MB/s at this offset)
	  and aggregate throughput has dropped to 500-600MB/s.

With 2.6.20-rc3 and dirty_ratio = 10, the performance drop-off part way
into the test does not occur and the output is almost identical to
2.6.18.out.

> I'm guessing CPU time isn't a problem, but if it is then I guess
> profiles as well.

Plenty of idle cpu so I don't think it's a problem.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

View attachment "2.6.18.out" of type "text/plain" (15040 bytes)

View attachment "2.6.20-rc3.out" of type "text/plain" (21242 bytes)

View attachment "2.6.20-rc3-worse.out" of type "text/plain" (9845 bytes)