linux-kernel - Re: kswapd craziness in 3.7

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 11 Dec 2012 01:19:31 +0100
From:	Zlatko Calusic <zlatko.calusic@...on.hr>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
CC:	Andrew Morton <akpm@...ux-foundation.org>,
	Mel Gorman <mgorman@...e.de>,
	Johannes Weiner <hannes@...xchg.org>,
	Rik van Riel <riel@...hat.com>,
	linux-mm <linux-mm@...ck.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Hugh Dickins <hughd@...gle.com>
Subject: Re: kswapd craziness in 3.7

> On 10.12.2012 20:13, Linus Torvalds wrote:
>>
>> It's worth giving this as much testing as is at all possible, but at
>> the same time I really don't think I can delay 3.7 any more without
>> messing up the holiday season too much. So unless something obvious
>> pops up, I will do the release tonight. So testing will be minimal -
>> but it's not like we haven't gone back-and-forth on this several times
>> already, and we revert to *mostly* the same old state as 3.6 anyway,
>> so it should be fairly safe.
>>

So, here's what I found. In short: close, but no cigar!

Kswapd is certainly no more CPU pig, and memory seems to be utilized 
properly (the kernel still likes to keep 400MB free, somebody else can 
confirm if that's to be expected on a 4GB THP-enabled machine). So it 
looks very decent, and much better than anything I run in last 10 days, 
barring !THP kernel.

What remains a mystery is that kswapd occassionaly still likes to get 
stuck in a D state, only now it recovers faster than before (sometimes 
in a matter of seconds, but sometimes it takes a few minutes). Now, I 
admit it's a small, maybe even cosmetic issue. But, it could also be a 
warning sign of a bigger problem that will reveal itself on a more 
loaded machine.

I will now make one last attempt, I've just reverted 2 Johannes' commits 
that were also applied in attempt to fix breakage that removing 
gfp_no_kswapd introduced, namely ed23ec4 & c702418. For various reasons 
the results of this test will be available tommorow, so it's your call 
Linus.

To better document this whole mess from my point of view, I've attached 
two graphs. First one nicely shows kswapd frenzy a week ago (blue 
mountains on a CPU graph). On Thu 06 & Mon 10 (until few hours ago) I 
run !THP kernels, better memory utilization is, I think, obvious (look 
at the bottom graph, lots of green is evil). CPU spikes at the end of 
every day are daily backup runs, which are CPU, NOT I/O bound. Notice 
L.A. close to 1 on !THP kernels (as it should be), and almost 2 (Fri & 
Sat 08) when the backup fought with kswapd (and also big CPU iowait 
times in that case). Finally, todays run is somewhere in between, that's 
why it deserves "close, but no cigar" qualification. ;)

The last graph shows CPU usage in more detail, yesterdays run was on a 
!THP kernel, todays THP run is the one with red spikes. That was kswapd 
in D state, in congestion_wait().

-- 
Zlatko

Download attachment "screenshot1.png" of type "image/png" (52963 bytes)

Download attachment "screenshot2.png" of type "image/png" (15122 bytes)