linux-kernel - Re: 2.6.22 -mm merge plans -- lumpy reclaim

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <46373A71.4030200@shadowen.org>
Date:	Tue, 01 May 2007 14:02:41 +0100
From:	Andy Whitcroft <apw@...dowen.org>
To:	Mel Gorman <mel@...net.ie>
CC:	Andrew Morton <akpm@...ux-foundation.org>,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org, clameter@....com,
	y-goto@...fujitsu.com, Peter Zijlstra <a.p.zijlstra@...llo.nl>
Subject: Re: 2.6.22 -mm merge plans -- lumpy reclaim

Mel Gorman wrote:

<snip>

>>  lumpy-reclaim-v4.patch
> 
> And I guess this patch also moves here
> 
> lumpy-move-to-using-pfn_valid_within.patch
> 
>> This is in a similar situation to the moveable-zone work.  Sounds great on
>> paper, but it needs considerable third-party testing and review.  It is a
>> major change to core MM and, we hope, a significant advance.  On paper.
> 
> Andy will probably comment more here. Like the fragmentation stuff, we have
> beaten this heavily in tests.

With this stack the basic functionality for Lumpy reclaim is complete.
Better integration with kswapd is desirable, but IMO that should be a
separate change.

In testing it has produced significant improvements the likelyhood of
reclaiming a page (reclaim effectiveness) at very high orders (where the
likelyhood of success is least), and effectiveness at lower orders
should be better again.  In general -mm testing lumpy is triggered for
any stalled allocation above order-0; it is common to see stack
allocations triggering lumpy under higher load.  kswapd also now
utilises lumpy when required.

As Mel has indicated a lot of automated testing has been done on these
patches.  As reclaim is only entered when low on memory, our testing
focuses on triggering pushing the system to a heavily fragmented state
where reclaim is used heavily.  This testing has not shown any
regressions and shows improved effectiveness particularly under load.

Effectiveness for regular reclaim is based on random distributions, as
such it is only likely to successfully reclaim pages at lower orders.
Lumpy reclaim improves on this by actively targeting reclaim on areas at
the orders required and so succeeds at significantly higher order.  Very
high order allocations require better layout, from the mobility patches.

I have some primitive stats patches which we have used performance
testing.  Perhaps those could be brought up to date to provide better
visibility into lumpy's operation.  Again this would be a separate patch.

> I'm not sure of it's review situation.

As lumpy reclaim and grouping-by-mobility are complementary patch sets
(in that they both assist at the highest order) we work pretty closely
and I generally pass all my patches past Mel before general release.
Early versions were based on patches from Peter Zijlstra who also
reviewed earlier versions if memory serves.  The changes since then have
been reviewed by Mel and Andrew Morton only to my knowledge.

Perhaps Peter would have some time to take a look over the latest stack
as it appears in -mm when that releases; ping me for a patch kit if you
want it before then :).

<snip>

-apw
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/