Message-ID: <20090522122609.GC29447@sgi.com>
Date:	Fri, 22 May 2009 07:26:09 -0500
From:	Robin Holt <holt@....com>
To:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>
Cc:	LKML <linux-kernel@...r.kernel.org>, linux-mm <linux-mm@...ck.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Rik van Riel <riel@...hat.com>,
	Christoph Lameter <cl@...ux-foundation.org>,
	Robin Holt <holt@....com>,
	"Zhang, Yanmin" <yanmin.zhang@...el.com>,
	Wu Fengguang <fengguang.wu@...el.com>
Subject: Re: [PATCH v3] zone_reclaim is always 0 by default

OK.  While I did not object earlier, I am starting to feel a NACK
coming on.

How did you determine this is the source of your problems?  What leads
you to believe this is the correct fix instead of an easy change which
affects some random benchmark?

Let me clear, I believe you are seeing an impact from reclaim.  I do
not agree it is necessarily a negative impact for the majority of users.


On Thu, May 21, 2009 at 11:47:01AM +0900, KOSAKI Motohiro wrote:
> 
> Subject: [PATCH v3] zone_reclaim is always 0 by default
> 
> Current linux policy is, zone_reclaim_mode is enabled by default if the machine
> has large remote node distance. it's because we could assume that large distance
> mean large server until recently.
> 
> Unfortunately, recent modern x86 CPU (e.g. Core i7, Opeteron) have P2P transport
> memory controller. IOW it's seen as NUMA from software view.
> Some Core i7 machine has large remote node distance.
> 
> Yanmin reported zone_reclaim_mode=1 cause large apache regression.
> 
>     One Nehalem machine has 12GB memory,
>     but there is always 2GB free although applications accesses lots of files.
>     Eventually we located the root cause as zone_reclaim_mode=1.

Your root cause analysis is suspect.  You found a knob to turn which
suddenly improved performance for one specific un-tuned server workload.

> Actually, zone_reclaim_mode=1 mean "I dislike remote node allocation rather than
> disk access", it makes performance improvement to HPC workload.
> but it makes performance regression desktop, file server and web server.

zone_reclaim_mode merely means try to free any local unused page before
going off node.  I have never seen off-node allocations precluded as
long as the local node's pages are in use.  The effect on your one test
shows that unused page cache pages get properly discarded and reused by
the allocator.

> In general, workload depended configuration shouldn't put into default settings.
> Plus, desktop and file/web server eco-system is much larger than hpc's.

I believe you are putting a workload dependent configuration in as the
default.  You have not shown this improves anything other than a poorly
configured system running apache responds better on your tests.  I can
make a common sense argument that both =1 and =0 are better.  I think
the fact that it has been =1 for so long and not caused significant
issues should at least be factored in.  Making an exception for the
new hardware on the block makes sense as well.

> Thus, zone_reclaim == 0 is better by default.

How did you determine better by default?  I think we already established
that apache is a server workload and not a desktop workload.  Earlier
you were arguing that we need this turned off to improve the desktop
environment.  You have not established this improves desktop performance.
Actually, you have not established it improves apache performance or
server performance.  You have documented it improves memory utilization,
but that is not always the same as faster.

Sorry for being difficult about this, but you are tweaking a knob that
completely changes performance for my typical workload.  Reclaim has
been the source of great frustration for me over the years.

Hopefully this is not arrogance on my part, but if you went back to
something equivalent to my earlier patch which allowed the architecture
to decide the default, I would go back to not objecting despite the lack
of proof this is the right fix.  You never did specify what was wrong
with that patch.  It was simple to understand, accomplished your needs
as well as mine, allowed flexibility in implementing the default as the
#define could be expanded to include arch specific checks if sub-arches
find they need a different default than the rest of the arch.  Compared to
"Just remove the default", that seems preferable.

Thanks,
Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists