linux-kernel - Re: Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1284703264.3408.1.camel@sli10-conroe.sh.intel.com>
Date:	Fri, 17 Sep 2010 14:01:04 +0800
From:	Shaohua Li <shaohua.li@...el.com>
To:	"robm@...tmail.fm" <robm@...tmail.fm>
Cc:	Christoph Lameter <cl@...ux.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Bron Gondwana <brong@...tmail.fm>,
	linux-mm <linux-mm@...ck.org>, Mel Gorman <mel@....ul.ie>
Subject: Re: Default zone_reclaim_mode = 1 on NUMA kernel is bad for
 file/email/web servers

On Fri, 2010-09-17 at 08:50 +0800, Robert Mueller wrote:
> > > > Having very little knowledge of what this actually does, I'd just
> > > > like to point out that from a users point of view, it's really
> > > > annoying for your machine to be crippled by a default kernel
> > > > setting that's pretty obscure.
> >
> > Thats an issue of the NUMA BIOS information. Kernel defaults to zone
> > reclaim if the cost of accessing remote memory vs local memory crosses
> > a certain threshhold which usually impacts performance.
> 
> We use what I thought was a fairly standard server type motherboard and
> CPU combination, and I was surprised that things were so badly broken
> for a standard usage scenario with a vanilla kernel with a default
> configuration.
> 
> I'd point out that the cost of a remote memory access is many, many
> orders of magnitude less than having to go back to disk! The problem is
> that with zone_reclaim_mode = 1 it seems lots of memory was being wasted
> that could be used as disk cache.
> 
> > > Yes, sadly intel motherboard turn on zone_reclaim_mode by
> > > default. and current zone_reclaim_mode doesn't fit file/web
> > > server usecase ;-)
> >
> > Or one could also say that the web servers are not designed to
> > properly distribute the load on a complex NUMA based memory
> > architecture of todays Intel machines.
> 
> I don't think this is any fault of how the software works. It's a *very*
> standard "pre-fork child processes, allocate incoming connections to a
> child process, open and mmap one or more files to read data from them".
> That's not exactly a weird programming model, and it's bad that the
> kernel is handling that case very badly with everything default.
maybe you incoming connection always happen on one CPU and you do the
page allocation in that cpu, so some nodes use out of memory but others
have a lot free. Try bind the child process to different nodes might
help.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/