linux-kernel - Re: Memory overcommit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20091030151544.GR9640@random.random>
Date:	Fri, 30 Oct 2009 16:15:44 +0100
From:	Andrea Arcangeli <aarcange@...hat.com>
To:	Vedran Furač <vedran.furac@...il.com>
Cc:	David Rientjes <rientjes@...gle.com>,
	Hugh Dickins <hugh.dickins@...cali.co.uk>,
	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>,
	linux-mm@...ck.org, linux-kernel@...r.kernel.org,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	minchan.kim@...il.com, Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: Memory overcommit

On Fri, Oct 30, 2009 at 03:41:12PM +0100, Vedran Furač wrote:
> Oh... so this is because apps "reserve" (Committed_AS?) more then they
> currently need.

They don't actually reserve, they end up "reserving" if overcommit is
set to 2 (OVERCOMMIT_NEVER)... Apps aren't reserving, more likely they
simply avoid a flood of mmap when a single one is enough to map an
huge MAP_PRIVATE region like shared libs that you may only execute
partially (this is why total_vm is usually much bigger than real ram
mapped by pagetables represented in rss). But those shared libs are
99% pageable and they don't need to stay in swap or ram, so
overcommit-as greatly overstimates the actual needs even if shared lib
loading wouldn't be 64bit optimized (i.e. large and a single one).

> A the time of "malloc: Cannot allocate memory":
> 
> CommitLimit:     3364440 kB
> Committed_AS:    3240200 kB
> 
> So probably everything is ok (and free is misleading). Overcommit is
> unfortunately necessary if I want to be able to use all my memory.

Add more swap.

> Btw. http://www.redhat.com/advice/tips/meminfo.html says Committed_AS is
> a (gu)estimate. Hope it is a good (not to high) guesstimate. :)

It is a guess in the sense to guarantee no ENOMEM it has to take into
account the worst possible case, that is all shared lib MAP_PRIVATE
mappings are cowed, which is very far from reality. Other than that
the overcommitas should exactly match all mmapped possibly writeable
space that can only fit in ram+swap, so from that point of view it's
not a guessed number (modulo the smp read out of order). The only
guess is how much slab, cache and other stuff is freeable, which
doesn't provide true perfection to OVERCOMMIT_NEVER.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/