linux-kernel - Re: computing drop-able caches

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Fri, 12 Feb 2016 17:15:09 -0500
From:	Johannes Weiner <hannes@...xchg.org>
To:	Dave Hansen <dave.hansen@...el.com>
Cc:	"Khalid Mughal (khalidm)" <khalidm@...co.com>,
	Rik van Riel <riel@...hat.com>,
	"Daniel Walker (danielwa)" <danielwa@...co.com>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	Michal Hocko <mhocko@...e.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	"xe-kernel@...ernal.cisco.com" <xe-kernel@...ernal.cisco.com>
Subject: Re: computing drop-able caches

On Fri, Feb 12, 2016 at 01:46:54PM -0800, Dave Hansen wrote:
> On 02/12/2016 10:01 AM, Khalid Mughal (khalidm) wrote:
> > If you look at the attached pdf, you will notice that OOM messages start
> > to appear when memAvailable is showing 253MB (259228 KB) Free, memFree is
> > 13.5MB (14008 KB) Free, and dropcache based calculation ³Available memory²
> > is showing 21MB (21720 KB) Free.
> > 
> > So, it appears that memAvailable is not as accurate, especially if data is
> > used to warn user about system running low on memory.
> 
> Yep, that's true.
> 
> But, MemAvailable is calculated from some very cheap counters.  The
> "dropcache-based-calculation" requires iterating over every 4k page
> cache page in the system.

It's also completely off when the dominating cache consumer uses
mmap() instead of buffered IO.

> We track dirty and writebackw with counters, so we should theoretically
> be able to pull those out of MemAvailable fairly cheaply.

Yeah, that sounds like a good idea.

But the fact remains that this will never be fully accurate, and there
will always be scenarios where the system will thrash and trigger OOM
before this counter depletes, simply because maintaining uptodate heat
information of the page cache would be crazy expensive.

IMO, the best way to use the MemAvailable counter is to calculate a
utilization percentage against MemTotal, and then maintaining a
healthy number like 80-90% - depending on expected runtime variance
and an educated guess of how hot the page cache is.