linux-kernel - Re: Conveying memory pressure to userspace?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4643CCDF.7060608@mbligh.org>
Date:	Thu, 10 May 2007 18:54:39 -0700
From:	"Martin J. Bligh" <mbligh@...igh.org>
To:	Bas Westerbaan <bas.westerbaan@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Conveying memory pressure to userspace?

Bas Westerbaan wrote:
> Hello,
> 
> Quite a lot of userspace applications cache.  Firefox caches pages;
> mySQL caches queries;  libc' free (like practically all other
> userspace allocators) won't directly return the first byte of memory
> freed, etc.
> 
> These applications are unaware of memory pressure.  While the kernel
> is trying its best to to free memory by for instance dropping,
> possibly more valuable caches, userspace is left blissfully unaware.
> 
> Obviously this isn't a really big problem, given that we've still got
> swap to swap out those rarely used caches, except for when the caches
> aren't _that_ rarely used and of which the backing store (eg.
> precomputed values) might be faster than the disk to swap back the
> pages from.
> 
> A solution would be to either
> 
>  a) let the application make the kernel aware of pages that, when in
> memory pressure, may be dropped.  This would be tricky to implement
> for the userspace: it's hard to avoid an application to race into a
> dropped page.  However, the kernel can directly free a page from
> userspace, which makes it use full when under real pressure.  This in
> contrast to
> b) letting the application register itself with a cache share
> priority.  The application (and other aware applications) would then
> be able to query how fair they are at the moment proportional to their
> cache share priority.  Freeing would still be completely in their own
> hands.
> 
> 
> The only relevant related matter I could find were madvise and mincore.
> 
> With madvise pages can be marked to be unnecessary and these should be
> swapped out earlier.  With mincore one can determine whether pages are
> resident (not cached).  This would make an existing alternative to
> solution a.  However, this doesn't eliminate the writes to the swap
> and polling everytime before accessing a cache isn't really pretty.
> 
> I did consider guessing the memory pressure by looking at
> /proc/meminfo, but I think it isn't that accurate.

The prev_priority field in the zoneinfo stuff is more useful for
memory pressure. I'm playing with making a blocking callback that
can wake someone up when this gets down to a certain priority level
(prio=12 => everything's rosy, prio=0 => we're in deep shit).

> Before hacking something together (and being uncertain about the
> thoroughness with which I searched for existing work, sorry), I would
> like your thoughts on this.
> 
> Please CC me, I'm not in the list.
> 
>  Bas
> 

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/