linux-kernel - Re: userspace pagecache management tool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20070306134046.5be821f5.akpm@linux-foundation.org>
Date:	Tue, 6 Mar 2007 13:40:46 -0800
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	Pádraig Brady <P@...igBrady.com>
Cc:	bert hubert <bert.hubert@...herlabs.nl>,
	Rik van Riel <riel@...hat.com>, linux-kernel@...r.kernel.org
Subject: Re: userspace pagecache management tool

On Tue, 06 Mar 2007 12:10:49 +0000
P__draig Brady <P@...igBrady.com> wrote:

> Andrew Morton wrote:
> > Yes.  Let's flesh it out the backup program policy some more:
> > 
> > - Unconditionally invalidate output files
> > 
> > - on entry to read(), probe pagecache, record which pages in the range are present
> > 
> > - on entry to next read(), shoot down those pages from the previous read
> >   which weren't in pagecache.
> > 
> > - But we can do better!  LRU the page's files up to a certain number of pages.
> > 
> > - Once that point is exceeded, we need to reclaim some pages.  Which
> >   ones?  Well, we've been observing all reads, so we can record which pages
> >   were referenced once, and which ones were referenced multiple times so we
> >   can do arbitrarily complex page aging in there.
> > 
> > - On close(), nuke all pages which weren't in core during open(), even if
> >   this app referenced them multiple times.
> > 
> > - If the backup program decided to read its input files with mmap we're
> >   rather screwed.  We can't intercept pagefaults so the best we can do is
> >   to restore the file's pagecache to its previous state on close().
> > 
> >   Or if it's really a problem, get control in there somehow and
> >   periodically poll the pagecache occupancy via mincore(), use madvise()
> >   then fadvise() to trim it back.
> > 
> > That all sounds reasonably doable.  It'd be pretty complex to do it
> > in-kernel but we could do it there too.  Problem is if course that the
> > above strategy is explicitly optimised for the backup program and if it's
> > in-kernel it becomes applicable to all other workloads.
> 
> I can see the above being possible, but I can't see the reason
> for exposing that complexity to userspace.

That's sophistication, not complexity.  It doesn't have to do all that stuff
to be effective.

> If I'm the target
> audience for that API then it's broken as I'd mess it up,
> or would take too long to get it right.
> 
> Can't we just fix the posix_fadvise() implementation to
> only evict pages paged in by the current process.

The kernel doesn't have that information.

> Perhaps one could possibly just evict pages with _mapcount==0 ?

That is the present fadvise(FADV_DONTNEED) behaviour.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/