linux-kernel - Re: userspace pagecache management tool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20070303170237.31d26382.akpm@linux-foundation.org>
Date:	Sat, 3 Mar 2007 17:02:37 -0800
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	Rik van Riel <riel@...hat.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: userspace pagecache management tool

On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <riel@...hat.com> wrote:

> Andrew Morton wrote:
> > On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <riel@...hat.com> wrote:
> > 
> >> backup program
> > 
> > A suitable policy for a backup program would probably be to invalidate any
> > output file(s) and to invalidate those pages of the input files which were
> > not in cache when the backup program first opened those files.  That way
> > the backup program will have no effect on the cache state, except for the
> > race situation where someone read an uncached file while the backup program
> > was reading from it too.
> 
> The use-once policy we have in the kernel should work
> perfectly fine for backups.  All we need to do is
> actually honor the accessed bit on active page cache
> pages, instead of flushing them onto the inactive
> list.
> 
> What am I overlooking?

That'll improve backups but will break other things.

To do this effectively we'd need to change the policy so that new pagecache
allocations cause no scanning of used-twice pages at all.  So that even
after many gigs of backing up, the working set is still there.

Problem is, (for example) what about the person who has 80% of memory in
used-twice state and who then reads a file or files which are 20% or more of
the size of memory, two or more times.  It'll be 100% cache misses, every time.
This will happen quite a lot.  IOW, once those pages are in used-twice state,
how does further pagecache activity ever get them _out_ of that state?  Only
by joining the used-twice page set, and that can't happen if the used-once-so-far
pages got reclaimed.

Doing a refault thing would help a bit, but stops working at a certain point.

> > This can be added in an hour or two with no kernel changes (use mincore).
> 
> mincore only works for mmaped areas, we'd need an fincore
> to work with file handles.

The LD_PRELOAD code has the fd and can mmap it to perform the pagecache
probe.

fincore() would be a bit neater, but given the rarity with which mincore()
is used it's perhaps hard to justify adding a slightly more efficient and
slightly more convenient subset of mincore().

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/