linux-kernel - Re: userspace pagecache management tool

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <20070308001235.d7cb099e.akpm@linux-foundation.org>
Date:	Thu, 8 Mar 2007 00:12:35 -0800
From:	Andrew Morton <akpm@...ux-foundation.org>
To:	Vaidyanathan Srinivasan <svaidy@...ux.vnet.ibm.com>
Cc:	bert.hubert@...herlabs.nl, riel@...hat.com,
	linux-kernel@...r.kernel.org
Subject: Re: userspace pagecache management tool

> On Thu, 08 Mar 2007 13:29:02 +0530 Vaidyanathan Srinivasan <svaidy@...ux.vnet.ibm.com> wrote:
> > That all sounds reasonably doable.  It'd be pretty complex to do it
> > in-kernel but we could do it there too.  Problem is if course that the
> > above strategy is explicitly optimised for the backup program and if it's
> > in-kernel it becomes applicable to all other workloads.
> 
> This strategy looks very good.  However we are not considering the
> performance impact on the 'backup' application as such.  By removing
> pagecache pages brought in by the application without the knowledge of
> the applications usage and behavior may severely affect its performance.
> 
> Certainly we are interested in improving system performance at the
> cost certain applications, but not to an extend that the backup
> process will drag on and on to an unreasonable amount of time.
> 
> Also backup processes may consist of a group of applications working
> on the same stream of data.  Like compression program, encryption
> program etc which could be independent applications.

Well yes, if the application is that funky then suitably funky userspace
tricks will be needed to avoid hurting it.

> We should consider having a limit on pagecache usage rather than
> denying any space in the pagecache for these applications.

That's what containerisation is for:

	run-in-container --memory=16M /bin/backup-program

This can be done today with x86_64 fake-numa, controlled by cpusets.  One
day, when we get our containerisation story sorted out, things will be more
convenient...

> Can fadvice() be enhanced to have a limit on pagecache usage and
> reclaim used pages in LRU order?  This way data stays for a little
> while for other applications to pickup from pagecache.
> 
> Pages already in memory or brought in by other applications need not
> be placed in this list and hence we prevent any collateral pageouts.

We could teach the presently-unimplemented POSIX_FADV_NOREUSE to dump this
file's pages at the tail of the inactive list (after cleaning them if
needed).  That way, they're the first to get reclaimed.

The standard says "Specifies that the application expects to access the
specified data once and then not reuse it thereafter." That's a bit
ambiguous: it it before the process accessed the data, or after?  Before, I
suspect.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/