linux-kernel - Re: [PATCH v5 3/3] fadvise: implement POSIX_FADV

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20120227104646.GA1700@thinkpad>
Date:	Mon, 27 Feb 2012 11:46:46 +0100
From:	Andrea Righi <andrea@...terlinux.com>
To:	KAMEZAWA Hiroyuki <kamezawa.hiroyu@...fujitsu.com>
Cc:	Andrew Morton <akpm@...ux-foundation.org>,
	Minchan Kim <minchan.kim@...il.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Johannes Weiner <jweiner@...hat.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Rik van Riel <riel@...hat.com>,
	Hugh Dickins <hughd@...gle.com>,
	Alexander Viro <viro@...iv.linux.org.uk>,
	Shaohua Li <shaohua.li@...el.com>,
	Pádraig Brady <P@...igBrady.com>,
	John Stultz <john.stultz@...aro.org>,
	Jerry James <jamesjer@...terlinux.com>,
	Julius Plenz <julius@...nz.com>, linux-mm <linux-mm@...ck.org>,
	linux-fsdevel@...r.kernel.org, LKML <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v5 3/3] fadvise: implement POSIX_FADV_NOREUSE

On Mon, Feb 27, 2012 at 11:33:38AM +0900, KAMEZAWA Hiroyuki wrote:
> On Sun, 12 Feb 2012 01:21:38 +0100
> Andrea Righi <andrea@...terlinux.com> wrote:
> 
> > According to the POSIX standard the POSIX_FADV_NOREUSE hint means that
> > the application expects to access the specified data once and then not
> > reuse it thereafter.
> > 
> > It seems that the expected behavior is to implement a drop-behind
> > policy where the application can set certain intervals of a file as
> > FADV_NOREUSE _before_ accessing the data.
> > 
> > An interesting usage of this hint is to guarantee that pages marked as
> > FADV_NOREUSE will never blow away the pages of the current working set.
> > 
> > A possible solution to satisfy this requirement is to prevent lru
> > activation of the pages marked as FADV_NOREUSE, in other words, never
> > add pages marked as FADV_NOREUSE to the active lru list. Moreover, all
> > the file cache pages in a FADV_NOREUSE range can be immediately dropped
> > after a read if the page was not present in the file cache before.
> > 
> > In general, the purpose of this approach is to preserve as much as
> > possible the previous state of the file cache memory before reading data
> > in ranges marked by FADV_NOREUSE.
> > 
> > All the pages read before (pre-)setting them as FADV_NOREUSE should be
> > treated as normal, so they can be added to the active lru list as usual
> > if they're accessed multiple times.
> > 
> > Only after setting them as FADV_NOREUSE we can prevent them for being
> > promoted to the active lru list. If they are already in the active lru
> > list before calling FADV_NOREUSE we should keep them there, but if they
> > quit from the active list they can't get back anymore (except by
> > explicitly setting a different caching hint).
> > 
> 
> >From this part, it seems the behavior of systemcall is highly depends on
> interanal kernel implemenatation...

Yes. If in a future kernel we'll decide to remove the active/inactive
lru lists we also need to change the implementation of FADV_NOREUSE.

However, I think that any solution for a feature that allows to not
disturb the state of the page cache has inevitably something dependent
on internal kernel implementation...

Probably a more generic concept to document is that FADV_NOREUSE is an
advice from the application to never consider the marked pages as part
of the working set (for any possible meaning/implementation of "working
set").

> 
> 
> > To achieve this goal we need to maintain the list of file ranges marked
> > as FADV_NOREUSE until the pages are dropped from the page cache, or the
> > inode is deleted, or they're explicitly marked to use a different cache
> > behavior (FADV_NORMAL | FADV_WILLNEED).
> > 
> > The list of FADV_NOREUSE ranges is maintained in the address_space
> > structure using an interval tree (kinterval).
> > 
> > Signed-off-by: Andrea Righi <andrea@...terlinux.com>
> 
> 
> Once an appliation sets a range of file as FILEMAP_CACHE_ONCE,
> the effects will last until the inode is dropped....right ?
> Won't this cause troubles which cannot be detected
> (because kinterval information is hidden.) ?
> 
> I'm not sure but FADV_NOREUSE seems like one-shot call and should not have
> very long time of effect (after the application exits.)
> Can't we ties the liftime of kinteval to the application/file descriptor ?

Yes, I'm also concerned about this. Using FADV_NOREUSE may also affect
the page cache behavior of other applications.

I like the idea to tie the FADV_NOREUSE ranges to the file descriptor.
In addition to the shorter lifetime it has the advantage that the policy
is applied only to the application that is actually using this feature
and not also to the other apps running in the system.

I'll consider this possibility for sure if I'll post a new version of
this patch set.

Thanks!
-Andrea
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/