Message-ID: <20090727165430.GD8332@duck.suse.cz>
Date:	Mon, 27 Jul 2009 18:54:30 +0200
From:	Jan Kara <jack@...e.cz>
To:	Jamie Lokier <jamie@...reable.org>
Cc:	Eric Paris <eparis@...hat.com>, linux-kernel@...r.kernel.org,
	linux-fsdevel@...r.kernel.org, malware-list@...sg.printk.net,
	Valdis.Kletnieks@...edu, greg@...ah.com, jcm@...hat.com,
	douglas.leeder@...hos.com, tytso@....edu, arjan@...radead.org,
	david@...g.hm, jengelh@...ozas.de, aviro@...hat.com,
	mrkafk@...il.com, alexl@...hat.com, jack@...e.cz,
	tvrtko.ursulin@...hos.com, a.p.zijlstra@...llo.nl,
	hch@...radead.org, alan@...rguk.ukuu.org.uk, mmorley@....in,
	pavel@...e.cz
Subject: Re: fanotify - overall design before I start sending patches

On Fri 24-07-09 23:48:14, Jamie Lokier wrote:
> > - subtree notification.
> > 	Currently to only watch /home and all of it's descendants one must
> > either register a directed watch on every directory or use a global
> > listener.  The global listener with ignored_mask is not as bad as it
> > sounds in my testing, but decent subtree registration and notification
> > would be a big win in a lot of people's mind.
> 
> I believe we've talked about one suggestion for how to do this, on
> lwn.net.  I'll repeat it here.
> 
> Efficient recursive notifications method:
> 
>    - You register for event on a directory with a RECURSIVE flag "give
>      me events for this directory and all paths below it".
> 
>    - That listener gets events for any access of the appropriate type
>      whose path is via that directory, *using the specific run-time
>      path used for the access*.
> 
>    - That _doesn't_ mean hard-link files need to know all their parent
>      directories, which would be silly and impossible.  The event path
>      is just the one used at run-time for access, by the application
>      attempting to open/write/whatever.
> 
>    - If a listener needs to track all accesses to a particular
>      hard-linked file, it's the responsibility of the listener to
>      ensure it listens to enough directories to cover every path to
>      that file - or listen to the file directly.  It knows from
>      i_nlink and the mount map when it has enough directories.
> 
>    - Notifying just the access path may seem counterintuitive, but in
>      fact it's what inotify and dnotify do already, and it does
>      actually work.  Often a listener is maintaining a cache or index
>      of some kind, in which case it will already have sufficient
>      knowledge about where the hard-linked files are (or know that it
>      needs an initial indexing), and whether it has covered enough
>      parent directories to see all accesses to them.
> 
>    - In practice it means each access traverses the path, following
>      parent directories until reaching a mount point, broadcasting
>      events on each one where there's a recursive listener.  That's
>      not as inefficient as it looks, because paths don't usually have
>      a large number of components.
> 
>    - I'm not sure exactly how fast/slow it is, though, and it may a
>      few thoughtfully cached flags in each dentry to elide traversals.
>      I won't discuss the details here, for fear of complicating the
>      discussion too much.  They might well mesh with the 'access
>      decision cache' flags you mentioned.
> 
>    - It is necessary that link(2) create an attribute-change event
>      (for i_nlink!) on the source path of the link.  dnotify/inotify
>      don't do that now (unless they changed recently), but they should
>      to make this work.
  About two years ago, I had a similar idea for a lightweight persistent
recursive modification. I even have a proof-of-concept patch against 2.6.23
(attached to get an idea) which works nicely. I've aimed at things like
efficient backup or desktop indexing which are interested in processing
lots of changes in a batch once in a longer period of time...  Actually I
believe this kind of use is quite different from the kind of use fanotify
aims at and maybe different approaches even make sence here... My approach
is only able to give the information "something in the subtree has changed"
via an inode flag in the directory inode and the application has to track
down what exactly it was (by recursively looking on the flags of the
subdirectories and stating regular files). The benefit is it's rather
scalable I believe.
  Generally the trouble with this approach is that one has to handle
hardlinks, bind mounts and filesystems which don't support persistent
storage of your attributes. It's all doable but tricky, and I'm still
trying to get all the details right in a shared library wrapping up the
kernel feature (well, one of the problems also is I get to this for only a
few days a year :().

								Honza
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR

View attachment "ext3-2.6.23-1-i_flags_atomicity.diff" of type "text/x-patch" (9480 bytes)

View attachment "ext3-2.6.23-2-make_frags_unused.diff" of type "text/x-patch" (5668 bytes)

View attachment "ext3-2.6.23-3-recursive_mtime.diff" of type "text/x-patch" (15004 bytes)

Powered by blists - more mailing lists