linux-kernel - Re: inotify limits - thousands (tens of thousands?) of watches

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <200905201448.21222.ms@teamix.de>
Date:	Wed, 20 May 2009 14:48:16 +0200
From:	Martin Steigerwald <ms@...mix.de>
To:	Marcin Krol <mrkafk@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: inotify limits - thousands (tens of thousands?) of watches

Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
> Martin Steigerwald wrote:
> > Hmmm, I think you could just run a rsync periodically. It might even be
> > faster detecting changed files.
>
> I beg to differ on this: rsync does quite intensive (in terms of disk
> activity and CPU activity) comparisons at the beginning of
> synchronization. It's pretty light later, true, but running rsync every
> few minutes on entire /home is IMO out of question.

It depends on the amount of directories being watched and the amount directory 
entries in there. Rsync version 3 starts of synchronizing before building up 
the entire trees on both sides. It uses an incremental approach and this 
appears to make quite a difference.

For syncing /home I wonder whether inotify or fsnotify/fanotify is the right 
approach. A cluster filesystem of some sort comes to my mind instead. But 
question is, which one really works good enough. Didn't do any testing in 
that area yet.

> > I wrote a ruby script using libinotify-ruby which does just that. I only
> > syncs on demand tough. I.e. when someplace places a special sync file in
> > a watched directory.
> >
> > That script is running productively for well over a year now.
>
> Good to know the idea is not totally off the wall.. Thanks.

Main difference is that the sync is only triggered on demand. But the watching 
part appears to work quite nicely. Its not used for /home, but for the static 
html/php/image content for an apache2.2 which handles a lot of domains.

> Anyway, I'll try using fsnotify / fanotify.
>
> My main gripe with it, though, is that it is not in the mainline kernel,
> and thus in all probability it is not tested as widely as inotify.

Well then it will get more testing now I think ;-).

> Are there any chances for its inclusion in the near future?

Indeed would be nice to have.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

Download attachment "signature.asc " of type "application/pgp-signature" (198 bytes)