linux-kernel - Re: solving(?) the updatedb problem w/ the kernel cache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20070727172219.GO3357@trakker.ca>
Date:	Fri, 27 Jul 2007 10:22:19 -0700
From:	Kevin Lindsay <kevin@...kker.ca>
To:	Ray Lee <ray-lk@...rabbit.org>
Cc:	Douglas J Hunley <doug@...ley.homeip.net>, slocate@...kker.ca,
	linux-kernel@...r.kernel.org
Subject: Re: solving(?) the updatedb problem w/ the kernel cache

On Fri, Jul 27, 2007 at 09:42:27AM -0700, Ray Lee wrote:

> On 7/27/07, Douglas J Hunley <doug@...ley.homeip.net> wrote:
> > I've been following lkml for a little while (not understanding it all, but
> > following nonetheless <g>) and I've noticed that in a lot of the talks about
> > schedulers, elevators, and performance, the issue of running updatedb and its
> > effects on the kernel's fs cache seems to recur. I've also yet to see anyone
> > present a solution that others think is worth pursuing. I'm curious why we're
> > trying to solve the problem, when we can simply avoid the problem to begin
> > with by making use of inotify and introducing a new user-space
> > daemon, 'located'.
> 
> inotify doesn't scale for lots of directories. I have about 18,000
> directories under ~ on my laptop, and that's with a few source trees
> that I use infrequently tarballed up.
> 
> But yes, if we had a full filesystem events notifier, then we could
> just toss updatedb aside and have the benefit of a live index into the
> system. It's been suggested before, at least by me. Other projects
> want this as well, such as an on-demand virus scanner, or a live
> backup to another site, or beagle/tracker who would like to index
> documents on the fly. beagled already uses inotify, I think, but as it
> takes over my system (in a bad way) whenever I tried to run it, I had
> no choice but to remove it.
> 
> Perhaps it was choking on the 18k subdirectories, dunno.

The interface for inotify requires you to explicity watch files and folders.
As Ray suggests, I am also skeptical that using inotify to watch 18k of inodes
is very efficient, although it would be nice to be wrong. Possibly someone
needs to take a peak into OS X's fsevents mechanism?

The other problems with this approach is that the locate DB uses incremental
encoding. Each change to the filesystem may require large protions or possibly
a complete re-encoding of the database. An alterntive DB format would need to
be considered.

Kevin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/