linux-kernel - Re: Recursive directory accounting for size, ctime, etc.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 5 Aug 2008 20:26:11 +0200
From:	Pavel Machek <pavel@...e.cz>
To:	Sage Weil <sage@...dream.net>
Cc:	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org,
	ceph-devel@...ts.sf.net
Subject: Re: Recursive directory accounting for size, ctime, etc.

On Tue 2008-07-15 11:28:22, Sage Weil wrote:
> All-
> 
> Ceph is a new distributed file system for Linux designed for scalability 
> (terabytes to exabytes, tens to thousands of storage nodes), reliability, 
> and performance.  The latest release (v0.3), aside from xattr support and 
> the usual slew of bugfixes, includes a unique (?) recursive accounting 
> infrastructure that allows statistics about all metadata nested beneath a 
> point in the directory hierarchy to be efficiently propagated up the tree.  
> Currently this includes a file and directory count, total bytes (summation 
> over file sizes), and most recent inode ctime.  For example, for a 
> directory like /home, Ceph can efficiently report the total number of 
> files, directories, and bytes contained by that entire subtree of the 
> directory hierarchy.
> 
> The file size summation is the most interesting, as it effectively gives 
> you directory-based quota space accounting with fine granularity.  In many 
> deployments, the quota _accounting_ is more important than actual 
> enforcement.  Anybody who has had to figure out what has filled/is filling 
> up a large volume will appreciate how cumbersome and inefficient 'du' can 
> be for that purpose--especially when you're in a hurry.
> 
> There are currently two ways to access the recursive stats via a standard 
> shell.  The first simply sets the directory st_size value to the 
> _recursive_ bytes ('rbytes') value (when the client is mounted with -o 
> rbytes).  For example (watch the directory sizes),
...

> Naturally, there are a few caveats:
> 
>  - There is some built-in delay before statistics fully propagate up 
> toward the root of the hierarchy.  Changes are propagated 
> opportunistically when lock/lease state allows, with an upper bound of (by 
> default) ~30 seconds for each level of directory nesting.

Having instant rctime would be very nice -- for stuff like locate and
speeding up kde startup.

> I'm extremely interested in what people think of overloading the file 
> system interface in this way.  Handy?  Crufty?  Dangerous?  Does anybody 

Too ugly to live.

What about new rstat() syscall?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/