linux-kernel - Re: efficient access to "rotational"; new fcntl?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 19 Sep 2009 13:25:23 +0200
From:	Willy Tarreau <w@....eu>
To:	Jim Meyering <jim@...ering.net>
Cc:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: efficient access to "rotational";  new fcntl?

On Fri, Sep 18, 2009 at 09:31:50PM +0200, Jim Meyering wrote:
> About a year ago, I fixed GNU rm to avoid quadratic readdir/stat seek
> penalties on ext3 and ext4 that became overwhelming on directories with
> many entries (1M entries started to look like infloop or DoS).
> 
>   From coreutils-7.0 NEWS:
> 
>     chgrp, chmod, chown, chcon, du, rm: now all display linear performance,
>     even when operating on million-entry directories on ext3 and ext4 file
>     systems.  Before, they would exhibit O(N^2) performance, due to linear
>     per-entry seek time cost when operating on entries in readdir order.
>     Rm was improved directly, while the others inherit the improvement
>     from the newer version of fts in gnulib.
> 
> To do that efficiently, I changed the core readdir loop (mainly via
> gnulib's fts.c) to preprocess entries, when needed, by sorting them
> on inode, and *then* processing them.  This optimization is enabled
> only when the affected file system is of a type likely to benefit,
> and when the number of directory entries is large enough to matter.

This is excellent. I've been used to play with variations of
'find -printf "%i %p\n"|sort -n|cut -f2|xargs <cmd>' to sort file
operations by inode to speed them up, especially when rm'ing a
kernel tree or when doing a prefetch to build from hot cache.

I think you might notice excellent results on CDs where seeks cost a
lot. That's where I first use the find|sort method above. I'm really
happy to see that such mechanisms are now implemented inside my
everyday tools, and I can't wait to upgrade!

Regards,
Willy

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/