linux-ext4 - Re: How capacious and well-indexed are ext4, xfs and btrfs directories?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <272939.1621322659@warthog.procyon.org.uk>
Date:   Tue, 18 May 2021 08:24:19 +0100
From:   David Howells <dhowells@...hat.com>
To:     Dave Chinner <david@...morbit.com>
Cc:     dhowells@...hat.com, Theodore Ts'o <tytso@....edu>,
        Andreas Dilger <adilger.kernel@...ger.ca>,
        "Darrick J. Wong" <djwong@...nel.org>, Chris Mason <clm@...com>,
        linux-ext4@...r.kernel.org, linux-xfs@...r.kernel.org,
        linux-btrfs@...r.kernel.org, linux-cachefs@...hat.com,
        linux-fsdevel@...r.kernel.org
Subject: Re: How capacious and well-indexed are ext4, xfs and btrfs directories?

Dave Chinner <david@...morbit.com> wrote:

> > What I'd like to do is remove the fanout directories, so that for each logical
> > "volume"[*] I have a single directory with all the files in it.  But that
> > means sticking massive amounts of entries into a single directory and hoping
> > it (a) isn't too slow and (b) doesn't hit the capacity limit.
> 
> Note that if you use a single directory, you are effectively single
> threading modifications to your file index. You still need to use
> fanout directories if you want concurrency during modification for
> the cachefiles index, but that's a different design criteria
> compared to directory capacity and modification/lookup scalability.

I knew there was something I was overlooking.  This might be a more important
criterion.  I should try benchmarking this, see what difference it makes
eliminating the extra lookup step (which is probably cheap) versus the
concurrency.

David