linux-ext4 - Re: How capacious and well-indexed are ext4, xfs and btrfs directories?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <9073312A-DB74-4ADF-8E12-6C04E15FE34C@fb.com>
Date:   Mon, 17 May 2021 23:40:51 +0000
From:   Chris Mason <clm@...com>
To:     Dave Chinner <david@...morbit.com>
CC:     David Howells <dhowells@...hat.com>, Theodore Ts'o <tytso@....edu>,
        Andreas Dilger <adilger.kernel@...ger.ca>,
        "Darrick J. Wong" <djwong@...nel.org>,
        "linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
        "linux-xfs@...r.kernel.org" <linux-xfs@...r.kernel.org>,
        "linux-btrfs@...r.kernel.org" <linux-btrfs@...r.kernel.org>,
        "linux-cachefs@...hat.com" <linux-cachefs@...hat.com>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>
Subject: Re: How capacious and well-indexed are ext4, xfs and btrfs
 directories?


> On May 17, 2021, at 7:22 PM, Dave Chinner <david@...morbit.com> wrote:
> 
> On Mon, May 17, 2021 at 04:06:58PM +0100, David Howells wrote:
> 
>> What I'd like to do is remove the fanout directories, so that for each logical
>> "volume"[*] I have a single directory with all the files in it.  But that
>> means sticking massive amounts of entries into a single directory and hoping
>> it (a) isn't too slow and (b) doesn't hit the capacity limit.
> 
> Note that if you use a single directory, you are effectively single
> threading modifications to your file index. You still need to use
> fanout directories if you want concurrency during modification for
> the cachefiles index, but that's a different design criteria
> compared to directory capacity and modification/lookup scalability.

Unless you’re doing one subvol per fan directory, the btrfs results should be really similar either way.  It’s all getting indexed in the same btree, the keys just change based on the parent dir.

The biggest difference should be what Dave calls out here, where directory locking at the vfs level might be a bottleneck.

-chris