linux-ext4 - Re: File limit inside a single directory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <160BB0B1-21A0-4689-8EF4-DBC3E3D52651@dilger.ca>
Date:	Sat, 14 Feb 2015 11:25:08 -0700
From:	Andreas Dilger <adilger@...ger.ca>
To:	"suntrop@....de" <suntrop@....de>
Cc:	Eric Sandeen <sandeen@...hat.com>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
Subject: Re: File limit inside a single directory

If you are using ext3 then you will get an EMLINK error at 32000 subdirs. 
If you are using ext4 with the "dir_nlink" feature enabled then you
can create several million subdirs if you want.

As for performance, if you are not regularly creating and deleting the
directory I don't think you would even notice the performance difference
of a large directory.  The create and unlink rate get somewhat slower at
100k files (from 25k/sec to 10k/sec) and 1M entries or so (to 5k/sec).
For lookups the whole directory will easily fit into cache (at about 40MB
for 1M entries) so the lookup rate is not affected much by size. 

Cheers, Andreas

> On Feb 14, 2015, at 03:17, suntrop@....de wrote:
> 
> Sorry for confusion, but now I am confused :-) My setup will be like
> /files/
> /files/1/
> /files/2/
> /files/3/
> (...)
> /files/24244/
> (...)
> /files/113524/
> 
> Thus all folders reside within the /files/ directory (can't change that). 
> What happens when the max link count reach 32,000 and I create more folders? Will this result in any errors or problems? I’m using ext3. Is the only consequence that the links aren’t counting up any more, like ext4?
>  
>  
> 
> Gesendet: Freitag, 13. Februar 2015 um 18:28 Uhr
> Von: "Eric Sandeen" <sandeen@...hat.com>
> An: suntrop@....de, "Andreas Dilger" <adilger@...ger.ca>
> Cc: "linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
> Betreff: Re: File limit inside a single directory
>> On 2/13/15 10:49 AM, suntrop@....de wrote:
>> Thanks guys. I was afraid of having a couple 100K (at most). The
>> server support team told me not to have more than 10 to 20K. There
>> seems to be a misconception (for me and people from the CMS) about
>> the 32K subdirectory limit, but this ins't for files/folders within a
>> single directory but rather nested directories like /1/2/3.../32000
> 
> You've confused things a bit here, FWIW.
> 
> The 32k (well, 32000 because, sure) limit on ext3 is max link count;
> each subdirectory increases the link count on its parent, but only
> its parent. It's not about deep nesting, or about files in a dir.
> It's only about subdirs in a parent dir.
> 
> # mkdir dir
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 2 // . and .. entries
> # mkdir dir/subdir1 dir/subdir2 dir/subdir3
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 5
> # mkdir dir/subdir1/subsubdir1 dir/subdir1/subsubdir2 dir/subdir1/subsubdir3
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 5
> 
> ext4 bumped that max to 64000, and just stops counting if that number
> gets exceeded...
> 
> -Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html