linux-ext4 - Re: Maildir quickly hitting max htree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <82C9B126-527B-4D41-8E01-84C560E06A3F@dilger.ca>
Date:   Sat, 13 Nov 2021 09:19:58 -0800
From:   Andreas Dilger <adilger@...ger.ca>
To:     Mark Hills <mark@...x.org>
Cc:     Ext4 Developers List <linux-ext4@...r.kernel.org>
Subject: Re: Maildir quickly hitting max htree

On Nov 13, 2021, at 04:05, Mark Hills <mark@...x.org> wrote:
> 
> Andreas, thanks for such a prompt reply.
> 
>> On Fri, 12 Nov 2021, Andreas Dilger wrote:
>> 
>>> On Nov 12, 2021, at 11:37, Mark Hills <mark@...x.org> wrote:
>>> 
>>> Surprised to hit a limit when handling a modest Maildir case; does 
>>> this reflect a bug?
>>> 
>>> rsync'ing to a new mail server, after fewer than 100,000 files there 
>>> are intermittent failures:
>> 
>> This is probably because you are using 1KB blocksize instead of 4KB, 
>> which reduces the size of each tree level by the cube of the ratio, so 
>> 64x. I guess that was selected because of very small files in the 
>> maildir?
> 
> Interesting! The 1Kb block size was not explicitly chosen. There was no 
> plan other than using the defaults.
> 
> However I did forget that this is a VM installed from a base image. The 
> root cause is likely to be that the /home partition has been enlarged from 
> a small size to 32Gb.
> 
> Is block size the only factor? If so, a patch like below (untested) could 
> make it clear it's relevant, and saved the question in this case.

The patch looks reasonable, but should be submitted separately with
[patch] in the subject so that it will not be lost.  

You can also add on your patch:

Reviewed-by: Andreas Dilger <adilger@...ger.ca>


Cheers, Andreas

> 
> [...]
>> If you have a relatively recent kernel, you can enable the "large_dir" 
>> feature to allow 3-level htree, which would be enough for another factor 
>> of 1024/8 = 128 more entries than now (~12M).
> 
> The system is not yet in use, so I think it's better we reformat here, and 
> get a block size chosen by the experts :)
> 
> These days I think VMs make it more common to enlarge a filesystem from a 
> small size. We could have picked this up earlier with a warning from 
> resize2fs; eg. if the block size will no longer match the one that would 
> be chosen by default. That would pick it up before anyone puts 1Kb block 
> size into production.
> 
> Thanks for identifying the issue.
> 
> -- 
> Mark
> 
> 
> From 8604c50be77a4bc56a91099598c409d5a3c1fdbe Mon Sep 17 00:00:00 2001
> From: Mark Hills <mark@...x.org>
> Date: Sat, 13 Nov 2021 11:46:50 +0000
> Subject: [PATCH] Block size has an effect on the index size
> 
> ---
> fs/ext4/namei.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
> index f3bbcd4efb56..8965bed4d7ff 100644
> --- a/fs/ext4/namei.c
> +++ b/fs/ext4/namei.c
> @@ -2454,8 +2454,9 @@ static int ext4_dx_add_entry(handle_t *handle, struct ext4_filename *fname,
>        }
>        if (add_level && levels == ext4_dir_htree_level(sb)) {
>            ext4_warning(sb, "Directory (ino: %lu) index full, "
> -                     "reach max htree level :%d",
> -                     dir->i_ino, levels);
> +                     "reach max htree level :%d"
> +                     "with block size %lu",
> +                     dir->i_ino, levels, sb->s_blocksize);
>            if (ext4_dir_htree_level(sb) < EXT4_HTREE_LEVEL) {
>                ext4_warning(sb, "Large directory feature is "
>                         "not enabled on this "
> -- 
> 2.33.1