linux-ext4 - Re: ext4 scaling limits ?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <32A4A230-566F-4476-A516-2C6C4BA5C1C6@dilger.ca>
Date:   Tue, 21 Mar 2017 17:48:11 -0400
From:   Andreas Dilger <adilger@...ger.ca>
To:     Manish Katiyar <mkatiyar@...il.com>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: ext4 scaling limits ?

While it is true that e2fsck does not free memory during operation, in
practice this is not a problem. Even for large filesystems (say 32-48TB)
it will only use around 8-12GB of RAM so that is very reasonable for a
server today. 

The rough estimate that I use for e2fsck is 1 byte of RAM per block. 

Cheers, Andreas

> On Mar 21, 2017, at 16:07, Manish Katiyar <mkatiyar@...il.com> wrote:
> 
> Hi,
> 
> I was looking at e2fsck code to see if there are any limits on running
> e2fsck on large ext4 filesystems. From the code it looks like all the
> required metadata while e2fsck is running is only kept in memory and
> is only flushed to disk when the appropriate changes are corrected.
> (Except the undo file case).
> There doesn't seem to be a case/code where we have to periodically
> flush some tracking metadata while it is running, just because we have
> too much of incore tracking data and may ran out of memory (looks like
> code will simply return failure if ext2fs_get_mem() returns failure)
> 
> Appreciate if someone can confirm that my understanding is correct ?
> 
> Thanks -
> Manish