linux-ext4 - Re: 20TB ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <A7DF8999-362B-459B-9D8A-007F257C9135@dilger.ca>
Date:	Wed, 15 Dec 2010 02:21:54 -0700
From:	Andreas Dilger <adilger@...ger.ca>
To:	Stephan Boettcher <boettcher@...sik.uni-kiel.de>
Cc:	linux-ext4@...r.kernel.org
Subject: Re: 20TB ext4

On 2010-12-14, at 13:51, Stephan Boettcher wrote:
> Stephan Boettcher <boettcher@...sik.uni-kiel.de> writes:
>> Andreas Dilger <adilger@...ger.ca> writes:
>>> If you have dedicated server nodes, and you want to be able to use
>>> these 20TB from multiple clients, you might consider using Lustre,
>>> which uses ext4 as the back-end storage, and can scale to many PB
>>> filesystems (largest known filesystem is 20PB, from 1344 * 8TB
>>> separate ext4 filesystems).
>> 
>> I like thinks to be as simple and transparent as possible :-) The plan
>> is to export the fs via NFS.  I will hit the 16 TB limit again, will I?
>> I did not test that part yet.  The NFS clients will then probably be
>> required to run 64-bit kernels as well.
> 
> Excuse me for not knowing all that much about how linux filesystems
> work. I was surprised that I could export the 20TB filesystem via NFS
> and mount it on a 32-bit (2.6.31) system.  Do I need to expect failures
> when I try to actually use it that way, or does the nfs filesystem not
> use the page cache or something, so that the 16TB limit does not apply?

The 16TB limit is related to the 32-bit page index * PAGE_SIZE (4kB), so 2^32 * 2^12 = 2^44 = 16 * 2^40 = 16TB.

Because NFS is not exporting the whole block device, just file access, the filesystem size is not limited by the 32-bit page index.  However, individual file access would be limited to 16TB, and it may be there are lower limits on the file size.

With Lustre, we had filesystems of hundreds of TB in size with 32-bit clients and servers.  The individual Lustre backing filesystems are still under 16TB, so there is not a problem to serve them from 32-bit nodes.  In any case, nobody buys 32-bit systems today so we don't test this much anymore.

Cheers, Andreas





--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html