linux-ext4 - Re: ext4 64bit (disk >16TB) question

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <487D0572.50503@gmail.com>
Date:	Tue, 15 Jul 2008 16:15:46 -0400
From:	Ric Wheeler <ricwheeler@...il.com>
To:	Andreas Dilger <adilger@....com>
CC:	"Jose R. Santos" <jrs@...ibm.com>,
	Goswin von Brederlow <goswin-v-b@....de>,
	linux-ext4@...r.kernel.org
Subject: Re: ext4 64bit (disk >16TB) question

Andreas Dilger wrote:
> On Jul 15, 2008  13:27 -0500, Jose R. Santos wrote:
>   
>> On Mon, 14 Jul 2008 21:50:56 +0200
>> Goswin von Brederlow <goswin-v-b@....de> wrote:
>>     
>>> we are using lustre on a cluster of servers and raid boxes. Currently
>>> lustre is based on the ext3 code and has a limit of 8TiB for each
>>> filesystem. For us that results on having to split a servers storage
>>> into up to 4 chunks and run one fs on each which I would rather avoid.
>>> The solution for this would be to rebase the lustre patches to use
>>> ext4 instead, which should also reduce the patch set considerably.
>>> Lustre already patches a lot of ext4 features into the ext3 base.
>>>
>>>
>>> But before I start rebasing lustre I though I would first test out
>>> plain ext4 so I know any bugs I find will be from my rebasing and not
>>> already existing in ext4 itself. And there I run into a big problem:
>>> Current e2fsprogs (1.41) seem to be totaly unable to handle the ext4 64BIT
>>> feature, i.e. filesystems larger than 16TiB. The mkfs.ext4 always
>>> stops saying the disk exceeds the 32bit block count. And looking at
>>> the code I see a lot of blk_t (instead of blk64_t) and unsigned long
>>> (instead of unsigned long long [or even better blk64_t]) usage.
>>>
>>> I found ext4 64bit patches for e2fsprogs 1.39 that fix at least
>>> mkfs. Does anyone know if there is an updated patch set for 1.41
>>> anywhere? And when will that be added to e2fsprogs upstream?
>>>       
>> I've recently submitted a set of patches that covers most of the API
>> changes needed to support >16TB file systems (missing Ted bitmap
>> support of course).  Once the bitmap support is included, it _SHOULD_
>> be relatively painless to add mke2fs support with this series of patches.
>>     
>
> Jose,
> while waiting for the "efficient bitmap" support, how hard would it be
> to implement "inefficient bitmaps" that just malloc some GB of memory
> if needed?  This would at least allow people with huge devices to test
> mke2fs/ext4/e2fsck in the meantime.
>
> Cheers, Andreas
> --
> Andreas Dilger
> Sr. Staff Engineer, Lustre Group
> Sun Microsystems of Canada, Inc.
>
>   

I think that would be very useful - how much DRAM would we need for a 
16TB file system ;-) ?

ric

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html