linux-ext4 - Re: bigalloc and max file size

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Thu, 27 Oct 2011 02:35:05 -0400
From:	Theodore Tso <tytso@....EDU>
To:	Andreas Dilger <aedilger@...il.com>
Cc:	Theodore Tso <tytso@....edu>,
	linux-ext4 development <linux-ext4@...r.kernel.org>,
	Alex Zhuravlev <bzzz@...mcloud.com>
Subject: Re: bigalloc and max file size

On Oct 26, 2011, at 7:36 PM, Andreas Dilger wrote:

> Ted,
> we were having a discussion about bigalloc and the maximum file size
> (as limited by the 2^32 logical block number in struct ext4_extent).
> 
> Currently the maximum file size is blocksize * 2^32, 16TB for 4kB blocks.
> 
> Since it is not possible to allocate sub-blocks in the bigalloc code,
> what about storing the "chunk number" in the extent logical block?

This adds all sorts of complexity, and it's why we can't simply increase the block size above a page size.  Basically, avoiding all of the complex changes needed was the whole point of the bigalloc feature.

Effectively, if we stored the "chunk number" in the extent tree blocks, it's equivalent of setting the block size to the larger chunk size.  Among other things, it means that we have to support setting or clearing the uninitialized bit in chunks larger than a page, and how dirtying a 4k page in a sparse file would require clearing 64k or 1 megabyte (whatever the chunk size might be) either on disk or in memory.

In any case, it's not a simple change that we can make before the merge window.   If someone wants to work on that, we can support that simply by supporting block size > page size, with all of the complexity and pain which has prevented us from supporting that for the last ten years….

-- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html