linux-ext4 - Re: bigalloc and max file size

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 01 Nov 2011 20:22:17 +0800
From:	Coly Li <i@...y.li>
To:	Theodore Tso <tytso@....EDU>
CC:	Andreas Dilger <adilger@...ger.ca>,
	Andreas Dilger <adilger@...mcloud.com>,
	linux-ext4 development <linux-ext4@...r.kernel.org>,
	Alex Zhuravlev <bzzz@...mcloud.com>, Tao Ma <tm@....ma>,
	"hao.bigrat@...il.com" <hao.bigrat@...il.com>
Subject: Re: bigalloc and max file size

On 2011年11月01日 19:47, Theodore Tso Wrote:
> 
> On Oct 31, 2011, at 9:10 PM, Coly Li wrote:
>>
>>> I assume the issue then is you
>>> want to minimize the number of extents, limited by the 15-bit extent
>>> length field?
>> Not only extents, but also minimize inode table blocks, bitmap blocks.
> 
> 
> So this makes no sense to me.  Bigalloc doesn't have any effect on the number of inode table blocks, and while it certainly shrinks the number block allocation bitmap blocks, changing the extent tree format has no effect on the number of bitmap blocks.
> 

In mkfs.ext4, with -N 16, there are still much more inodes allocated, because for each block group there has to be some
inode blocks to be allocated. For bigalloc, same file system size may have less block groups, which results less inode
blocks.

>>>
>>> What cluster size are you thinking about?
>> Currently we test 1MB cluster size. The extreme ideal configuration (of one use case) is, there is only one block group
>> on the whole file system. (In this use case) we are willing to try biggest possible cluster size if we are able to.
> 
> This is where you have a single file which is nearly as big as the entire file system?   In that case, why are you using an ext4 file system at all?   Why not just use a raw partition instead, plus an auxiliary partition for the smaller files?
> 

To make management simple and easy, sys-admin team trends to use on-hand Linux tools to manage the data, a raw disk
format is the last choice. Yes, your suggestion make senses, while there is open source project using raw disk for the
similar purpose, but it takes quit long time to deploy another stable system to online servers ...

> I'm not being critical; I'm just trying to understand your use case and constraints.
> 

I try to explain in one line why we are interested on bigalloc: try best to minimize all kinds of metadata space, both
on disk and in memory.

Thanks.
-- 
Coly Li
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html