linux-ext4 - Re: bigalloc and max file size

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4EAE6BD4.9080705@coly.li>
Date:	Mon, 31 Oct 2011 17:35:16 +0800
From:	Coly Li <i@...y.li>
To:	Theodore Tso <tytso@....EDU>
CC:	Andreas Dilger <adilger@...mcloud.com>,
	linux-ext4 development <linux-ext4@...r.kernel.org>,
	Alex Zhuravlev <bzzz@...mcloud.com>, Tao Ma <tm@....ma>,
	"hao.bigrat@...il.com" <hao.bigrat@...il.com>
Subject: Re: bigalloc and max file size

On 2011年10月31日 03:49, Theodore Tso Wrote:
> 
> On Oct 30, 2011, at 1:37 AM, Coly Li wrote:
> 
>> Forgive me if this is out of topic.
>> In our test, allocating directories W/ bigalloc and W/O inline-data may occupy most of disk space. By now Ext4
>> inline-data is not merged yet, I just wondering how Google uses bigalloc without inline-data patch set ?
> 
> It depends on how many directories you have (i.e, how deep your directory structure is) and how many small files you have in the file system as to whether bigalloc w/o inline-data has an acceptable overhead or not.
[snip]
> I'm not against your patch set, however; I just haven't had time to look at them, at all (nor the secure delete patch set, etc.) .   Between organizing the kernel summit, the kernel.org compromise, and some high priority bugs at $WORK, things have just been too busy.  Sorry for that; I'll get to them after the merge window and post-merge bug fixing is under control.

Hi Ted,

In our test, bigalloc without inline-data dose not work very well with deep directory structure, e.g. Hadoop or Squid,
because small directories occupies all disk space. That's why I asked the question. Thanks for your patient reply, it
makes sense for me :-)

Back to our topic, Ext4 doesn't have too much on-disk incompatible flag-bits now. If we get current bigalloc code merged
now, we have to use another incompatible bit when we merge cluster/chunk based extent patch set. Further more, we
observe performance regression without cluster-based-extent on file system umount (as Tao mentioned in this thread).
IMHO, without inline-data and cluster-based-extent, current bigalloc code is a little bit inperfect for many users.

Bigalloc is a very useful feature, can we consider making it better before getting merged ?

Thanks.
-- 
Coly Li
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html