linux-ext4 - Re: bigalloc and max file size

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4EAE780D.3090005@tao.ma>
Date:	Mon, 31 Oct 2011 18:27:25 +0800
From:	Tao Ma <tm@....ma>
To:	Theodore Tso <tytso@....EDU>
CC:	Andreas Dilger <adilger@...mcloud.com>,
	linux-ext4 development <linux-ext4@...r.kernel.org>,
	Alex Zhuravlev <bzzz@...mcloud.com>,
	"hao.bigrat@...il.com" <hao.bigrat@...il.com>
Subject: Re: bigalloc and max file size

On 10/31/2011 06:15 PM, Theodore Tso wrote:
> 
> On Oct 27, 2011, at 11:31 PM, Tao Ma wrote:
> 
>> Forget to say, if we increase the extent length to be cluster, there are
>> also a good side effect. ;) Current bigalloc has a severe performance
>> regression in the following test case:
>> mount -t ext4 /dev/sdb1 /mnt/ext4
>> cp linux-3.0.tar.gz /mnt/ext4
>> cd /mnt/ext4
>> tar zxvf linux-3.0.tar.gz
>> umount /mnt/ext4
> 
> I've been traveling, so I haven't had a chance to test this, but it makes no sense that changing the encoding fro the extent length would change the performance of the forced writeback caused by amount.   There may be a performance bug that we should fix, or may have been fixed by accident with the extent encoding change. 
> 
> Have you investigated why this got better when you changed the meaning of the extent length field?   It makes no sense that such a format change would have such an impact….
OK, so let me explain why the big cluster length works.

In the new bigalloc case if chunk size=64k, and with the linux-3.0
source, every file will be allocated a chunk, but they aren't contiguous
if we only write the 1st 4k bytes. In this case, writeback and the block
layer below can't merge all the requests sent by ext4. And in our test
case, the total io will be around 20000. While with the cluster size, we
have to zero the whole cluster. From the upper point of view. we have to
write more bytes. But from the block layer, the write is contiguous and
it can merge them to be a big one. In our test, it will only do around
2000 ios. So it helps the test case.

Thanks
Tao
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html