linux-ext4 - Re: bigalloc and max file size

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20111031200053.GI16825@thunk.org>
Date:	Mon, 31 Oct 2011 16:00:53 -0400
From:	Ted Ts'o <tytso@....edu>
To:	Tao Ma <tm@....ma>
Cc:	Andreas Dilger <adilger@...mcloud.com>,
	linux-ext4 development <linux-ext4@...r.kernel.org>,
	Alex Zhuravlev <bzzz@...mcloud.com>,
	"hao.bigrat@...il.com" <hao.bigrat@...il.com>
Subject: Re: bigalloc and max file size

On Mon, Oct 31, 2011 at 06:27:25PM +0800, Tao Ma wrote:
> In the new bigalloc case if chunk size=64k, and with the linux-3.0
> source, every file will be allocated a chunk, but they aren't contiguous
> if we only write the 1st 4k bytes. In this case, writeback and the block
> layer below can't merge all the requests sent by ext4. And in our test
> case, the total io will be around 20000. While with the cluster size, we
> have to zero the whole cluster. From the upper point of view. we have to
> write more bytes. But from the block layer, the write is contiguous and
> it can merge them to be a big one. In our test, it will only do around
> 2000 ios. So it helps the test case.

This is test case then where there are lot of sub-64k files, and so
the system administrator would be ill-advised to use a 64k bigalloc
cluster size in the first place.  So don't really consider that a
strong argument; in fact, if the block device is a SSD or a
thin-provisioned device with an allocation size smaller than the
cluster size, the behaviour you describe would in fact be detrimental,
not a benefit.

In the case of a hard drive where seeks are expensive relative to
small writes, this is something which we could do (zero out the whole
cluster) with the current bigalloc file system format.  I could
imagine trying to turn this on automatically with a hueristic, but
since we can't know the underlying allocation size of a
thin-provisioned block device, that would be tricky at best...

Regards,

		       	       	    	     - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html