lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20130313190016.GE5604@thunk.org>
Date:	Wed, 13 Mar 2013 15:00:16 -0400
From:	Theodore Ts'o <tytso@....edu>
To:	Andrey Sidorov <qrxd43@...orola.com>
Cc:	linux-ext4@...r.kernel.org
Subject: Re: mke2fs with bigalloc is too slow

On Wed, Mar 13, 2013 at 08:04:57PM +0400, Andrey Sidorov wrote:
> 
> It takes 29 seconds to format 1TB partition as ext4 with 256k cluster
> size on MIPS. e2fsprogs version is 1.42.7.
> The most time consumer is ext2fs_convert_subcluster_bitmap which folds
> 30M into 500K walking through block bitmap bit by bit. Afair, it takes
> less time on x86 since there are asm bit ops for x86, but mips uses
> generic ones.

Actually, these days we use a rbtree to encode the bitmap.  That means
that it should be possible to create a very efficient find_first_set
and find_first_zero functions.  This would work especially well for
mke2fs since the allocation bitmap will be mostly empty.  This I think
would improve things dramatically.

> First thing I did is allocated per-cluster bitmap instead of per-block
> bitmap in ext2fs_initialize so that conversion doesn't occur. That
> dropped mke2fs time from 29s to 2.5s. e2fsck -f found this fresh fs as
> a good one and mounting/writing/reading/unmounting also went good. Of
> course groups are allocated at different offsets and about 60M of
> usable space are lost if compared to 'slow' formatting. That's fine,
> we can live with that.
> Are there any long-term consequences that I've missed? Are there any
> reasons for using block bitmap instead of cluster bitmap except for
> meta-data space efficiency?

Fragmenting the block allocation bitmaps slows down operations that
need to read in multiple block bitmaps in sequence.  This includes
dumpe2fs, e2fsck, and in some cases, block allocation.  So making this
change is not zero-cost.  I agree that 30 seconds for mke2fs is
non-optimal, but I'm surprised you're finding this to be a
showstopper.  I assume you're worried about substantially bigger file
systems than just 1TB?

					- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists