linux-ext4 - Re: ext2/3 create large filesystem takes too much time; solutions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20060917075712.GJ6441@schatzie.adilger.int>
Date:	Sun, 17 Sep 2006 01:57:12 -0600
From:	Andreas Dilger <adilger@...sterfs.com>
To:	Theodore Tso <tytso@....edu>
Cc:	Pavel Mironchik <tibor0@...il.com>, linux-ext4@...r.kernel.org
Subject: Re: ext2/3 create large filesystem takes too much time; solutions

On Sep 16, 2006  16:06 -0400, Theodore Tso wrote:
> On Sat, Sep 16, 2006 at 05:56:02PM +0300, Pavel Mironchik wrote:
> > I agree with you and I would prefer to send something more
> > serious on that list than those previous patches - I like your
> > idea with counters. Btw I assume crc is more preferable than
> > just control sum for block group descriptors....
> 
> Yes, when I said checksum I meant a cyclic redundancy checksum, and
> not an additive checksum...  (and one of the things we can do is to
> build in the superblock UUID into the CRC, so that if the filesystem
> gets recreated we can distinguish an old inode from a new one).

Just to avoid duplication of effort, I'm attaching the current
work-in-progress patches for the uninitialized groups (kernel + e2fsprogs).
They are really at the "barely compile" stage (if that), but at least
people can look at them and start improving them instead of starting from
scratch.  The patches are based on work done by Anshu Goel
<anshu@...syssoft.com>, but have been reworked a fair amount since they
were given to me (i.e. bugs added are mine).  I've been sitting on them
for too long and they should see the light of day instead of continuing
to stagnate.

I also just incorporated Ted's suggestion to include the filesystem UUID
into the checksum.  I previously had added in the group number, so that
if the block is written out to the wrong location it wouldn't verify
correctly.

Things that need to be done:
- the kernel block/inode allocation needs to be reworked:
  - initialize a whole block worth of inodes at one time instead
    of single inodes.
  - I don't think we need to zero out the unused inodes - the kernel
    should already be doing this if the inode block is unused
  - find a happy medium between using existing groups (inodes/blocks)
    and initializing new ones
- we likely need to verify the checksum in more places in e2fsck before
  trusting the UNINIT flags

I won't be able to work more on this for a while, so have at it :-).

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

View attachment "ext3-uninit_groups-2.6.12.patch" of type "text/plain" (19918 bytes)

View attachment "e2fsprogs-uninit.patch" of type "text/plain" (29587 bytes)