linux-ext4 - Re: large file system & high object count testing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-id: <20090831201932.GD4197@webber.adilger.int>
Date:	Mon, 31 Aug 2009 14:19:32 -0600
From:	Andreas Dilger <adilger@....com>
To:	Ric Wheeler <rwheeler@...hat.com>
Cc:	linux-ext4@...r.kernel.org, "Ted Ts'o" <tytso@...nk.org>
Subject: Re: large file system & high object count testing

On Aug 31, 2009  12:34 -0400, Ric Wheeler wrote:
> We have put together a very large, relatively slow JBOD to test  
> scalability with (big server, 40GB of DRAM, 8 CPU's + 4 SAS expansion  
> shelves, each with 16 2TB WD S-ATA drives).
>
> In all, this is pulled together with DM (striped) to give us a bit over  
> 116TB.
>
> Testing was done on 2.6.31-rc6 along with the pu branches e2fsprogs.
>
> Everything went well until after the fsck - I think that I have  
> reproduced that earlier issue with a failed mount.
>
> mkfs took a very long time - longer than fsck. fsck (with around 500  
> million 20KB files) finished in just under 2 hours.

Fixing the kernel to do the "safe zeroing of inode table blocks" would
allow mke2fs to be MUCH faster than it is today...

> real    230m6.362s
> user    2m30.844s
> sys    200m1.002s

Ouch, 4h is a long time, but hopefully not many people have to reformat
their 120TB filesystem on a regular basis.

> [root@...adeth e2fsck]# time ./e2fsck -f -tt /dev/vg_wdc_disks/lv_wdc_disks
> e2fsck 1.41.8 (20-Jul-2009)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 1: Memory used: 1280k/18014398508273796k (1130k/151k), time:  
> 4630.05/780.40/3580.01

Sigh, we need better memory accounting in e2fsck.  Rather than depending
on the VM/glibc to track that for us, how hard would it be to just add
a counter into e2fsck_{get,free,resize}_mem() to track this?

> REMOUNT:
>
> [root@...adeth e2fsck]# mount  /dev/vg_wdc_disks/lv_wdc_disks /test_fs/
> mount: wrong fs type, bad option, bad superblock on  
> /dev/mapper/vg_wdc_disks-lv_wdc_disks,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
>
> [root@...adeth ~]# tail -20 /var/log/messages
> <snip>
> Aug 31 12:27:12 megadeth kernel: EXT4-fs (dm-75):  
> ext4_check_descriptors: Checksum for group 487 failed (59799!=46827)
> Aug 31 12:27:12 megadeth kernel: EXT4-fs (dm-75): group descriptors  
> corrupted!

Hmm, is e2fsck computing the 64-byte group descriptor checksum differently
than the kernel?  Can we dump the group descriptors before and after the
e2fsck run to see whether they have been modified without any messages to
the console?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html