linux-ext4 - Re: e2fsck extremly slow after: EXT4-fs.. ext4_check_descriptors: Checksum for group .. failed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20121113212400.GA13850@thunk.org>
Date:	Tue, 13 Nov 2012 16:24:00 -0500
From:	tytso@...nk.org (Theodore Ts'o)
To:	Andreas Dilger <adilger@...ger.ca>
Cc:	"kaefert@...il.com" <kaefert@...il.com>, linux-ext4@...r.kernel.org
Subject: Re: e2fsck extremly slow after: EXT4-fs.. ext4_check_descriptors:
 Checksum for group .. failed

To follow up on the list since Thomas and I have had a number of
e-mail exchanges that were off-list, and he has sent me an compressed,
raw e2image dump of his file system which I have investigated

The proximate cause of the fs corruption seems to be a few inode table
blocks written offset by a 1024 bytes --- there were 3 pairs of inodes
of the form (N, N+4) which had the exact same contents in the inode
structure (same generation number, same mtime/ctime/atimes, same
extents).  This pattern of corruption is quite odd given that the file
system has a 4k block size.  The best bet is that the corruption
happened at the USB device layer, since the mis-written inodes were
offset by a 2 512 byte sectors, as opposed to by an incorrect block
number.  Thomas tells me this particular device has had a flaky USB
controller and this is the not the first such failure.

There also seems to be a bug in e2fsck which caused it not to be able
to repair the corrupted file system.  I have not had a chance to track
down the bug yet.  It may have been caused by how we handle extent
tree blocks getting cached while trying to clone the data block.
Something which we should fix, but ultimately, the use of metadata
checksums is going to be the best way to deal with cases of the inode
table block getting written to the wrong place on disk, since we will
then know which inode not to trust, and just have e2fsck zap it.

Speaking of zapping, I've given Thomas instructions on how to clri
three of the duplicated inodes using debugfs, and that allowed e2fsck
to be able to repair his file system.  He will have suffered some data
loss due to the corrupted inode table, but at least this way he'll be
able to gain access to most of the files on the disk.

     	     	       	       	   - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html