Message-ID: <20150520225033.GA7368@cloud>
Date:	Wed, 20 May 2015 15:50:33 -0700
From:	josh@...htriplett.org
To:	Theodore Ts'o <tytso@....edu>, Lukas Czerner <lczerner@...hat.com>,
	linux-kernel@...r.kernel.org
Subject: Re: Nature of ext4 corruption fixed by recent patch?

On Tue, May 19, 2015 at 01:50:24PM -0400, Theodore Ts'o wrote:
> On Tue, May 19, 2015 at 09:37:40AM -0700, Josh Triplett wrote:
> > In particular, I didn't realize this was *only* the data of the
> > delayed-extent-based files.  The bug here seems to have struck various
> > recently-written files and directories.  (Recent in days, not seconds,
> > as far as I can tell; and it isn't universal based on age.) The initial
> > symptom was ext4 noticing that a directory was corrupt (truncated, IIRC)
> > and immediately marking the whole filesystem read-only.
> 
> Do you have the transcript of fsck run on the file system?  Either
> with -n, or as you were trying to fix it?  I'd need to know a lot more
> about the pattern of corruptions to hazard a guess.

Well, I *was* going to say that I didn't have the logs or transcripts of
fsck because the filesystem got remounted read-only and I didn't log
fsck.  However, it looks like *after* the fsck, the problem occurred
again:

[173581.359925] EXT4-fs error (device md0): ext4_ext_remove_space:2976: inode #8395881: comm rm: pblk 33595732 bad header/extent: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
[173581.360054] Aborting journal on device md0-8.
[173581.360100] EXT4-fs (md0): Remounting filesystem read-only
[173581.360117] EXT4-fs error (device md0) in ext4_ext_remove_space:3048: IO failure
[173581.360189] EXT4-fs error (device md0) in ext4_ext_truncate:4669: IO failure
[173581.360262] EXT4-fs error (device md0) in ext4_reserve_inode_write:4837: Journal has aborted
[173581.360337] EXT4-fs error (device md0) in ext4_truncate:3668: Journal has aborted
[173581.360411] EXT4-fs error (device md0) in ext4_reserve_inode_write:4837: Journal has aborted
[173581.360485] EXT4-fs error (device md0) in ext4_orphan_del:2694: Journal has aborted
[173581.360559] EXT4-fs error (device md0) in ext4_reserve_inode_write:4837: Journal has aborted

And since this is after the fsck, and I'm assuming fsck would have
noticed that kind of corruption, then this would have to be new
filesystem corruption.

Here's the result of "fsck -n" from the still-running system, since the
filesystem is read-only anyway:

fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/md0 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 2097220 has zero dtime.  Fix? no

Inodes that were part of a corrupted orphan linked list found.  Fix? no

Inode 4725949 was part of the orphaned inode list.  IGNORED.
Inode 6172273 has an invalid extent node (blk 24679449, lblk 0)
Clear? no

Inode 6172273, i_blocks is 152, should be 0.  Fix? no

Inode 8395881 was part of the orphaned inode list.  IGNORED.
Inode 9970463 has an invalid extent node (blk 39892628, lblk 0)
Clear? no

Inode 9970463, i_blocks is 21072, should be 0.  Fix? no

Inode 18488219 has an invalid extent node (blk 73989773, lblk 0)
Clear? no

Inode 18488219, i_blocks is 2459648, should be 2185832.  Fix? no

Pass 2: Checking directory structure
Directory inode 4470367, block #0, offset 0: directory corrupted
Salvage? no

e2fsck: aborted

/dev/md0: ********** WARNING: Filesystem still has errors **********


These look roughly similar to those that came up during the previous
issue, though at that point there were far more of them.  Other messages
in the previous round of fsck not shown above include the usual repair
procedures (adding in . and .., correcting link counts); it's possible
that there were other messages as well, but I don't recall which ones
and I don't have a transcript.

Is there some additional data I can collect to help determine what the
issue might be here?  I can leave this system running for a bit in this
read-only state, before I start trying to recover it.

> The sorts of corruption that turn into a large number of file system
> errors are (a) corruptions in the block allocation bitmap, so blockes
> get used for more than one purpose, or (b) garbage (or the wrong
> portion of an inode table) getting written into the inode table.  But
> these all have their own distinctive signatures in terms of the file
> system problems reported by e2fsck.
> 
> In general though this doesn't cause large number of files to contain
> NULLs. though.  So it doesn't smell like a file system problem, but
> I'd want to see a detailed listing of the problems reported by e2fsck
> before making a definitive statement.
> 
> Were you using LVM, raid, or anything else between the file system and
> the storage device(s)?

md-based RAID0, on top of a pair of SSDs.

- Josh Triplett
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists