Date:	Sun, 10 May 2009 16:51:30 +0200
From:	"Marcel Partap" <mpartap@....net>
To:	Theodore Tso <tytso@....edu>
Cc:	tytso@...nk.org, linux-ext4@...r.kernel.org
Subject: Re: Fwd: fsck ate my ext4 home partition, help!?


> On Sat, May 09, 2009 at 04:56:55PM +0200, Marcel Partap wrote: Hi
> > Theodore, i have run into quite an unlucky situation, developing
> > within this thread http://markmail.org/message/5vnfqx7jfzy7dk4f and
> > no longer true to the title (actually, it was an ext3 volume and i
> > can't blame e2fsck..)
> 
> OK, so when people have this happen to them, what has happened is that
> somehow, blocks have gotten written to the wrong place on disk.  In
> some cases, such has here:
> 
> > -----------------------------------------------
> > debugfs:  pwd
> > [pwd]   INODE: 28409857  PATH: /homedirs
> > [root]  INODE:      2  PATH: /
> > debugfs:  stat currenthomebase
> > Inode: 27672580   Type: regular    Mode:  0644   Flags: 0x0
> > Generation: 3792779416    Version: 0x00000000
> > User:  1000   Group:   100   Size: 9715
> > File ACL: 0    Directory ACL: 0
> > Links: 1   Blockcount: 24
> > Fragment:  Address: 0    Number: 0    Size: 0
> > ctime: 0x48c67d84 -- Tue Sep  9 15:43:32 2008
> > atime: 0x48c67d84 -- Tue Sep  9 15:43:32 2008
> > mtime: 0x443915e6 -- Sun Apr  9 16:10:46 2006
> > BLOCKS:
> > (0):3704694, (1-2):3704826-3704827
> > TOTAL: 3
> > -----------------------------------------------
> > SHOULD be a directory. 
> 
> Was probably the case of some inode table block gettin written to the
> wrong place on diks.
Maybe because of the half-unmounted state things were in.

> In the case of these error messages:
> 
> > Inode 65633 was part of the orphaned inode list.  IGNORED.
> > Inode 65633 is in use, but has dtime set.  Fix? no
> > 
> > Inode 65633 has imagic flag set.  Clear? no
> > 
> > Inode 65634 was part of the orphaned inode list.  IGNORED.
> > Inode 65634 is in use, but has dtime set.  Fix? no
> > 
> > Inode 65634 has imagic flag set.  Clear? no
> > 
> > Inode 65635 is in use, but has dtime set.  Fix? no
> > 
> > Inode 65636 is in use, but has dtime set.  Fix? no
> > 
> > Inode 65636 has imagic flag set.  Clear? no
> > 
> > Inode 65637 is in use, but has dtime set.  Fix? no
> > 
> > Inode 65638 was part of the orphaned inode list.  IGNORED.
> > Inode 65638 is in use, but has dtime set.  Fix? no
> 
> It's probably some data block getting written to the inode table.
> Many things can cause this problem.  Filesystem corruption which
> causes indirect blocks to point at the inode table can do this
> (although it doesn't explain an inode table block getting written to
> the wrong place on disk and overwriting another inode table block).
> It can be caused by a kernel bug.  It can also be caused by hardware
> problems; a hiccup in the controller or in the disk drive can cause a
> disk to be written to the wrong location on disk.  (The most likely
> cause of an inode table block getting written to the wrong location on
> disk.)
Even though smartd regularly warns the drive had two unrecoverable sector incidents, i highly doubt it's the hardware's fault. Strongly assume full scale user error instead.

> Hardware screwing up happens more often that hard drive manufacturers
> like to admit.  There's a reason why enterprise databases put
> checksums and block location information in each table block they
> write out, so they can catch this.  As another example, the SCSI Data
> Integrity Field[1] is used by hardware to provide an separate
> end-to-end check that the block will be written to the right location
> on disk.  Unfortunately, only the most expensive, high-end storage
> devices supports the DIF feature.
> [1] http://lwn.net/Articles/290145/
> 
Well a software RAID setup, redunant parity (PAR2) files and regular backups minimize the risk of data loss much more price worthy for us non-professionals.. but i haven't had it setup yet. Someone hit me with a large stick. Nah - BIGGER.

> In any case, it's not as simple as:
> > Can i just change the mode of the inode and run fsck? 
> The problem is that it's not just the matter of the mode on the inode
> being incorrect; the entire inode is probably from another location on
> disk.  Judging from the timestamp, it hasn't been modified in a while:
> 
> > ctime: 0x48c67d84 -- Tue Sep  9 15:43:32 2008
> > atime: 0x48c67d84 -- Tue Sep  9 15:43:32 2008
> > mtime: 0x443915e6 -- Sun Apr  9 16:10:46 2006
> 
> And if you were to use debugfs to dump out the contents of the file:
> 
> debugfs: dump <27672580> /tmp/currenthomebase-contents
> 
> And then run "file" over that file, you'll probably find that it is
> some text file or executable file -- but not a directory.
Yes it is - it's auth.php from phpBB, might belong to a local mirror of a site i maintain. Duh.

> The only way to recover this, I'm afraid, is to let fsck:
>
> > attaches all thousands of sub objects into lost+found, which is not
> > really usable (see fsck -vn /dev/sdd4 log attached).
Nooooh :O
Not giving up yet. Must find my home dir's inode somewhere!! Not transcending state of denial yet.

> You may find that this is much more useful than you think.  For
> directories that are dumped into lost+found, you can use ls to look at
> the contents (i.e., ls /mnt/lost+found/#12345) and then from that,
> reconstruct where it was supposed to go.
I know, i tried it. All (almost?) data ends up there, and as the files mostly resided in subdirectories, which get recovered. But the directory contained about 8K objects, and i'm not yet letting loose on that metadata.
> If you have a locatedb
> available (was this the root directory or some secondary filesystem?),
> you can sometimes use the locate command to figure out where directory
> is supposed to end up.  i.e.,
Yeah stupid me. I even pondered in time: duh, maybe i really should stop that slocate cron from recreating the database. At that time though, i really wasn't as close to acknowledging the loss of that directory as i am now. I didn't do it, so the locate db has already lost this information. Is there a chance the old one can be found on the partition it resides on? Gonna grep through the partition if there's no better way to find it..

> 
> % ls /mnt/lost+found/#12345
> certdata     krb5-smtp.c     Makefile.in       prot.o	util.o
> config.h     krb5-smtp.o     outgoing-465.txt  retry.c
> CVS	     krb5-smtp.save  outgoing-587.txt  util.c
> exitcodes.h  Makefile	     prot.c	       util.h
> krb5-smtp    Makefile~	     prot.h	       util.h~
> % locate krb5-smtp.c
> /usr/projects/krb5-smtp/imtest/krb5-smtp.c
> 
> Ah, hah!  So therefore #12345 is supposed to be
> /usr/local/krb5-smtp/imtest
> 
> For regular files dumped in /lost+found, you'll have to use the "file"
> command to see what they are, and then examine them by hand, and look
> at the file ownership, to figure out where they were supposed to go.
Yeah i kinda know how to proceed with that. As explained, because of the amount of files and also because of the actual low importance of them files, i'd really try and take any other measures to recover them filenames before doing that.
I know some of the filenames that were in that directory. Is there any more usable method to find these inodes? for example, i know there was a symlink drive_c in there, and a file 'snippetz'. Can't i use this to somehow locate some of the dangling meta information? For sure it can't be all overwritten... If i grep the partition image for that filename and look up any resulting blocks with debugfs, is that getting me anywhere nearer the list of files in that directory?

> No, it won't be easy, but at least you haven't lost the contents of
> the files!  You should be able to recover 90+% of your files, and for
> the rest hopefully you'll be able to recover them from your backups
> (you *WERE* doing regular backups, *RIGHT*?  :-)
Uhhmm yeah backups, RIIIGHT *g
Not of that directory though. Going to start doing that - as soon as i resolved this mess.

> There are other things you can do if you are regularly having this
> problem, including using e2image to take backups of your metadata
> (which can help you recover overwritten inode table blocks) --- this
> is *still* not a substitute for regular backups, but if this is
> happening a lot, especially with ext3, I'd take a very long, hard look
> at your hardware.  
> 
> This BTW, is one of the reasons why debugging the problem with ext4 is
> so frustrating.  By the time people report the symptoms, the damage
> was done long ago, and it might not have been the filesystem code's
> fault; it could just as easily be a bug in the md layer, or the a
> hardware problem.  However, it's getting reported *just* frequently
> enough, and by enough people, that we're worried it's being caused by
> an ext4 bug.  But we don't know that for sure.   
Well in this incident, this was a situation where i neglected the subtle signs of something (clean unmounting) going wrong, so: user error.

> If you're seeing this with ext3, and you're also using the md
> (software raid) driver, maybe there's something to the suspicion that
> it might be an md bug.  On the other hand, there are lots of people
> using the md layer, so I would have thought we would have gotten more
> bug reports from ext3 users of the md driver.  Or maybe it's just
> hardware bugs, and we're just seeing a statistical blip such that
> we've gotten 3 or 4 reports from ext4 users.  I really doubt that (as
> much as I would like to believe it), but it's possible.
Not using md and don't think any kernel bugs involved. Messing with half-dangling file systems is just never going to be a good idea.

> In any case, I suspect your best bet is to let e2fsck dump the results
> into lost+found, and then try to recover your files from there.  You
Denial. Disbelief. Shock. Angst. Horror. Rage. Iterating that randomly since incident occured one week ago..

> probably will have lost some files, since it looks like at least one
> or two other inode table blocks were also corrupted.  Hopefully though
> they are system files that can be reinstaled, and/or files for which
> you have backups.
Well the system is still up, the files are not critical to anything, life goes on. But state of denial continues, not giving up. Must try everything...

> Good luck,
Thx, and thx for your time Theo.
regards, marcel.
-- 
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss für nur 17,95 Euro/mtl.!* http://dslspecial.gmx.de/freedsl-surfflat/?ac=OM.AD.PD003K11308T4569a
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists