Date:	Thu, 26 Sep 2013 22:18:33 +0200
From:	Jan Kara <jack@...e.cz>
To:	InvTraySts <invtrasys@...il.com>
Cc:	Jan Kara <jack@...e.cz>, Eric Sandeen <sandeen@...hat.com>,
	linux-ext4@...r.kernel.org, linux-fsdevel@...r.kernel.org
Subject: Re: Fwd: Need help with Data Recovery on Ext4 partitions that became
 corrupted on running OS

On Thu 26-09-13 13:45:34, InvTraySts wrote:
> root@...ver:~# debugfs /dev/sdf1
> debugfs 1.42.5 (29-Jul-2012)
> debugfs:  stat /
> stat: A block group is missing an inode table while reading inode 2
> debugfs:  dump /etc/
> dump: Usage: dump_inode [-p] <file> <output_file>
> debugfs:  dump /etc/passwd /root/passwd.recovery
> /etc/passwd: A block group is missing an inode table
> debugfs:  ls /etc/
> /etc/: A block group is missing an inode table
> 
> So now I am getting a different error message, even though literally
> nothing has changed other than the time I ran the command. I figured
> the passwd file would be the easiest thing to recover, since I don't
> remember the exact paths for other application configurations without
> doing some digging for the paths and such.
> 
> At one point, before the mailing list and me joining the #ext channel,
> I was able to see that there was a lost and found folder, I just
> couldn't navigate it. So, I am not sure why that has changed either. I
> assume logical corruption to the extent it has could cause almost
> anything.
  Isn't the fs already modified from fsck? Anyway, the fs really looks
hopelessly damaged so I'm afraid we won't be able to get any data from it.

								Honza

> On Thu, Sep 26, 2013 at 4:53 AM, Jan Kara <jack@...e.cz> wrote:
> > On Wed 25-09-13 20:08:38, InvTraySts wrote:
> >> (Going to merge these two back, because both of you are actually
> >> helping me, but with the two conversations being segregated like this
> >> it makes it hard to correlate with you both)
> >> The partprobe did work with getting the partition table reread.
> >> After that, the tune2fs sorta worked.
> >> root@...ver:~# tune2fs -f -O ^has_journal /dev/sdf1
> >> tune2fs 1.42.5 (29-Jul-2012)
> >> root@...ver:~# mount /dev/sdf1 /media/tmp
> >> root@...ver:~# ls -l /media/tmp/
> >> total 0
> >>
> >> When I try and use the debugfs /dev/sdf1
> >>
> >> root@...ver:~# debugfs /dev/sdf1
> >> debugfs 1.42.5 (29-Jul-2012)
> >> debugfs:  ls
> >> EXT2 directory corrupted
> >> debugfs:  ls /
> >> /: EXT2 directory corrupted
> >>
> >> The drive is supposed to be ext4.
> >   'EXT2' in the message doesn't mean anything. It is always there. But we
> > see that even root directory is corrupted. That's bad. You can try what
> > 'stat /' says in debugfs? Possibly also run 'dump / <somefile>' to dump raw
> > directory data to <somefile> and attach the file here. But also given the
> > output of e2fsck I'm afraid there won't be much to save on the drive...
> >
> >                                                                 Honza
> >
> >> root@...ver:~# dumpe2fs /dev/sdf1
> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> Filesystem volume name:   root
> >> Last mounted on:          /media/ubuntu/root
> >> Filesystem UUID:         removed
> >> Filesystem magic number:  0xEF53
> >> Filesystem revision #:    1 (dynamic)
> >> Filesystem features:      ext_attr resize_inode dir_index filetype
> >> extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
> >> extra_isize
> >> Filesystem flags:         signed_directory_hash
> >> Default mount options:    user_xattr acl
> >> Filesystem state:         not clean with errors
> >> Errors behavior:          Continue
> >> Filesystem OS type:       Linux
> >> Inode count:              4849664
> >> Block count:              19398144
> >> Reserved block count:     969907
> >> Free blocks:              17066988
> >> Free inodes:              4592929
> >> First block:              0
> >> Block size:               4096
> >> Fragment size:            4096
> >> Reserved GDT blocks:      1019
> >> Blocks per group:         32768
> >> Fragments per group:      32768
> >> Inodes per group:         8192
> >> Inode blocks per group:   512
> >> Flex block group size:    16
> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> Last mount time:          Wed Sep 25 19:59:07 2013
> >> Last write time:          Wed Sep 25 19:59:51 2013
> >> Mount count:              1
> >> Maximum mount count:      -1
> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> Check interval:           0 (<none>)
> >> Lifetime writes:          107 GB
> >> Reserved blocks uid:      0 (user root)
> >> Reserved blocks gid:      0 (group root)
> >> First inode:              11
> >> Inode size:               256
> >> Required extra isize:     28
> >> Desired extra isize:      28
> >> Default directory hash:   half_md4
> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> Journal backup:           inode blocks
> >> FS Error count:           9
> >> First error time:         Sat Aug 24 13:44:55 2013
> >> First error function:     ext4_iget
> >> First error line #:       3889
> >> First error inode #:      8
> >> First error block #:      0
> >> Last error time:          Wed Sep 25 19:59:12 2013
> >> Last error function:      htree_dirblock_to_tree
> >> Last error line #:        892
> >> Last error inode #:       2
> >> Last error block #:       20037
> >>
> >> [...]
> >> [output scrolls very fast, and is very large]
> >> Group 10: (Blocks 327680-360447) [ITABLE_ZEROED]
> >>   Checksum 0xecaa, unused inodes 0
> >>   Block bitmap at 1035 (bg #0 + 1035), Inode bitmap at 1051 (bg #0 + 1051)
> >>   Inode table at 6177-6688 (bg #0 + 6177)
> >>   0 free blocks, 16 free inodes, 0 directories
> >>   Free blocks: 327680, 327683-327685, 327687, 327689-327690,
> >> 327692-327693, 327695-327697, 327700-327701, 327703, 327705,
> >> 327707-327709, 327711-327715, 327718-327727, 327730-327808,
> >> 327810-327823, 327825-327842, 327846-327855, 327857-327875, 327878,
> >> 327881
> >>
> >> (the total amount of pages that the dumpe2fs comes out with is about
> >> 240 pages. something that can't be pasted. If attachments would be
> >> required, I can attach a file with it).
> >>
> >> On Wed, Sep 25, 2013 at 5:28 PM, Jan Kara <jack@...e.cz> wrote:
> >> > On Wed 25-09-13 15:24:34, InvTraySts wrote:
> >> >> And am cloning the drive without the sync parameter this time.
> >> >> root@...ver:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
> >> >> After it finished, I attempted to run dumpe2fs and it still responds with:
> >> >> root@...ver:~# dumpe2fs /dev/sdf1
> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
> >> >> Couldn't find valid filesystem superblock.
> >> >   Well, that's likely because the partition table on /dev/sdf didn't get
> >> > reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
> >> > partition table.
> >> >
> >> >                                                                 Honza
> >> >
> >> >> So I went ahead and tried to run the tune2fs command:
> >> >> root@...ver:~# tune2fs -f -O ^has_journal /dev/sda1
> >> >> tune2fs 1.42.5 (29-Jul-2012)
> >> >> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
> >> >> Couldn't find valid filesystem superblock.
> >> >>
> >> >> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
> >> >>
> >> >>
> >> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@...e.cz> wrote:
> >> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> >> So long story short, I had a server running that had a processor fail
> >> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> >> recommendation he suggested me to post something to the mailing list.
> >> >> >>
> >> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> >> card.
> >> >> >> If I try to mount this using:
> >> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >> >>
> >> >> >> It complains in dmesg with the following output:
> >> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >> >>
> >> >> >>
> >> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> >> root@...ver:~# dumpe2fs -f /dev/sda1
> >> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> >> Filesystem volume name:   root
> >> >> >> Last mounted on:          /media/ubuntu/root
> >> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> >> Filesystem magic number:  0xEF53
> >> >> >> Filesystem revision #:    1 (dynamic)
> >> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> >> dir_nlink extra_isize
> >> >> >> Filesystem flags:         signed_directory_hash
> >> >> >> Default mount options:    user_xattr acl
> >> >> >> Filesystem state:         not clean with errors
> >> >> >> Errors behavior:          Continue
> >> >> >> Filesystem OS type:       Linux
> >> >> >> Inode count:              4849664
> >> >> >> Block count:              19398144
> >> >> >> Reserved block count:     969907
> >> >> >> Free blocks:              17034219
> >> >> >> Free inodes:              4592929
> >> >> >> First block:              0
> >> >> >> Block size:               4096
> >> >> >> Fragment size:            4096
> >> >> >> Reserved GDT blocks:      1019
> >> >> >> Blocks per group:         32768
> >> >> >> Fragments per group:      32768
> >> >> >> Inodes per group:         8192
> >> >> >> Inode blocks per group:   512
> >> >> >> Flex block group size:    16
> >> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> >> Mount count:              0
> >> >> >> Maximum mount count:      -1
> >> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> >> Check interval:           0 (<none>)
> >> >> >> Lifetime writes:          107 GB
> >> >> >> Reserved blocks uid:      0 (user root)
> >> >> >> Reserved blocks gid:      0 (group root)
> >> >> >> First inode:              11
> >> >> >> Inode size:               256
> >> >> >> Required extra isize:     28
> >> >> >> Desired extra isize:      28
> >> >> >> Journal inode:            8
> >> >> >> Default directory hash:   half_md4
> >> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> >> Journal backup:           inode blocks
> >> >> >> FS Error count:           8
> >> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> >> First error function:     ext4_iget
> >> >> >> First error line #:       3889
> >> >> >> First error inode #:      8
> >> >> >> First error block #:      0
> >> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> >> Last error function:      ext4_iget
> >> >> >> Last error line #:        3888
> >> >> >> Last error inode #:       8
> >> >> >> Last error block #:       0
> >> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> >> > OK.
> >> >> >
> >> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> >> and currently I am having more problems with the cloned drive than I
> >> >> >> am with the original.
> >> >> >>
> >> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> >> has_journal flag, but I might need some assistance with that.
> >> >> >   Yes, you can do that with:
> >> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >> >
> >> >> >   Let's see what mount will say after that.
> >> >> >
> >> >> >   Another option is to run
> >> >> > debugfs /dev/sda1
> >> >> >
> >> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> >> > chance of getting at least some data back.
> >> >> >
> >> >> >                                                                 Honza
> >> >> > --
> >> >> > Jan Kara <jack@...e.cz>
> >> >> > SUSE Labs, CR
> >> >>
> >> >>
> >> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@...e.cz> wrote:
> >> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> >> So long story short, I had a server running that had a processor fail
> >> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> >> recommendation he suggested me to post something to the mailing list.
> >> >> >>
> >> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> >> card.
> >> >> >> If I try to mount this using:
> >> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >> >>
> >> >> >> It complains in dmesg with the following output:
> >> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >> >>
> >> >> >>
> >> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> >> root@...ver:~# dumpe2fs -f /dev/sda1
> >> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> >> Filesystem volume name:   root
> >> >> >> Last mounted on:          /media/ubuntu/root
> >> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> >> Filesystem magic number:  0xEF53
> >> >> >> Filesystem revision #:    1 (dynamic)
> >> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> >> dir_nlink extra_isize
> >> >> >> Filesystem flags:         signed_directory_hash
> >> >> >> Default mount options:    user_xattr acl
> >> >> >> Filesystem state:         not clean with errors
> >> >> >> Errors behavior:          Continue
> >> >> >> Filesystem OS type:       Linux
> >> >> >> Inode count:              4849664
> >> >> >> Block count:              19398144
> >> >> >> Reserved block count:     969907
> >> >> >> Free blocks:              17034219
> >> >> >> Free inodes:              4592929
> >> >> >> First block:              0
> >> >> >> Block size:               4096
> >> >> >> Fragment size:            4096
> >> >> >> Reserved GDT blocks:      1019
> >> >> >> Blocks per group:         32768
> >> >> >> Fragments per group:      32768
> >> >> >> Inodes per group:         8192
> >> >> >> Inode blocks per group:   512
> >> >> >> Flex block group size:    16
> >> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> >> Mount count:              0
> >> >> >> Maximum mount count:      -1
> >> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> >> Check interval:           0 (<none>)
> >> >> >> Lifetime writes:          107 GB
> >> >> >> Reserved blocks uid:      0 (user root)
> >> >> >> Reserved blocks gid:      0 (group root)
> >> >> >> First inode:              11
> >> >> >> Inode size:               256
> >> >> >> Required extra isize:     28
> >> >> >> Desired extra isize:      28
> >> >> >> Journal inode:            8
> >> >> >> Default directory hash:   half_md4
> >> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> >> Journal backup:           inode blocks
> >> >> >> FS Error count:           8
> >> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> >> First error function:     ext4_iget
> >> >> >> First error line #:       3889
> >> >> >> First error inode #:      8
> >> >> >> First error block #:      0
> >> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> >> Last error function:      ext4_iget
> >> >> >> Last error line #:        3888
> >> >> >> Last error inode #:       8
> >> >> >> Last error block #:       0
> >> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> >> > OK.
> >> >> >
> >> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> >> and currently I am having more problems with the cloned drive than I
> >> >> >> am with the original.
> >> >> >>
> >> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> >> has_journal flag, but I might need some assistance with that.
> >> >> >   Yes, you can do that with:
> >> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >> >
> >> >> >   Let's see what mount will say after that.
> >> >> >
> >> >> >   Another option is to run
> >> >> > debugfs /dev/sda1
> >> >> >
> >> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> >> > chance of getting at least some data back.
> >> >> >
> >> >> >                                                                 Honza
> >> >> > --
> >> >> > Jan Kara <jack@...e.cz>
> >> >> > SUSE Labs, CR
> >> > --
> >> > Jan Kara <jack@...e.cz>
> >> > SUSE Labs, CR
> > --
> > Jan Kara <jack@...e.cz>
> > SUSE Labs, CR
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists