linux-kernel - Re: Half-Disaster switching from ext3 to ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Pine.LNX.4.64.0901052023260.11020@death.krwtech.com>
Date:	Mon, 5 Jan 2009 21:11:27 -0500 (EST)
From:	Ken Witherow <ken@...tech.com>
To:	Theodore Tso <tytso@....edu>
cc:	Giangiacomo Mariotti <gg.mariotti@...il.com>,
	linux-kernel@...r.kernel.org
Subject: Re: Half-Disaster switching from ext3 to ext4

On Mon, 5 Jan 2009, Theodore Tso wrote:

> On Sun, Jan 04, 2009 at 04:23:42AM +0100, Giangiacomo Mariotti wrote:
>> the partitions(except for root obviously), I synced some times before
>> doing that and then I shut down the pc with a "shutdown -h -P now".
>> when I booted again the system, /dev/sda2(the partition with the most
>> important data. Murphy's law really works!) failed to get mounted with
>> a message saying something about corrupted groups(maybe, I don't
>> remember exactly). Trying to do a fsck.ext4 -p /dev/sda2, I got
>> something like : fsck.ext4 : Invalid argument while checking ext3
>> journal for /dev/sda2.
>
> What version of e2fsprogs do you have installed, and can you dig up
> the exact error message at mount time?  It should be in the your
> system logs....
>
>> remounted it, all files were gone! I was starting to use the sleuth
>> kit, when I noticed that in the search+found directory there were
>> almost all the files, but not all of them and in a rearrenged
>> directory structure.
>
> Sounds like the root inode got zapped.  As for why, it's hard to say.
> It could be a simple disk failure that happend as bad luck.  I haven't
> seen any other bug reports that report corruption similar to yours, so
> without a lot more information or a way to reproduce the error,
> there's not much more I can say.

I had something similar happen last week on an image I mount via a loop 
device (it's an image I use just for my gentoo portage directory so it 
doesn't thrash the rest of my filesystem). My nightly sync started (rsyncs 
the portage directory from gentoo) and a couple days later, I was getting 
filesystem errors that indicated corruption. Not thinking, after fsck 
moved a bunch of files to lost+found, I simply created a new filesystem 
in its place and it has been fine since. badblocks showed no problems on 
the drive and it passes a SMART long check.

The filesystem is also exported read-only via NFS. It was a freshly 
created ext4 filesystem, not a migrated ext3->ext4 filesystem.

Linux death 2.6.28-gentoo #1 SMP Thu Dec 25 22:56:59 EST 2008 x86_64 AMD 
Athlon(tm) 64 X2 Dual Core Processor 4400+ AuthenticAMD GNU/Linux

Gnu C                  4.3.2
Gnu make               3.81
binutils               2.18
util-linux             2.14.1
mount                  support
module-init-tools      3.4
e2fsprogs              1.41.3
Linux C Library        2.9
Dynamic linker (ldd)   2.9
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.13
Sh-utils               6.10
udev                   124
Modules Loaded         vboxdrv it87 hwmon_vid nvidia snd_hda_intel advansys joydev

I attached the relevant info from my log. You'll see lots of:
Jan  1 22:30:18 death EXT4-fs error (device loop0): ext4_mb_generate_buddy: EXT4-fs: group 2: 403 blocks in bitmap, 407 in gd
Jan  1 22:30:18 death
Jan  1 22:30:21 death EXT4-fs error (device loop0): mb_free_blocks: double-free of inode 0's block 58359(bit 25591 in group 1)

with mount attempts failing with:
Jan  3 20:46:11 death EXT4-fs: ext4_check_descriptors: Checksum for group 0 failed (59714!=40674)
Jan  3 20:46:11 death EXT4-fs: group descriptors corrupted!

View attachment "ext4.log" of type "TEXT/PLAIN" (53125 bytes)

View attachment "config-death-2.6.28-gentoo" of type "TEXT/PLAIN" (56420 bytes)