linux-ext4 - RE: 2nd Attempt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:	Wed, 8 May 2013 15:45:23 +0100
From:	"Stephen Elliott" <techweb@...world.com>
To:	"'Theodore Ts'o'" <tytso@....edu>
Cc:	"'Andreas Dilger'" <adilger@...ger.ca>,
	<linux-ext4@...r.kernel.org>
Subject: RE: 2nd Attempt - FSCK Errors

Hi,

Well... I have some new and interesting information. 

After deleting the file, and re-creating from backup, the file system got
mounted read only and the following errors were logged:

May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904376:freeing already freed block
(bit 1144)
May  8 14:58:15 despair kernel: Aborting journal on device dm-0-8.
May  8 14:58:15 despair kernel: EXT4-fs (dm-0): Remounting filesystem
read-only
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904377:freeing already freed block
(bit 1145)
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904378:freeing already freed block
(bit 1146)
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904379:freeing already freed block
(bit 1147)
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904380:freeing already freed block
(bit 1148)
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904381:freeing already freed block
(bit 1149)
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904382:freeing already freed block
(bit 1150)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904383:freeing already freed block
(bit 1151)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904384:freeing already freed block
(bit 1152)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904385:freeing already freed block
(bit 1153)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904386:freeing already freed block
(bit 1154)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949296:freeing already freed block
(bit 13296)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949297:freeing already freed block
(bit 13297)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949298:freeing already freed block
(bit 13298)
May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949299:freeing already freed block
(bit 13299)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949300:freeing already freed block
(bit 13300)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949301:freeing already freed block
(bit 13301)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949302:freeing already freed block
(bit 13302)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949303:freeing already freed block
(bit 13303)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949304:freeing already freed block
(bit 13304)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949305:freeing already freed block
(bit 13305)
May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5125block 167949306:freeing already freed block
(bit 13306)

These are the same blocks slated as multiply claimed :)

 And then running an FSCK, we got the following:

***** File system check forced at Wed May  8 15:16:50 WEST 2013 ***** fsck
1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012)
/dev/c/c: recovering journal
Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory
structure Pass 3: Checking directory connectivity Pass 4: Checking reference
counts Pass 5: Checking group summary information Free blocks count wrong
for group #5124 (28170, counted=28159).
Fix? yes

Free blocks count wrong for group #5125 (25861, counted=25850).
Fix? yes

Free blocks count wrong (420683133, counted=420644972).
Fix? yes

Free inodes count wrong (29595347, counted=29595271).
Fix? yes


/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 616825/30212096 files (13.6% non-contiguous), 62748564/483393536
blocks

Volume seems clean now after reboot and keeps coming back clean.

Any ideas based on this? The old Database was over 1.6 GB in size. The
re-created one from backup is much smaller 33MB (as it was compacted before
restoring). Once these DB files grow they don't get smaller until compacted.
Although I'm sure file size should not have a bearing on this.

Many Thanks
Stephen Elliott


-----Original Message-----
From: Theodore Ts'o [mailto:tytso@....edu] 
Sent: 03 May 2013 22:32
To: Stephen Elliott
Cc: 'Andreas Dilger'; linux-ext4@...r.kernel.org
Subject: Re: 2nd Attempt - FSCK Errors

On Fri, May 03, 2013 at 07:42:55PM +0100, Stephen Elliott wrote:
> One thing maybe you could explain (and Andreas gave me his take too) 
> is how you can multiply assigned blocks shared with "0" files. Andreas 
> offered the suggestion that they may be in the same file. If this were 
> really the case, I would suspect there to be some file corruption issues
etc...

It could be that the blocks appear multiple times in the file; or it could
be the storage system is returning different data on subsequent reads from
the device.

What we need is a debugfs dump of the inode on the file system that is
having trouble.  A debugfs dump of the inode from a completely underlated
file system is not useful....

						- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html