| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sat, 9 May 2009 13:14:56 -0400
From: "Don Bowman" <don@...dvine.com>
To: <linux-ext4@...r.kernel.org>
Subject: ext4 corruption on md [7x1TB in RAID5]
I cleanly shut down my system, and when it came back up, this was
reported:
* Checking file systems...
Sat May 9 13:00:53 2009
fsck 1.41.4 (27-Jan-2009)
fsck.ext4: Unable to resolve
'UUID=52e18bf7-cbe9-4f58-824c-5c943dd074de'^M
fsck died with exit status 8
For some reason it thinks it is ext2:
root@...ver:/var/log/fsck# fsck -n /dev/md0 | head -5
fsck 1.41.4 (27-Jan-2009)
e2fsck 1.41.4 (27-Jan-2009)
fsck.ext2: Group descriptors look bad... trying backup blocks...
Inode table for group 0 is not in group. (block 3251545658)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? No
Upon investigation, i'm not sure what to do. I really don't want to lose
the data
on this disk as it will take ages to rebuild (like weeks!).
The system has been in operation for about 2-3months. The data on the
disk
is mostly video + some virtual machines.
Any suggestions for repairing it? I have not done any repairs so far.
Should i just
Run fsck.ext4? Is there something i should try?
root@...ver:/var/log/fsck# uname -a
Linux server 2.6.28-12-server #43-Ubuntu SMP Fri May 1 20:22:39 UTC 2009
x86_64 GNU/Linux
root@...ver:/var/log/fsck# fsck.ext4 -n /dev/md0 | head -5
e2fsck 1.41.4 (27-Jan-2009)
fsck.ext4: Group descriptors look bad... trying backup blocks...
Inode table for group 0 is not in group. (block 3251545658)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? No
root@...ver:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Sun Mar 15 12:15:20 2009
Raid Level : raid5
Array Size : 5860574976 (5589.08 GiB 6001.23 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sat May 9 12:42:47 2009
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : c589dbe9:8dfff933:01f9e43d:ac30fbff (local to host
server)
Events : 0.305524
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 32 2 active sync /dev/sdc
3 8 64 3 active sync /dev/sde
4 8 80 4 active sync /dev/sdf
5 8 96 5 active sync /dev/sdg
6 8 112 6 active sync /dev/sdh
root@...ver:~#
root@...ver:/var/log/fsck# dumpe2fs /dev/md0 |head -60
dumpe2fs 1.41.4 (27-Jan-2009)
ext2fs_read_bb_inode: Invalid argument
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 52e18bf7-cbe9-4f58-824c-5c943dd074de
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: ext_attr resize_inode dir_index filetype
extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: not clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 366288896
Block count: 1465143744
Reserved block count: 73257187
Free blocks: 1041313557
Free inodes: 366151494
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 674
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Sun Mar 15 12:17:10 2009
Last mount time: Wed Apr 22 19:04:50 2009
Last write time: Sat May 9 12:31:43 2009
Mount count: 9
Maximum mount count: 29
Last checked: Sun Mar 15 12:17:10 2009
Check interval: 15552000 (6 months)
Next check after: Fri Sep 11 12:17:10 2009
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Default directory hash: half_md4
Directory Hash Seed: 49785c19-388f-444c-977a-886c147a8ae6
Journal backup: inode blocks
Group 0: (Blocks 0-32767)
Checksum 0x70e6, unused inodes 34168
Primary superblock at 0, Group descriptors at 1-350
Reserved GDT blocks at 351-1024
Block bitmap at 98318778 (+98318778), Inode bitmap at 837719297
(+837719297)
Inode table at 3251545658-3251546169 (+3251545658)
58662 free blocks, 33159 free inodes, 56553 directories, 34168 unused
inodes
Group 1: (Blocks 32768-65535) [INODE_UNINIT, ITABLE_ZEROED]
Checksum 0x6275, unused inodes 29973
Backup superblock at 32768, Group descriptors at 32769-33118
Reserved GDT blocks at 33119-33792
Block bitmap at 1123273429 (+1123240661), Inode bitmap at 3031071618
(+3031038850)
Inode table at 3149300410-3149300921 (+3149267642)
1881 free blocks, 46059 free inodes, 25281 directories, 29973 unused
inodes
Group 2: (Blocks 65536-98303) [INODE_UNINIT, ITABLE_ZEROED]
Checksum 0x1a0b, unused inodes 51590
Block bitmap at 1509085182 (+1509019646), Inode bitmap at 2884607537
(+2884542001)
Inode table at 1940584034-1940584545 (+1940518498)
3770 free blocks, 40683 free inodes, 43709 directories, 51590 unused
inodes
Group 3: (Blocks 98304-131071) [ITABLE_ZEROED]
Checksum 0xc3ec, unused inodes 47423
Backup superblock at 98304, Group descriptors at 98305-98654
Reserved GDT blocks at 98655-99328
Block bitmap at 383505502 (+383407198), Inode bitmap at 1784600367
(+1784502063)
Inode table at 798930664-798931175 (+798832360)
63118 free blocks, 7143 free inodes, 19773 directories, 47423 unused
inodes
root@...ver:/var/log/fsck# dmesg |egrep -i "sd|md|raid|fsck"
[ 0.000000] AMD AuthenticAMD
[ 0.000000] RAMDISK: 37778000 - 37fef9b2
[ 0.000000] ACPI: RSDP 000FE020, 0014 (r0 INTEL )
[ 0.000000] ACPI: RSDT CFEFD038, 004C (r1 INTEL D975XBX2 AE8
1000013)
[ 0.000000] ACPI: DSDT CFEF8000, 3F11 (r1 INTEL D975XBX2 AE8
MSFT 1000013)
[ 0.000000] ACPI: SSDT CFEF2000, 01BC (r1 INTEL CpuPm AE8
MSFT 1000013)
[ 0.000000] ACPI: SSDT CFEF1000, 0175 (r1 INTEL Cpu0Ist AE8
MSFT 1000013)
[ 0.000000] ACPI: SSDT CFEF0000, 0175 (r1 INTEL Cpu1Ist AE8
MSFT 1000013)
[ 0.000000] ACPI: SSDT CFEAB000, 0175 (r1 INTEL Cpu2Ist AE8
MSFT 1000013)
[ 0.000000] ACPI: SSDT CFEAA000, 0175 (r1 INTEL Cpu3Ist AE8
MSFT 1000013)
[ 0.000000] #3 [0037778000 - 0037fef9b2] RAMDISK ==>
[0037778000 - 0037fef9b2]
[ 0.000000] [ffffe20000000000-ffffe200043fffff] PMD ->
[ffff880028200000-ffff88002c5fffff] on node 0
[ 0.013732] ACPI: Checking initramfs for custom DSDT
[ 0.920745] ACPI: EC: Look up EC in DSDT
[ 1.835081] Fixed MDIO Bus: probed
[ 1.835147] Driver 'sd' needs updating - please use bus_type methods
[ 1.835891] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x40b0
irq 14
[ 1.835893] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x40b8
irq 15
[ 2.056689] ata3: SATA max UDMA/133 cmd 0x40c8 ctl 0x40e4 bmdma
0x40a0 irq 19
[ 2.056691] ata4: SATA max UDMA/133 cmd 0x40c0 ctl 0x40e0 bmdma
0x40a8 irq 19
[ 2.250605] ata3.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 2.510628] ata4.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 2.591311] scsi 2:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 2.591402] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 2.591414] sd 2:0:0:0: [sda] Write Protect is off
[ 2.591416] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 2.591435] sd 2:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 2.591478] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 2.591489] sd 2:0:0:0: [sda] Write Protect is off
[ 2.591490] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 2.591508] sd 2:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 2.591511] sda: unknown partition table
[ 2.600717] sd 2:0:0:0: [sda] Attached SCSI disk
[ 2.600745] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 2.600855] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 2.600866] sd 2:0:1:0: [sdb] Write Protect is off
[ 2.600868] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[ 2.600886] sd 2:0:1:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 2.600924] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 2.600934] sd 2:0:1:0: [sdb] Write Protect is off
[ 2.600936] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[ 2.600954] sd 2:0:1:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 2.600957] sdb: unknown partition table
[ 3.072070] sd 2:0:1:0: [sdb] Attached SCSI disk
[ 3.072115] sd 2:0:1:0: Attached scsi generic sg2 type 0
[ 3.072178] scsi 3:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 3.072237] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 3.072248] sd 3:0:0:0: [sdc] Write Protect is off
[ 3.072249] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 3.072268] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 3.072303] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 3.072313] sd 3:0:0:0: [sdc] Write Protect is off
[ 3.072315] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 3.072333] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 3.072336] sdc: unknown partition table
[ 3.079128] sd 3:0:0:0: [sdc] Attached SCSI disk
[ 3.079156] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 3.079257] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors:
(150 GB/139 GiB)
[ 3.079268] sd 3:0:1:0: [sdd] Write Protect is off
[ 3.079269] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
[ 3.079287] sd 3:0:1:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 3.079322] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors:
(150 GB/139 GiB)
[ 3.079332] sd 3:0:1:0: [sdd] Write Protect is off
[ 3.079334] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
[ 3.079352] sd 3:0:1:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 3.079354] sdd: sdd1 sdd2 < sdd5 >
[ 3.101054] sd 3:0:1:0: [sdd] Attached SCSI disk
[ 3.101107] sd 3:0:1:0: Attached scsi generic sg4 type 0
[ 3.470676] ata5.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 3.880672] ata6.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 4.290698] ata7.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 4.700696] ata8.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[ 4.740750] scsi 4:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 4.740831] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.740842] sd 4:0:0:0: [sde] Write Protect is off
[ 4.740844] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
[ 4.740862] sd 4:0:0:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.740898] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.740909] sd 4:0:0:0: [sde] Write Protect is off
[ 4.740910] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
[ 4.740928] sd 4:0:0:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.740930] sde: unknown partition table
[ 4.749416] sd 4:0:0:0: [sde] Attached SCSI disk
[ 4.749443] sd 4:0:0:0: Attached scsi generic sg5 type 0
[ 4.749488] scsi 5:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 4.749546] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.749557] sd 5:0:0:0: [sdf] Write Protect is off
[ 4.749559] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 4.749576] sd 5:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.749613] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.749623] sd 5:0:0:0: [sdf] Write Protect is off
[ 4.749624] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[ 4.749642] sd 5:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.749645] sdf: unknown partition table
[ 4.759788] sd 5:0:0:0: [sdf] Attached SCSI disk
[ 4.759815] sd 5:0:0:0: Attached scsi generic sg6 type 0
[ 4.759859] scsi 6:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 4.759919] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.759930] sd 6:0:0:0: [sdg] Write Protect is off
[ 4.759932] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00
[ 4.759950] sd 6:0:0:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.759986] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.759997] sd 6:0:0:0: [sdg] Write Protect is off
[ 4.759998] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00
[ 4.760022] sd 6:0:0:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.760027] sdg: unknown partition table
[ 4.771408] sd 6:0:0:0: [sdg] Attached SCSI disk
[ 4.771439] sd 6:0:0:0: Attached scsi generic sg7 type 0
[ 4.771483] scsi 7:0:0:0: Direct-Access ATA ST31000340AS
SD1A PQ: 0 ANSI: 5
[ 4.771540] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.771551] sd 7:0:0:0: [sdh] Write Protect is off
[ 4.771552] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00
[ 4.771570] sd 7:0:0:0: [sdh] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.771608] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[ 4.771618] sd 7:0:0:0: [sdh] Write Protect is off
[ 4.771620] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00
[ 4.771637] sd 7:0:0:0: [sdh] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 4.771640] sdh: unknown partition table
[ 4.781982] sd 7:0:0:0: [sdh] Attached SCSI disk
[ 4.782012] sd 7:0:0:0: Attached scsi generic sg8 type 0
[ 4.942482] block sdd5: hash matches
[ 5.048354] md: linear personality registered for level -1
[ 5.050051] md: multipath personality registered for level -4
[ 5.051538] md: raid0 personality registered for level 0
[ 5.053697] md: raid1 personality registered for level 1
[ 5.271272] raid6: int64x1 1922 MB/s
[ 5.441269] raid6: int64x2 2639 MB/s
[ 5.611280] raid6: int64x4 2025 MB/s
[ 5.781295] raid6: int64x8 1789 MB/s
[ 5.951290] raid6: sse2x1 3078 MB/s
[ 6.121297] raid6: sse2x2 3651 MB/s
[ 6.291275] raid6: sse2x4 7193 MB/s
[ 6.291281] raid6: using algorithm sse2x4 (7193 MB/s)
[ 6.291283] md: raid6 personality registered for level 6
[ 6.291285] md: raid5 personality registered for level 5
[ 6.291286] md: raid4 personality registered for level 4
[ 6.296306] md: raid10 personality registered for level 10
[ 6.494887] md: bind<sde>
[ 6.555824] md: bind<sdf>
[ 6.622315] md: bind<sdh>
[ 6.703948] md: bind<sda>
[ 6.780815] md: bind<sdg>
[ 7.023507] md: bind<sdc>
[ 7.223400] md: bind<sdb>
[ 7.226582] raid5: device sdb operational as raid disk 1
[ 7.226584] raid5: device sdc operational as raid disk 2
[ 7.226586] raid5: device sdg operational as raid disk 5
[ 7.226587] raid5: device sda operational as raid disk 0
[ 7.226588] raid5: device sdh operational as raid disk 6
[ 7.226589] raid5: device sdf operational as raid disk 4
[ 7.226591] raid5: device sde operational as raid disk 3
[ 7.227201] raid5: allocated 7450kB for md0
[ 7.227203] raid5: raid level 5 set md0 active with 7 out of 7
devices, algorithm 2
[ 7.227205] RAID5 conf printout:
[ 7.227207] disk 0, o:1, dev:sda
[ 7.227208] disk 1, o:1, dev:sdb
[ 7.227209] disk 2, o:1, dev:sdc
[ 7.227210] disk 3, o:1, dev:sde
[ 7.227212] disk 4, o:1, dev:sdf
[ 7.227213] disk 5, o:1, dev:sdg
[ 7.227214] disk 6, o:1, dev:sdh
[ 7.228333] md0: unknown partition table
[ 12.709829] Adding 5992204k swap on /dev/sdd5. Priority:-1 extents:1
across:5992204k
[ 13.245731] EXT3 FS on sdd1, internal journal
[ 317.618298] type=1505 audit(1241888758.168:7):
operation="profile_load" name="/usr/sbin/cupsd" name2="default" pid=2550
[ 318.016442] Installing knfsd (copyright (C) 1996 okir@...ad.swb.de).
[ 322.767731] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
recovery directory
[ 322.774119] NFSD: starting 90-second grace period
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists