linux-ext4 - Re: Running fsck of huge ext4 partition takes weeks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAETWcfti7NL3N96K1pEAX9ezKUrZn0QeGTwjn0qUni0u6i=Gcg@mail.gmail.com>
Date:	Thu, 27 Aug 2015 08:28:04 +0300
From:	Alexander Afonyashin <a.afonyashin@...net-team.ru>
To:	Andreas Dilger <adilger@...ger.ca>
Cc:	linux-ext4@...r.kernel.org
Subject: Re: Running fsck of huge ext4 partition takes weeks

Hi,

The last output (2 days ago) from fsck:

[skipped]
Block #524296 (1235508688) causes directory to be too big.  CLEARED.
Block #524297 (4003498426) causes directory to be too big.  CLEARED.
Block #524298 (3113378389) causes directory to be too big.  CLEARED.
Block #524299 (1368545889) causes directory to be too big.  CLEARED.
Too many illegal blocks in inode 4425477.
Clear inode? yes

---------------------------
iostat output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.00   14.52    0.00   85.48

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
loop0             0.00     0.00    2.00    0.00    12.00     0.00
12.00     0.09   46.00   46.00    0.00  46.00   9.20
sda               0.00     0.00   87.00    0.00   348.00     0.00
8.00     1.00   11.86   11.86    0.00  11.45  99.60

---------------------------
strace ouput:

root@...cue ~ # strace -f -t -p 4779
Process 4779 attached - interrupt to quit
07:26:54 lseek(4, 14154266963968, SEEK_SET) = 14154266963968
07:26:54 read(4,
"\277\224\312\371\302\356\tJC{P\244#3\"2P\327*2Q5\372\206\262\20\\\373\226\262\21\316"...,
4096) = 4096
07:27:02 lseek(4, 1408506736640, SEEK_SET) = 1408506736640
07:27:02 read(4,
"\352\3041\345\1\337p\263l;\354\377E[\17\350\235\260\r\344\265\337\3655\223E\216\226\376\263!\n"...,
4096) = 4096
07:27:08 lseek(4, 5948177264640, SEEK_SET) = 5948177264640
07:27:08 read(4,
"\321}\226m;1\253Z\301f\205\235\25\201\334?\311AQN(\22!\23{\345\214Vi\240=y"...,
4096) = 4096
07:27:10 brk(0x8cf18e000)               = 0x8cf18e000
07:27:14 lseek(4, 6408024879104, SEEK_SET) = 6408024879104
07:27:14 read(4,
"\254n\fn\r\302$\t\213\231\256\2774\326\34\364\fY\v\365`*Br\354X\7T3J\243K"...,
4096) = 4096
07:27:21 lseek(4, 8640894586880, SEEK_SET) = 8640894586880
07:27:21 read(4,
"3\372\24\357\3579\254\31\214L\rYrurj\376\250\352%\2\242\255\252\22\347XU\327\235\362\337"...,
4096) = 4096
^CProcess 4779 detached

Regards,
Alexander

On Tue, Aug 25, 2015 at 10:43 PM, Andreas Dilger <adilger@...ger.ca> wrote:
> On Aug 25, 2015, at 9:30 AM, Alexander Afonyashin <a.afonyashin@...net-team.ru> wrote:
>>
>> Hi,
>>
>> Recently I had to run fsck on 47TB ext4 partition backed by hardware
>> RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck
>> is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats
>> 100% of single CPU. It detected errors (and fixed them) but doesn't
>> finish yet.
>>
>> Rescue disc is based on Debian 7.8.
>> kernel: 4.1.4-5
>> e2fsprogs: 1.42.5-1.1+deb7u1
>>
>> Any suggestions?
>
> Usually the only reason for e2fsck to run so long is because of
> duplicate block pass 1b/1c.
>
> Having some of the actual output of e2fsck would allow us to give
> some useful advice.
>
> The only thing I can offer is for you to run "strace -p <e2fsck_pid>"
> and/or "ltrace -p <e2fsck_pid>" to see what it is doing.
>
> Cheers, Andreas
>
>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html