linux-ext4 - Re: sudden (big) performance drop in writes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <67D336C0-4529-477D-B0B3-D840B8032F6F@dilger.ca>
Date:	Thu, 14 Jun 2012 23:34:59 -0600
From:	Andreas Dilger <adilger@...ger.ca>
To:	Carlos Carvalho <carlos@...ica.ufpr.br>
Cc:	linux-ext4@...r.kernel.org
Subject: Re: sudden (big) performance drop in writes

On 2012-06-14, at 10:20 PM, Carlos Carvalho wrote:
> Our server has suddenly become extremely slow in writes. With
> 
> % dd if=/dev/zero of=zero bs=2M count=2000
> 
> I get only 1.4MB/s in an almost idle machine. It has a raid6 with 6
> disks, running 3.3.7. 3.4.2 doesn't improve matters.
> 
> The important point is that it became so slow yesterday. No hardware
> has changed and all disks are fine. Reading from the filesystem is
> unaffected, at more than 85MB/s. The problem happens both in the root
> and home partitions.
> 
> During the dd the disk utilization measured by sar is small:
> 
> DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
> sdb     93.30    254.40    931.20     12.71      0.57      6.10      4.16     38.77
> sdc    100.50    244.00    970.40     12.08      0.76      7.59      3.80     38.20
> sdd    101.40    261.60    927.20     11.72      0.68      6.68      3.74     37.90
> sde     86.80    300.00    780.80     12.45      0.69      7.93      4.29     37.20
> sdf     82.90    315.20    810.40     13.58      0.55      6.60      4.39     36.37
> sda     96.70    220.00    984.00     12.45      0.47      4.87      3.72     35.94
> 
> So it seems that it's not a disk problem. Both filesystems reached 90%
> usage in space but less than 10% in inodes shortly before the slowness
> appeared. Now they're at 57% and 77% but still crawl. Could this be
> related?

Definitely yes.  I was going to ask this question even before I got to
the end and saw your comment.  If the filesystem gets too full, it can
cause new allocations to become fragmented, and then even if you delete
other files, the fragmented files end up leaving individual allocated
blocks all around your filesystem, making future allocations bad also.

You can run "e2freefrag" on the block device to report the current sizes
for free space in the filesystem.  There is the "e4defrag" tool that can
defragment files, but I don't recall whether this is working well or not.

Cheers, Andreas





--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html