[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-Id: <F59D6EFA-C8EC-4678-97CE-8726260606EE@neuf.fr>
Date: Mon, 7 Jan 2008 15:58:29 +0100
From: Guillaume Laurès <guillaume-laures@...f.fr>
To: linux-kernel@...r.kernel.org
Subject: PROBLEM: I/O scheduler problem with an 8 SATA disks raid 5 under heavy load ?
Hello,
Dear kernel developers, my dmesg asked me to report this, so here I
go ;)
Here is what I found in my dmesg: "anticipatory: forced dispatching
is broken (nr_sorted=1), please report this".
- First, let's talk about the machine: it's quite pushed so maybe the
cause is me doing something wrong rather than a bug in the kernel.
I got this alert on a dual core amd64 xen host. It has 8 SATA drives
making a raid 5 array. This array makes a virtual block device for
one of the virtual machines: an Openfiler appliance. Openfiler then
manages logical volumes on this device including an XFS partition
shared via NFS. 2 MythTV hosts continuously write MPEG2 tv shows on
it (1 to 4Gb each).
Still following ? Here is a summary: MPEG2 files -> NFS -> XFS -> LVM
-> Xen VBD -> RAID 5 -> 8x SATA disks.
- Next, the symptoms.
This setup is only 2 weeks old. Behavior was quite good, except for
some unexplained failures from the sata_nv attached disks. Not always
from the same disk. Never from any disks attached through the
sata_sil HBA.
Eventually a second disk would go down before the end of the raid
reconstruction (still a sata_nv attached one).
Since the disks showed nothing wrong with smartmontools I re-added
them each time. So far the raid array was strong enough to be fully
recovered, mdadm --force and xfs_check are my friends ;-)
It seems to happen more often now that the XFS partition is quite
heavily fragmented, and I can't even run the defragmenter without a
quick failure.
I didn't payed big attention to the logs and quickly decided to buy a
SATA Sil PCI card to get rid of the Nvidia SATA HBA.
- Now the problem.
Yesterday, however, the MPEG2 streams hanged for a few tens of
seconds just as usual. But there were no disk failure. The array was
still in good shape, although dmesg showed the same "ata[56]:
Resetting port", "SCSI errors" etc. fuss.
However this was new in dmesg: "anticipatory: forced dispatching is
broken (nr_sorted=1), please report this". Got 4 identical in a row.
Maybe managing 8 block devices queues under load with the
anticipatory scheduler is too much ? I immediately switched to
deadline on the 8 disks, and I'll see if it it happens again by
stressing the whole system more and more.
I have no clue if anticipatory is a good choice or definitely not in
my case, anyone can point some documentation or good advices ?
- How to reproduce.
Here is what I would do:
Harness a small CPU with lots of sata/scsi drives.
Do raid 5 with big block size (1-4Mb) on it.
Make a 50G XFS file system with sunit/swidth options
Trigger bonnie++ with 1G<files<4G and fill the FS to 80-95%, trying
to achieve 98%+ fragmentation.
Defrag !
- Finally the usual bug report stuff is attached.
Download attachment "cpuinfo" of type "application/octet-stream" (1238 bytes)
View attachment "dmesg-06-01_07-01.txt" of type "text/plain" (30863 bytes)
Download attachment "lspci" of type "application/octet-stream" (25799 bytes)
Download attachment "modules" of type "application/octet-stream" (818 bytes)
Download attachment "scsi" of type "application/octet-stream" (1918 bytes)
Download attachment "ver_linux" of type "application/octet-stream" (925 bytes)
Thanks for your help !
GoM
Powered by blists - more mailing lists