Message-ID: <4FA93BB2.9050509@pocock.com.au>
Date:	Tue, 08 May 2012 15:28:50 +0000
From:	Daniel Pocock <daniel@...ock.com.au>
To:	Martin Steigerwald <ms@...mix.de>
CC:	Martin Steigerwald <Martin@...htvoll.de>,
	Andreas Dilger <adilger@...ger.ca>, linux-ext4@...r.kernel.org
Subject: Re: ext4, barrier, md/RAID1 and write cache



On 08/05/12 14:55, Martin Steigerwald wrote:
> Am Dienstag, 8. Mai 2012 schrieb Daniel Pocock:
>> On 08/05/12 00:24, Martin Steigerwald wrote:
>>> Am Montag, 7. Mai 2012 schrieb Daniel Pocock:
>>>> On 07/05/12 20:59, Martin Steigerwald wrote:
>>>>> Am Montag, 7. Mai 2012 schrieb Daniel Pocock:
>>>>>>> Possibly the older disk is lying about doing cache flushes.  The
>>>>>>> wonderful disk manufacturers do that with commodity drives to make
>>>>>>> their benchmark numbers look better.  If you run some random IOPS
>>>>>>> test against this disk, and it has performance much over 100 IOPS
>>>>>>> then it is definitely not doing real cache flushes.
>>>>>
>>>>> […]
>>>>>
>>>>> I think an IOPS benchmark would be better. I.e. something like:
>>>>>
>>>>> /usr/share/doc/fio/examples/ssd-test
>>>>>
>>>>> (from flexible I/O tester debian package, also included in upstream
>>>>> tarball of course)
>>>>>
>>>>> adapted to your needs.
>>>>>
>>>>> Maybe with different iodepth or numjobs (to simulate several threads
>>>>> generating higher iodepths). With iodepth=1 I have seen 54 IOPS on a
>>>>> Hitachi 5400 rpm harddisk connected via eSATA.
>>>>>
>>>>> Important is direct=1 to bypass the pagecache.
>>>>
>>>> Thanks for suggesting this tool, I've run it against the USB disk and
>>>> an LV on my AHCI/SATA/md array
>>>>
>>>> Incidentally, I upgraded the Seagate firmware (model 7200.12 from CC34
>>>> to CC49) and one of the disks went offline shortly after I brought the
>>>> system back up.  To avoid the risk that a bad drive might interfere
>>>> with the SATA performance, I completely removed it before running any
>>>> tests. Tomorrow I'm out to buy some enterprise grade drives, I'm
>>>> thinking about Seagate Constellation SATA or even SAS.
>>>>
>>>> Anyway, onto the test results:
>>>>
>>>> USB disk (Seagate  9SD2A3-500 320GB):
>>>>
>>>> rand-write: (groupid=3, jobs=1): err= 0: pid=22519
>>>>
>>>>   write: io=46680KB, bw=796512B/s, iops=194, runt= 60012msec
>>>>   
>>>>     slat (usec): min=13, max=25264, avg=106.02, stdev=525.18
>>>>     clat (usec): min=993, max=103568, avg=20444.19, stdev=11622.11
>>>>     bw (KB/s) : min=  521, max= 1224, per=100.06%, avg=777.48,
>>>>
>>>> stdev=97.07 cpu          : usr=0.73%, sys=2.33%, ctx=12024, majf=0,
>>>> minf=20 IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%,
>>>> 32=0.0%,
>>>
>>> Please repeat the test with iodepth=1.
>>
>> For the USB device:
>>
>> rand-write: (groupid=3, jobs=1): err= 0: pid=11855
>>   write: io=49320KB, bw=841713B/s, iops=205, runt= 60001msec
>>     slat (usec): min=67, max=6234, avg=112.62, stdev=136.92
>>     clat (usec): min=684, max=97358, avg=4737.20, stdev=4824.08
>>     bw (KB/s) : min=  588, max= 1029, per=100.46%, avg=824.74, stdev=84.47
>>   cpu          : usr=0.64%, sys=2.89%, ctx=12751, majf=0, minf=21
>>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>
>>> =64=0.0%
>>
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>
>>> =64=0.0%
>>
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>
>>> =64=0.0%
>>
>>      issued r/w: total=0/12330, short=0/0
>>      lat (usec): 750=0.02%, 1000=0.48%
>>      lat (msec): 2=1.05%, 4=66.65%, 10=26.32%, 20=1.46%, 50=3.99%
>>      lat (msec): 100=0.03%
>>
>> and for the SATA disk:
>>
>> rand-write: (groupid=3, jobs=1): err= 0: pid=12256
>>   write: io=28020KB, bw=478168B/s, iops=116, runt= 60005msec
>>     slat (usec): min=58, max=132637, avg=110.51, stdev=1623.80
>>     clat (msec): min=2, max=206, avg= 8.44, stdev= 7.10
>>     bw (KB/s) : min=   95, max=  566, per=100.24%, avg=467.11, stdev=97.64
>>   cpu          : usr=0.36%, sys=1.17%, ctx=7196, majf=0, minf=21
>>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
> […]
>>      issued r/w: total=0/7005, short=0/0
>>
>>      lat (msec): 4=6.31%, 10=69.54%, 20=22.68%, 50=0.63%, 100=0.76%
>>      lat (msec): 250=0.09%
>>
>>> 194 IOPS appears to be highly unrealistic unless NCQ or something like
>>> that is in use. At least if thats a 5400/7200 RPM sata drive (didn´t
>>> check vendor information).
>>
>> The SATA disk does have NCQ
>>
>> USB disk is supposed to be 5400RPM, USB2, but reporting iops=205
>>
>> SATA disk is 7200 RPM, 3 Gigabit SATA, but reporting iops=116
>>
>> Does this suggest that the USB disk is caching data but telling Linux
>> the data is on disk?
> 
> Looks like it.
> 
> Some older values for a 1.5 TB WD Green Disk:
> 
> mango:~# fio -readonly -name iops -rw=randread -bs=512  -runtime=100 -iodepth 1 
> -filename /dev/sda -ioengine  libaio -direct=1
> [...] iops: (groupid=0, jobs=1): err= 0: pid=9939
>   read : io=1,859KB, bw=19,031B/s, iops=37, runt=100024msec [...]</pre>
> 
> 
> mango:~# fio -readonly -name iops -rw=randread -bs=512  -runtime=100 -iodepth 
> 32 -filename /dev/sda -ioengine  libaio -direct=1
> iops: (groupid=0, jobs=1): err= 0: pid=10304
>   read : io=2,726KB, bw=27,842B/s, iops=54, runt=100257msec
> 
> mango:~# hdparm -I /dev/sda | grep -i queue
>         Queue depth: 32
>            *    Native Command Queueing (NCQ)
> 
> - 1,5 TB Western Digital, WDC WD15EADS-00P8B0
> - Pentium 4 mit 2,80 GHz
> - 4 GB RAM, 32-Bit Linux
> - Linux Kernel 2.6.36
> - fio 1.38-1
> 
>>>> The IOPS scores look similar, but I checked carefully and I'm fairly
>>>> certain the disks were mounted correctly when the tests ran.
>>>>
>>>> Should I run this tool over NFS, will the results be meaningful?
>>>>
>>>> Given the need to replace a drive anyway, I'm really thinking about one
>>>> of the following approaches:
>>>> - same controller, upgrade to enterprise SATA drives
>>>> - buy a dedicated SAS/SATA controller, upgrade to enterprise SATA
>>>> drives
>>>> - buy a dedicated SAS/SATA controller, upgrade to SAS drives
>>>>
>>>> My HP N36L is quite small, one PCIe x16 slot, the internal drive cage
>>>> has an SFF-8087 (mini SAS) plug, so I'm thinking I can grab something
>>>> small like the Adaptec 1405 - will any of these solutions offer a
>>>> definite win with my NFS issues though?
>>>
>>> First I would like to understand more closely what your NFS issues are.
>>> Before throwing money at the problem its important to understand what the
>>> problem actually is.
>>
>> When I do things like unpacking a large source tarball, iostat reports
>> throughput to the drive between 500-1000kBytes/second
>>
>> When I do the same operation onto the USB drive over NFS, I see over
>> 5000kBytes/second - but it appears from the iops test figures that the
>> USB drive is cheating, so we'll ignore that.
>>
>> - if I just dd to the SATA drive over NFS (with conv=fsync), I see much
>> faster speeds
> 
> Easy. Less roundtrips.
> 
> Just watch nfsstat -3 while untarring a tarball over NFS to see what I mean.
> 
>> - if I'm logged in to the server, and I unpack the same tarball onto the
>> same LV, the operation completes at 30MBytes/sec
> 
> No network.
> 
> Thats the LV on the internal disk?


Yes

>> It is a gigabit network and I think that the performance of the dd
>> command proves it is not something silly like a cable fault (I have come
>> across such faults elsewhere though)
> 
> What is the latency?
> 
$ ping -s 1000 192.168.1.2
PING 192.168.1.2 (192.168.1.2) 1000(1028) bytes of data.
1008 bytes from 192.168.1.2: icmp_req=1 ttl=64 time=0.307 ms
1008 bytes from 192.168.1.2: icmp_req=2 ttl=64 time=0.341 ms
1008 bytes from 192.168.1.2: icmp_req=3 ttl=64 time=0.336 ms


>>> Anyway, 15000 RPM SAS drives should give you more IOPS than 7200 RPM SATA
>>> drives, but SATA drives are cheaper and thus you could - depending on
>>> RAID level - increase IOPS by just using more drives.
>>
>> I was thinking about the large (2TB or 3TB) 7200 RPM SAS or SATA drives
>> in the Seagate `Constellation' enterprise drive range.  I need more
>> space anyway, and I need to replace the drive that failed, so I have to
>> spend some money anyway - I just want to throw it in the right direction
>> (e.g. buying a drive, or if the cheap on-board SATA controller is a
>> bottleneck or just extremely unsophisticated, I don't mind getting a
>> dedicated controller)
>>
>> For example, if I knew that the controller is simply not suitable with
>> barriers, NFS, etc and that a $200 RAID card or even a $500 RAID card
>> will guarantee better performance with my current kernel, I would buy
>> that.  (However, I do want to use md RAID rather than a proprietary
>> format, so any RAID card would be in JBOD mode)
> 
> They point is: How much of the performance will arrive at NFS? I can't say 
> yet.

My impression is that the faster performance of the USB disk was a red
herring, and the problem really is just the nature of the NFS protocol
and the way it is stricter about server-side caching (when sync is
enabled) and consequently it needs more iops.

I've turned two more machines (a HP Z800 with SATA disk and a Lenovo
X220 with SSD disk) into NFSv3 servers, repeated the same tests, and
found similar performance on the Z800, but 20x faster on the SSD (which
can support more IOPS)

>>> But still first I´d like to understand *why* its slow.
>>>
>>> What does
>>>
>>> iostat -x -d -m 5
>>> vmstat 5
>>>
>>> say when excersing the slow (and probably a faster) setup? See [1].
>>
>> All the iostat output is typically like this:
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s
>> avgrq-sz avgqu-sz   await  svctm  %util
>> dm-23             0.00     0.00    0.20  187.60     0.00     0.81
>> 8.89     2.02   10.79   5.07  95.20
>> dm-23             0.00     0.00    0.20  189.80     0.00     0.91
>> 9.84     1.95   10.29   4.97  94.48
>> dm-23             0.00     0.00    0.20  228.60     0.00     1.00
>> 8.92     1.97    8.58   4.10  93.92
>> dm-23             0.00     0.00    0.20  231.80     0.00     0.98
>> 8.70     1.96    8.49   4.06  94.16
>> dm-23             0.00     0.00    0.20  229.20     0.00     0.94
>> 8.40     1.92    8.39   4.10  94.08
> 
> Hmmm, disk looks quite utilitzed. Are there other I/O workloads on the 
> machine?

No, just me testing it

>> and vmstat:
>>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
>> id wa
>> ...
>>  0  1      0 6881772 118660 576712    0    0     1  1033  720 1553  0  2
>> 60 38
>>  0  1      0 6879068 120220 577892    0    0     1   918  793 1595  0  2
>> 56 41
>>  0  1      0 6876208 122200 578684    0    0     1  1055  767 1731  0  2
>> 67 31
>>  1  1      0 6873356 124176 579392    0    0     1  1014  742 1688  0  2
>> 66 32
>>  0  1      0 6870628 126132 579904    0    0     1  1007  753 1683  0  2
>> 66 32
> 
> And wait I/O is quite high.
> 
> Thus it seems this workload can be faster with faster / more disks or a RAID 
> controller with battery (and disabling barriers / cache flushes).

You mean barrier=0,data=writeback?  Or just barrier=0,data=ordered?

In theory that sounds good, but in practice I understand it creates some
different problems, eg:

- monitoring the battery, replacing it periodically

- batteries only hold the charge for a few hours, so if there is a power
outage on a Sunday, someone tries to turn on the server on  Monday
morning and the battery has died, cache is empty and disk is corrupt

- some RAID controllers (e.g. HP SmartArray) insist on writing their
metadata to all volumes - so you become locked in to the RAID vendor.  I
prefer to just use RAID1 or RAID10 with Linux md onto the raw disks.  On
some Adaptec controllers, `JBOD' mode allows md to access the disks
directly, although I haven't verified that yet.

I'm tempted to just put a UPS on the server and enable NFS `async' mode,
and avoid running anything on the server that may cause a crash.

>> and nfsstat -s -o all -l -Z5
>>
>> nfs v3 server        total:      319
>> ------------- ------------- --------
>> nfs v3 server      getattr:        1
>> nfs v3 server      setattr:      126
>> nfs v3 server       access:        6
>> nfs v3 server        write:       61
>> nfs v3 server       create:       61
>> nfs v3 server        mkdir:        3
>> nfs v3 server       commit:       61
> 
> I would like to see nfsiostat from newer nfs-utils, cause it includes 
> latencies.
> 
>>> [1]
>>> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_whe
>>> n_reporting_a_problem.3F
>>
>> I've also tested onto btrfs and the performance was equally bad, so it
>> may not be an ext4 issue
>>
>> The environment is:
>> Linux srv1 3.2.0-0.bpo.2-amd64 #1 SMP Mon Apr 23 08:38:01 UTC 2012
>> x86_64 GNU/Linux
>> (Debian squeeze)
>> Kernel NFS v3
>> HP N36L server, onboard AHCI
>>  md RAID1 as a 1TB device (/dev/md2)
>> /dev/md2 is a PV for LVM - no other devices attached
>>
>> As mentioned before, I've tried with and without write cache.
>> dmesg reports that ext4 (and btrfs) seem to be happy to accept the
>> barrier=1 or barrier=0 setting with the drives.
> 
> 3.2 doesn't report failure on barriers anymore. Barriers have been switched to 
> cache flush requests and these will not report back failure. So you have to 
> make sure cache flushes work in other ways.
> 
>> dmesg and hdparm also appear to report accurate information about write
>> cache status.
>>
>>> (quite some of this should be relevant when reporting with ext4 as well)
>>>
>>> As for testing with NFS: I except the values to drop. NFS has quite some
>>> protocol overhead due to network roundtrips. On my nasic tests NFSv4 even
>>> more so than NFSv3. As for NFS I suggest trying nfsiostat python script
>>> from newer nfs-utils. It also shows latencies.
>>
>> I agree - but 500kBytes/sec is just so much slower than anything I've
>> seen with any IO device in recent years.  I don't expect to get 90% of
>> the performance of a local disk, but is getting 30-50% reasonable?
> 
> Depends on the workload.
> 
> You might consider using FS-Cache with cachefilesd for local client side 
> caching.
> 
> Ciao,
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists