Message-ID: <4FA7FBDB.7070205@pocock.com.au>
Date:	Mon, 07 May 2012 18:44:11 +0200
From:	Daniel Pocock <daniel@...ock.com.au>
To:	Martin Steigerwald <Martin@...htvoll.de>
CC:	linux-ext4@...r.kernel.org
Subject: Re: ext4, barrier, md/RAID1 and write cache

On 07/05/12 18:25, Martin Steigerwald wrote:
> Am Montag, 7. Mai 2012 schrieb Daniel Pocock:
>   
>> I've been having some NFS performance issues, and have been
>> experimenting with the server filesystem (ext4) to see if that is a
>> factor.
>>     
> Which NFS version is this?
>   
Normally I am using NFSv3.  I've also tried NFSv4 to see if that fixed
the problem, it made no difference.

>  
>   
>> The setup is like this:
>>
>> (Debian 6, kernel 2.6.39)
>>     
I've just updated to 3.2 from squeeze-backports, it doesn't resolve the
issue either

>> 2x SATA drive (NCQ, 32MB cache, no hardware RAID)
>> md RAID1
>> LVM
>> ext4
>>
>> a) If I use data=ordered,barrier=1 and `hdparm -W 1' on the drive, I
>> observe write performance over NFS of 1MB/sec (unpacking a big source
>> tarball)
>>     
> Is this a realistic workload scenario for production use?
>
>   

Yes, it is a small server with few users, I keep some open source
projects on it, git repositories, compiling, etc.  So the usual workload
involves unpacking source code, git checkouts, compiles that generate
many object files.  All these operations are excruciatingly slow unless
I run NFS in `async' mode, which is not considered safe.

>> b) If I use data=writeback,barrier=0 and `hdparm -W 1' on the drive, I
>> observe write performance over NFS of 10MB/sec
>>
>> c) If I just use the async option on NFS, I observe up to 30MB/sec
>>
>> I believe (b) and (c) are not considered safe against filesystem
>> corruption, so I can't use them in practice.
>>     
> Partly.
>
> b) can harm filesystem consistency unless you disable write cache on the 
> disks
>
>   

(b) only achieves any performance improvement if the write cache is
enabled - so it is not a production solution

> c) won´t harm local filesystem consistency, but should the nfs server break 
> down all data that the NFS clients sent to the server for writing which is 
> not written yet is gone.
>
>   
Most of the access is from NFS, so (c) is not a good solution either.

>> - or must I just use option (b) but make it safer with battery-backed
>> write cache?
>>     
> If you want performance and safety that is the best option from the ones 
> you mentioned, if the workload is really I/O bound on the local filesystem. 
>
> Of course you can try the usual tricks like noatime, remove rsize and 
> wsize options on the NFS client if they have a new enough kernel (they 
> autotune to much higher than the often recommended 8192 or 32768 bytes, 
> look at /proc/mounts), put ext4 journal onto an extra disk to reduce head 
> seeks, check whether enough NFS server threads are running, try a different 
> filesystem and so on.
>
>   
One further discovery I made: I decided to eliminate md and LVM.  I had
enough space to create a 256MB partition on one of the disks, and format
it directly with ext4

Writing to that partition from the NFS3 client:
- less than 500kBytes/sec (for unpacking a tarball of source code)
- around 50MB/sec (dd if=/dev/zero conv=fsync bs=65536)

and I then connected an old 5400rpm USB disk to the machine, ran the
same test from the NFS client:
- 5MBytes/sec (for unpacking a tarball of source code) - 10x faster than
the 72k SATA disk

This last test (comparing my AHCI SATA disk to the USB disk, with no md
or LVM) makes me think it is not an NFS problem, I feel it is some issue
with the barriers when used with this AHCI or SATA disk.


--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists