linux-kernel - Re: NFS: low read/stat performance on small files

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1332505002.20500.14.camel@lade.trondhjem.org>
Date:	Fri, 23 Mar 2012 12:16:40 +0000
From:	"Myklebust, Trond" <Trond.Myklebust@...app.com>
To:	Jim Rees <rees@...ch.edu>
CC:	Vivek Trivedi <vtrivedi018@...il.com>,
	"linux-nfs@...r.kernel.org" <linux-nfs@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	Namjae Jeon <linkinjeon@...il.com>,
	"amit.sahrawat83@...il.com" <amit.sahrawat83@...il.com>
Subject: Re: NFS: low read/stat performance on small files

On Fri, 2012-03-23 at 07:49 -0400, Jim Rees wrote:
> Vivek Trivedi wrote:
> 
>   204800 bytes (200.0KB) copied, 0.027074 seconds, 7.2MB/s
>   Read speed for 200KB file is 7.2 MB
> 
>   104857600 bytes (100.0MB) copied, 9.351221 seconds, 10.7MB/s
>   Read speed for 100MB file is 10.7 MB
>   
>   As you see read speed for 200KB file is only 7.2MB/sec while it is
>   10.7 MB/sec when we read 100MB file.
>   Why there is so much difference in read performance ?
>   Is there any way to achieve high read speed for small files ?
> 
> That seems excellent to me.  204800 bytes at 11213252 per sec would be 18.2
> msec, so your per-file overhead is around 9 msec.  The disk latency alone
> would normally be more than that.

...and the reason why the performance is worse for the 200K file
compared to the 100M one is easily explained.

When opening the file for reading, the client has a number of
synchronous RPC calls to make: it needs to look up the file, check
access permissions and possibly revalidate its cache. All these tasks
have to be done in series (you cannot do them in parallel), and so the
latency of each task is limited by the round-trip time to the server.

On the other hand, once you get to doing READs, the client can send a
bunch of readahead requests in parallel, thus ensuring that the server
can use all the bandwidth available to the TCP connection.

So your result is basically showing that for small files, the proportion
of (readahead) tasks that can be done in parallel is smaller. This is as
expected.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@...app.com
www.netapp.com