linux-kernel - Re: using splice/vmsplice to improve file receive performance

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c70ff3ad0612220318i54e7569fn161cf781d9bf0669@mail.gmail.com>
Date:	Fri, 22 Dec 2006 13:18:24 +0200
From:	"saeed bishara" <saeed.bishara@...il.com>
To:	"Jens Axboe" <jens.axboe@...cle.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: using splice/vmsplice to improve file receive performance

On 12/22/06, Jens Axboe <jens.axboe@...cle.com> wrote:
> On Thu, Dec 21 2006, saeed bishara wrote:
> > Hi,
> > I'm trying to use the splice/vmsplice system calls to improve the
> > samba server write throughput, but before touching the smbd, I started
> > to improve the ttcp tool since it simple and has the same flow. I'm
> > expecting to avoid the "copy_from_user" path when using those
> > syscalls.
> > so far, I couldn't make any improvement, actually the throughput get
> > worst. the new receive flow looks like this (code also attached):
> > 1. read tcp packet (64 pages) to page aligned buffer.
> > 2. vmsplice the buffer to pipe with SPLICE_F_MOVE.
> > 3. splice the pipe to the file, also with SPLICE_F_MOVE.
> >
> > the strace shows that the splice takes a lot of time. also when
> > profiling the kernel, I found that the memcpy() called to often !!
>
> (didn't see this until now, axboe@...e.de doesn't work anymore)
>
> I'm assuming that you mean you vmsplice with SPLICE_F_GIFT, to hand
> ownership of the pages to the kernel (in which case SPLICE_F_MOVE will
> work, otherwise you get a copy)? If not, that'll surely cost you a data
> copy
   I'll try the vmplice with SPLICE_F_GIFT and splice with MOVE. btw,
I noticed that the  splice system call takes the bulk of the time,
does it mean anything?
>
> This sounds remarkably like a recent thread on lkml, you may want to
> read up on that. Basically using splice for network receive is a bit of
> a work-around now, since you do need the one copy and then vmsplice that
> into a pipe. To realize the full potential of splice, we first need
> socket receive support so you can skip that step (splice from socket to
> pipe, splice pipe to file).
Ashwini Kulkarni posted patches that implements that, see
http://lkml.org/lkml/2006/9/20/272 .  is that right?
>
> There was no test code attached, btw.
sorry, here it is.
can you please add sample application to your test tools (splice,fio
,,) that demonstrates my flow; socket to file using read & vmsplice?
>
> --
> Jens Axboe
>
>

View attachment "ttcp.c" of type "text/plain" (17804 bytes)

View attachment "splice.h" of type "text/plain" (3448 bytes)

Download attachment "Makefile" of type "application/octet-stream" (720 bytes)