netdev - Re: splice() performance for TCP socket forwarding

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <cfb3ed6d-ff44-c0ad-4531-9fc3d2a63e8d@gmail.com>
Date:   Thu, 13 Dec 2018 06:05:48 -0800
From:   Eric Dumazet <eric.dumazet@...il.com>
To:     Marek Majkowski <marek@...udflare.com>
Cc:     netdev@...r.kernel.org
Subject: Re: splice() performance for TCP socket forwarding



On 12/13/2018 06:03 AM, Eric Dumazet wrote:
> 
> 
> On 12/13/2018 05:33 AM, Marek Majkowski wrote:
>> Ok, 4.19 does seem to kinda fix the SO_RCVLOWAT with splice, but I
>> don't fully understand it:
>>
>> fcntl(8, F_SETPIPE_SZ, 1048576)         = 1048576 <0.000033>
>> setsockopt(4, SOL_SOCKET, SO_RCVLOWAT, [131072], 4) = 0 <0.000014>
>> splice(4, NULL, 9, NULL, 1048576, SPLICE_F_MOVE) = 121435 <71.039385>
>> splice(8, NULL, 5, NULL, 121435, SPLICE_F_MOVE) = 121435 <0.000118>
>> splice(4, NULL, 9, NULL, 1048576, SPLICE_F_MOVE) = 11806 <0.000019>
>> splice(8, NULL, 5, NULL, 11806, SPLICE_F_MOVE) = 11806 <0.000018>
>>
> 
> Good point.
> 
> At this moment SO_RCVLOWAT only tries to reduce number of POLLIN events.
> 
> But if your splice() system call is performed while there are already
> available skbs in the receive queue, splice() wont block and deliver
> what is available in the queue.
> 
> I guess that we would need to add some logic in recvmsg() and tcp_splice_read()
> to truly implement SO_RCVLOWAT : block until at least sk->sk_rcvlowat bytes are
> available in receive queue.
> 

You could also work around the problem by inserting a poll() system call before the splice(),
since poll() would only deliver the POLLIN event when sk->sk_rcvlowat bytes are present in the queue.