netdev - Re: splice() performance for TCP socket forwarding

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20181213135720.GB16149@1wt.eu>
Date:   Thu, 13 Dec 2018 14:57:20 +0100
From:   Willy Tarreau <w@....eu>
To:     Eric Dumazet <eric.dumazet@...il.com>
Cc:     Marek Majkowski <marek@...udflare.com>, netdev@...r.kernel.org
Subject: Re: splice() performance for TCP socket forwarding

Hi Eric!

On Thu, Dec 13, 2018 at 05:37:11AM -0800, Eric Dumazet wrote:
> Maybe mlx5 driver is in LRO mode, packing TCP payload in 4K pages ?

I could be wrong but I don't think so : I remember having been used to
LRO on myri10ge a decade ago giving me good performance which would
degrade with concurrent connections, till the point LRO got deprecated
when GRO started to work quite well. Thus this got me used to always
disabling LRO to be sure to measure something durable ;-)

> bnx2x GRO/LRO has this mode, meaning that around 8 pages are used for a GRO packets of ~32 KB,
> while mlx4 for instance would use one page frag for every ~1428 bytes of payload.

I remember that it was the same on myri10ge (1 segment per page), making
splice() return rougly 21 or 22kB per call for a 64kB pipe. BTW, I think
I said bullshit and that 3 years ago it was mlx4 and not mlx5 that I've
been using.

Willy