netdev - Re: kTLS in combination with mlx4 is very unstable

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <0f651f8e-98a3-3899-3f7d-b71378f91c4d@tomt.net>
Date:   Tue, 1 May 2018 19:41:08 +0200
From:   Andre Tomt <andre@...t.net>
To:     Dave Watson <davejwatson@...com>
Cc:     netdev <netdev@...r.kernel.org>, borisp@...lanox.com,
        Aviad Yehezkel <aviadye@...lanox.com>
Subject: Re: kTLS in combination with mlx4 is very unstable

On 01. mai 2018 18:09, Dave Watson wrote:
> On 04/24/18 10:01 AM, Dave Watson wrote:
>> On 04/22/18 11:21 PM, Andre Tomt wrote:
>>> The kernel seems to get increasingly unstable as I load it up with client
>>> connections. At about 9Gbps and 700 connections, it is okay at least for a
>>> while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps,
>>> the kernel will usually start spewing OOPSes within minutes and the traffic
>>> drops.
>>>
>>> Some bad interaction between mlx4 and kTLS?
> I tried to repro, but wasn't able to - of course I don't have an mlx4
> test setup.  If I manually add a tls_write_space call after
> do_tcp_sendpages, I get a similar stack though.
> 
> Something like the following should work, can you test?  Thanks

Thank you!

This does indeed seem to have fixed this problem. It has been sustaining 
~36Gbps and about 3000 clients for about an hour now without any crashes.

Tested on 4.17-rc3 git snapshot as of a few hours ago.

As for performance I am very happy with kTLS. This is some very cool 
stuff. I dig it. I'm getting a bit over 10Gbps per 2.5Ghz Broadwell-DE 
core on this low power quad core system. Nearly ideal network conditions 
and all the data is hot in pagecache but still. I'm going to have to add 
another port. ;-)