netdev - Re: kTLS in combination with mlx4 is very unstable

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <91d415bd-e919-dc87-d9ea-d8c30a7e90f0@tomt.net>
Date:   Mon, 23 Apr 2018 13:44:13 +0200
From:   Andre Tomt <andre@...t.net>
To:     netdev <netdev@...r.kernel.org>
Cc:     davejwatson@...com, Tariq Toukan <tariqt@...lanox.com>
Subject: Re: kTLS in combination with mlx4 is very unstable

On 22. april 2018 23:21, Andre Tomt wrote:
> Hello!
> 
> kTLS looks fun, so I decided to play with it. It is quite spiffy - 
> however with mlx4 I get kernel crashes I'm not seeing when testing on 
> ixgbe.
> 
> For testing I'm using a git build of the "stream reflector" cubemap[1] 
> configured with kTLS and 8 worker threads running on 4 physical cores, 
> loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV.
> 
> The kernel seems to get increasingly unstable as I load it up with 
> client connections. At about 9Gbps and 700 connections, it is okay at 
> least for a while - it might run fine for say 45 minutes. Once it gets 
> to 20 - 30Gbps, the kernel will usually start spewing OOPSes within 
> minutes and the traffic drops.
> 
> Some bad interaction between mlx4 and kTLS?
> 
> Hardware is a quad core Xeon-D 1520 using a dual port Mellanox 
> ConnectX-3 VPI with a single 40Gbps ethernet link configured. Mellanox 
> mlx4 driver settings are kernel.org upstream defaults. Interface is 
> configured with FQ qdisc and sockets are using BBR congestion control.
> 
> Tested on kernel 4.14.34, 4.15.17, and 4.16.2 - 4.16.3.
> 
> [1] https://git.sesse.net/?p=cubemap
> 
> First OOPS (from 4.16.3)

It also blows up with a similar trace on 4.17-rc2.