netdev - RE: [EXT] mvpp2: incorrect max mtu?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CO6PR18MB3873503C45634C7EBAE49AA4B02C9@CO6PR18MB3873.namprd18.prod.outlook.com>
Date:   Tue, 18 May 2021 06:09:12 +0000
From:   Stefan Chulski <stefanc@...vell.com>
To:     Russell King <linux@...linux.org.uk>
CC:     Marcin Wojtas <mw@...ihalf.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: RE: [EXT] mvpp2: incorrect max mtu?



> -----Original Message-----
> From: Russell King <linux@...linux.org.uk>
> Sent: Friday, May 14, 2021 4:00 PM
> To: Stefan Chulski <stefanc@...vell.com>
> Cc: Marcin Wojtas <mw@...ihalf.com>; netdev@...r.kernel.org
> Subject: [EXT] mvpp2: incorrect max mtu?
> 
> External Email
> 
> ----------------------------------------------------------------------
> Hi all,
> 
> While testing out the 10G speeds on my Macchiatobin platforms, the first
> thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> As expected, this increases when the MTU is increased - a MTU of 9000
> works, and gives a useful performance boost.
> 
> Then comes the obvious question - what is the maximum MTU.
> 
> #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
> 
> So, one may assume that 9856 is the maximum. However:
> 
> # ip li set dev eth0 mtu 9888
> # ip li set dev eth0 mtu 9889
> Error: mtu greater than device maximum.
> 
> So, the maximum that userspace can set appears to be 9888. If this is set,
> then, while running iperf3, we get:
> 
> mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error),
> size=9888
> 
> So clearly this is too large, and we should not be allowing userspace to set
> this large a MTU.
> 
> At this point, it seems to be impossible to regain the previous speed of the
> interface by lowering the MTU. Here is a MTU of 9000:
> 
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec   40   17.5 KBytes
> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   2.00-3.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   3.00-4.00   sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> [  5]   4.00-5.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec   39   8.74 KBytes
> [  5]   6.00-7.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   8.00-9.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   9.00-10.00  sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  11.9 MBytes  9.99 Mbits/sec  379             sender
> [  5]   0.00-10.00  sec  11.7 MBytes  9.80 Mbits/sec                  receiver
> 
> Whereas before the test, it was:
> 
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec   729 MBytes  6.11 Gbits/sec
> [  5]   1.00-2.00   sec   719 MBytes  6.03 Gbits/sec
> [  5]   2.00-3.00   sec   773 MBytes  6.49 Gbits/sec
> [  5]   3.00-4.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   4.00-5.00   sec   779 MBytes  6.54 Gbits/sec
> [  5]   5.00-6.00   sec   784 MBytes  6.58 Gbits/sec
> [  5]   6.00-7.00   sec   777 MBytes  6.52 Gbits/sec
> [  5]   7.00-8.00   sec   774 MBytes  6.50 Gbits/sec
> [  5]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   9.00-10.00  sec   774 MBytes  6.49 Gbits/sec
> [  5]  10.00-10.00  sec  3.07 MBytes  5.37 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  7.47 GBytes  6.41 Gbits/sec                  receiver
> 
> (this is on the server end of iperf3, the others are the client end, but the
> results were pretty very similar to that.)
> 
> So, clearly something bad has happened to the buffer management as a
> result of raising the MTU so high.
> 
> As the end which has suffered this issue is the mcbin VM host, I'm not
> currently in a position I can reboot it without cause major disruption to my
> network. However, thoughts on this (and... can others reproduce
> it) would be useful.
> 
> Thanks.

Look like PPv2 tried scatter frame since it was larger than Jumbo buffer size and it drained buffer pool(Buffers never released).
Received packet should be less than value set in MVPP2_POOL_BUF_SIZE_REG for long pool.

Stefan.