netdev - Re: mvpp2: incorrect max mtu?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAPv3WKcRpk+7y_TN1dsSE0rS90vTk5opU59i5=4=XP-805axfQ@mail.gmail.com>
Date:   Fri, 14 May 2021 16:25:48 +0200
From:   Marcin Wojtas <mw@...ihalf.com>
To:     "Russell King (Oracle)" <linux@...linux.org.uk>
Cc:     Stefan Chulski <stefanc@...vell.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: mvpp2: incorrect max mtu?

Hi Russell,


pt., 14 maj 2021 o 15:00 Russell King (Oracle) <linux@...linux.org.uk>
napisał(a):
>
> Hi all,
>
> While testing out the 10G speeds on my Macchiatobin platforms, the first
> thing I notice is that they only manage about 1Gbps at a MTU of 1500.
> As expected, this increases when the MTU is increased - a MTU of 9000
> works, and gives a useful performance boost.
>
> Then comes the obvious question - what is the maximum MTU.
>
> #define MVPP2_BM_JUMBO_FRAME_SIZE       10432   /* frame size 9856 */
>
> So, one may assume that 9856 is the maximum. However:
>
> # ip li set dev eth0 mtu 9888
> # ip li set dev eth0 mtu 9889
> Error: mtu greater than device maximum.
>
> So, the maximum that userspace can set appears to be 9888. If this is
> set, then, while running iperf3, we get:
>
> mvpp2 f2000000.ethernet eth0: bad rx status 9202e510 (resource error), size=9888
>
> So clearly this is too large, and we should not be allowing userspace
> to set this large a MTU.
>
> At this point, it seems to be impossible to regain the previous speed of
> the interface by lowering the MTU. Here is a MTU of 9000:
>
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec   40   17.5 KBytes
> [  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   2.00-3.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   3.00-4.00   sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> [  5]   4.00-5.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec   39   8.74 KBytes
> [  5]   6.00-7.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   39   8.74 KBytes
> [  5]   8.00-9.00   sec  1.13 MBytes  9.45 Mbits/sec   36   17.5 KBytes
> [  5]   9.00-10.00  sec  1.13 MBytes  9.45 Mbits/sec   39   8.74 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  11.9 MBytes  9.99 Mbits/sec  379             sender
> [  5]   0.00-10.00  sec  11.7 MBytes  9.80 Mbits/sec                  receiver
>
> Whereas before the test, it was:
>
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-1.00   sec   729 MBytes  6.11 Gbits/sec
> [  5]   1.00-2.00   sec   719 MBytes  6.03 Gbits/sec
> [  5]   2.00-3.00   sec   773 MBytes  6.49 Gbits/sec
> [  5]   3.00-4.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   4.00-5.00   sec   779 MBytes  6.54 Gbits/sec
> [  5]   5.00-6.00   sec   784 MBytes  6.58 Gbits/sec
> [  5]   6.00-7.00   sec   777 MBytes  6.52 Gbits/sec
> [  5]   7.00-8.00   sec   774 MBytes  6.50 Gbits/sec
> [  5]   8.00-9.00   sec   769 MBytes  6.45 Gbits/sec
> [  5]   9.00-10.00  sec   774 MBytes  6.49 Gbits/sec
> [  5]  10.00-10.00  sec  3.07 MBytes  5.37 Gbits/sec
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate
> [  5]   0.00-10.00  sec  7.47 GBytes  6.41 Gbits/sec                  receiver
>
> (this is on the server end of iperf3, the others are the client end,
> but the results were pretty very similar to that.)
>
> So, clearly something bad has happened to the buffer management as a
> result of raising the MTU so high.
>
> As the end which has suffered this issue is the mcbin VM host, I'm not
> currently in a position I can reboot it without cause major disruption
> to my network. However, thoughts on this (and... can others reproduce
> it) would be useful.
>

Thank your for the information. I will take a look after the weekend.
To be aligned - what exactly kernel baseline are you using?

Best regards,
Marcin