netdev - Re: iperf performance regression since Linux 5.18

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a35b1a27-575f-4d19-ad2d-95bf4ded40e9@gmx.net>
Date: Fri, 13 Oct 2023 15:37:04 +0200
From: Stefan Wahren <wahrenst@....net>
To: Neal Cardwell <ncardwell@...gle.com>, Eric Dumazet <edumazet@...gle.com>
Cc: Jakub Kicinski <kuba@...nel.org>, Fabio Estevam <festevam@...il.com>,
 linux-imx@....com, Stefan Wahren <stefan.wahren@...rgebyte.com>,
 Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org
Subject: Re: iperf performance regression since Linux 5.18

Hi,

Am 09.10.23 um 21:19 schrieb Neal Cardwell:
> On Mon, Oct 9, 2023 at 3:11 PM Eric Dumazet <edumazet@...gle.com> wrote:
>> On Mon, Oct 9, 2023 at 8:58 PM Stefan Wahren <wahrenst@....net> wrote:
>>> Hi,
>>> we recently switched on our ARM NXP i.MX6ULL based embedded device
>>> (Tarragon Master [1]) from an older kernel version to Linux 6.1. After
>>> that we noticed a measurable performance regression on the Ethernet
>>> interface (driver: fec, 100 Mbit link) while running iperf client on the
>>> device:
>>>
>>> BAD
>>>
>>> # iperf -t 10 -i 1 -c 192.168.1.129
>>> ------------------------------------------------------------
>>> Client connecting to 192.168.1.129, TCP port 5001
>>> TCP window size: 96.2 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 192.168.1.12 port 56022 connected with 192.168.1.129 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  9.88 MBytes  82.8 Mbits/sec
>>> [  3]  1.0- 2.0 sec  9.62 MBytes  80.7 Mbits/sec
>>> [  3]  2.0- 3.0 sec  9.75 MBytes  81.8 Mbits/sec
>>> [  3]  3.0- 4.0 sec  9.62 MBytes  80.7 Mbits/sec
>>> [  3]  4.0- 5.0 sec  9.62 MBytes  80.7 Mbits/sec
>>> [  3]  5.0- 6.0 sec  9.62 MBytes  80.7 Mbits/sec
>>> [  3]  6.0- 7.0 sec  9.50 MBytes  79.7 Mbits/sec
>>> [  3]  7.0- 8.0 sec  9.75 MBytes  81.8 Mbits/sec
>>> [  3]  8.0- 9.0 sec  9.62 MBytes  80.7 Mbits/sec
>>> [  3]  9.0-10.0 sec  9.50 MBytes  79.7 Mbits/sec
>>> [  3]  0.0-10.0 sec  96.5 MBytes  80.9 Mbits/sec
>>>
>>> GOOD
>>>
>>> # iperf -t 10 -i 1 -c 192.168.1.129
>>> ------------------------------------------------------------
>>> Client connecting to 192.168.1.129, TCP port 5001
>>> TCP window size: 96.2 KByte (default)
>>> ------------------------------------------------------------
>>> [  3] local 192.168.1.12 port 54898 connected with 192.168.1.129 port 5001
>>> [ ID] Interval       Transfer     Bandwidth
>>> [  3]  0.0- 1.0 sec  11.2 MBytes  94.4 Mbits/sec
>>> [  3]  1.0- 2.0 sec  11.0 MBytes  92.3 Mbits/sec
>>> [  3]  2.0- 3.0 sec  10.8 MBytes  90.2 Mbits/sec
>>> [  3]  3.0- 4.0 sec  11.0 MBytes  92.3 Mbits/sec
>>> [  3]  4.0- 5.0 sec  10.9 MBytes  91.2 Mbits/sec
>>> [  3]  5.0- 6.0 sec  10.9 MBytes  91.2 Mbits/sec
>>> [  3]  6.0- 7.0 sec  10.8 MBytes  90.2 Mbits/sec
>>> [  3]  7.0- 8.0 sec  10.9 MBytes  91.2 Mbits/sec
>>> [  3]  8.0- 9.0 sec  10.9 MBytes  91.2 Mbits/sec
>>> [  3]  9.0-10.0 sec  10.9 MBytes  91.2 Mbits/sec
>>> [  3]  0.0-10.0 sec   109 MBytes  91.4 Mbits/sec
>>>
>>> We were able to bisect this down to this commit:
>>>
>>> first bad commit: [65466904b015f6eeb9225b51aeb29b01a1d4b59c] tcp: adjust
>>> TSO packet sizes based on min_rtt
>>>
>>> Disabling this new setting via:
>>>
>>> echo 0 > /proc/sys/net/ipv4/tcp_tso_rtt_log
>>>
>>> confirm that this was the cause of the performance regression.
>>>
>>> Is it expected that the new default setting has such a performance impact?
> Indeed, thanks for the report.
>
> In addition to the "ss" output Eric mentioned, could you please grab
> "nstat" output, which should allow us to calculate the average TSO/GSO
> and LRO/GRO burst sizes, which is the key thing tuned with the
> tcp_tso_rtt_log knob.
>
> So it would be great to have the following from both data sender and
> data receiver, for both the good case and bad case, if you could start
> these before your test and kill them after the test stops:
>
> (while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
> nstat -n; (while true; do date; nstat; sleep 1; done)  > /root/nstat.txt
i upload everything here:
https://github.com/lategoodbye/tcp_tso_rtt_log_regress

The server part is a Ubuntu installation connected to the internet. At
first i logged the good case, then i continued with the bad case.
Accidentally i delete a log file of bad case, so i repeated the whole
bad case again. So the uploaded bad case files are from the third run.

>
> Thanks!
> neal