netdev - Re: iperf performance regression since Linux 5.18

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CADVnQymV=nv53YaC8kLC1qT1ufhJL9+w_wcZ+8AHwPRG+JRdnw@mail.gmail.com>
Date: Mon, 9 Oct 2023 15:19:47 -0400
From: Neal Cardwell <ncardwell@...gle.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Stefan Wahren <wahrenst@....net>, Jakub Kicinski <kuba@...nel.org>, 
	Fabio Estevam <festevam@...il.com>, linux-imx@....com, 
	Stefan Wahren <stefan.wahren@...rgebyte.com>, Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org
Subject: Re: iperf performance regression since Linux 5.18

On Mon, Oct 9, 2023 at 3:11 PM Eric Dumazet <edumazet@...gle.com> wrote:
>
> On Mon, Oct 9, 2023 at 8:58 PM Stefan Wahren <wahrenst@....net> wrote:
> >
> > Hi,
> > we recently switched on our ARM NXP i.MX6ULL based embedded device
> > (Tarragon Master [1]) from an older kernel version to Linux 6.1. After
> > that we noticed a measurable performance regression on the Ethernet
> > interface (driver: fec, 100 Mbit link) while running iperf client on the
> > device:
> >
> > BAD
> >
> > # iperf -t 10 -i 1 -c 192.168.1.129
> > ------------------------------------------------------------
> > Client connecting to 192.168.1.129, TCP port 5001
> > TCP window size: 96.2 KByte (default)
> > ------------------------------------------------------------
> > [  3] local 192.168.1.12 port 56022 connected with 192.168.1.129 port 5001
> > [ ID] Interval       Transfer     Bandwidth
> > [  3]  0.0- 1.0 sec  9.88 MBytes  82.8 Mbits/sec
> > [  3]  1.0- 2.0 sec  9.62 MBytes  80.7 Mbits/sec
> > [  3]  2.0- 3.0 sec  9.75 MBytes  81.8 Mbits/sec
> > [  3]  3.0- 4.0 sec  9.62 MBytes  80.7 Mbits/sec
> > [  3]  4.0- 5.0 sec  9.62 MBytes  80.7 Mbits/sec
> > [  3]  5.0- 6.0 sec  9.62 MBytes  80.7 Mbits/sec
> > [  3]  6.0- 7.0 sec  9.50 MBytes  79.7 Mbits/sec
> > [  3]  7.0- 8.0 sec  9.75 MBytes  81.8 Mbits/sec
> > [  3]  8.0- 9.0 sec  9.62 MBytes  80.7 Mbits/sec
> > [  3]  9.0-10.0 sec  9.50 MBytes  79.7 Mbits/sec
> > [  3]  0.0-10.0 sec  96.5 MBytes  80.9 Mbits/sec
> >
> > GOOD
> >
> > # iperf -t 10 -i 1 -c 192.168.1.129
> > ------------------------------------------------------------
> > Client connecting to 192.168.1.129, TCP port 5001
> > TCP window size: 96.2 KByte (default)
> > ------------------------------------------------------------
> > [  3] local 192.168.1.12 port 54898 connected with 192.168.1.129 port 5001
> > [ ID] Interval       Transfer     Bandwidth
> > [  3]  0.0- 1.0 sec  11.2 MBytes  94.4 Mbits/sec
> > [  3]  1.0- 2.0 sec  11.0 MBytes  92.3 Mbits/sec
> > [  3]  2.0- 3.0 sec  10.8 MBytes  90.2 Mbits/sec
> > [  3]  3.0- 4.0 sec  11.0 MBytes  92.3 Mbits/sec
> > [  3]  4.0- 5.0 sec  10.9 MBytes  91.2 Mbits/sec
> > [  3]  5.0- 6.0 sec  10.9 MBytes  91.2 Mbits/sec
> > [  3]  6.0- 7.0 sec  10.8 MBytes  90.2 Mbits/sec
> > [  3]  7.0- 8.0 sec  10.9 MBytes  91.2 Mbits/sec
> > [  3]  8.0- 9.0 sec  10.9 MBytes  91.2 Mbits/sec
> > [  3]  9.0-10.0 sec  10.9 MBytes  91.2 Mbits/sec
> > [  3]  0.0-10.0 sec   109 MBytes  91.4 Mbits/sec
> >
> > We were able to bisect this down to this commit:
> >
> > first bad commit: [65466904b015f6eeb9225b51aeb29b01a1d4b59c] tcp: adjust
> > TSO packet sizes based on min_rtt
> >
> > Disabling this new setting via:
> >
> > echo 0 > /proc/sys/net/ipv4/tcp_tso_rtt_log
> >
> > confirm that this was the cause of the performance regression.
> >
> > Is it expected that the new default setting has such a performance impact?

Indeed, thanks for the report.

In addition to the "ss" output Eric mentioned, could you please grab
"nstat" output, which should allow us to calculate the average TSO/GSO
and LRO/GRO burst sizes, which is the key thing tuned with the
tcp_tso_rtt_log knob.

So it would be great to have the following from both data sender and
data receiver, for both the good case and bad case, if you could start
these before your test and kill them after the test stops:

(while true; do date; ss -tenmoi; sleep 1; done) > /root/ss.txt &
nstat -n; (while true; do date; nstat; sleep 1; done)  > /root/nstat.txt

Thanks!
neal