[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <b0bbc989-d6c7-4a3e-aa31-a63116542348@gmx.net>
Date: Mon, 16 Oct 2023 20:21:04 +0200
From: Stefan Wahren <wahrenst@....net>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Neal Cardwell <ncardwell@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
Fabio Estevam <festevam@...il.com>, linux-imx@....com,
Stefan Wahren <stefan.wahren@...rgebyte.com>,
Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org,
Yuchung Cheng <ycheng@...gle.com>
Subject: Re: iperf performance regression since Linux 5.18
Hi Eric,
Am 16.10.23 um 11:49 schrieb Eric Dumazet:
> On Sun, Oct 15, 2023 at 12:23 PM Stefan Wahren <wahrenst@....net> wrote:
>> Hi,
>>
>> Am 15.10.23 um 01:26 schrieb Stefan Wahren:
>>> Hi Eric,
>>>
>>> Am 15.10.23 um 00:51 schrieb Eric Dumazet:
>>>> On Sat, Oct 14, 2023 at 9:40 PM Neal Cardwell <ncardwell@...gle.com>
>>>> wrote:
>> ...
>>>> Hmm, we receive ~3200 acks per second, I am not sure the
>>>> tcp_tso_should_defer() logic
>>>> would hurt ?
>>>>
>>>> Also the ss binary on the client seems very old, or its output has
>>>> been mangled perhaps ?
>>> this binary is from Yocto kirkstone:
>>>
>>> # ss --version
>>> ss utility, iproute2-5.17.0
>>>
>>> This shouldn't be too old. Maybe some missing kernel settings?
>>>
>> i think i was able to fix the issue by enable the proper kernel
>> settings. I rerun initial bad and good case again and overwrote the log
>> files:
>>
>> https://github.com/lategoodbye/tcp_tso_rtt_log_regress/commit/93615c94ba1bf36bd47cc2b91dd44a3f58c601bc
> Excellent, thanks.
>
> I see your kernel uses HZ=100, have you tried HZ=1000 by any chance ?
>
> CONFIG_HZ_1000=y
> CONFIG_HZ=1000
i tried, but it doesn't have any influence.
> I see that the bad run seems to be stuck for a while with cwnd=66, but
> a smaller amount of packets in flight (26 in following ss extract)
>
> ESTAB 0 315664 192.168.1.12:60542 192.168.1.129:5001
> timer:(on,030ms,0) ino:13011 sk:2 <->
> skmem:(r0,rb131072,t48488,tb295680,f3696,w319888,o0,bl0,d0) ts sack
> cubic wscale:7,6 rto:210 rtt:3.418/1.117 mss:1448 pmtu:1500 rcvmss:536
> advmss:1448 cwnd:66 ssthresh:20 bytes_sent:43874400
> bytes_acked:43836753 segs_out:30302 segs_in:14110 data_segs_out:30300
> send 223681685bps lastsnd:10 lastrcv:4310 pacing_rate 268408200bps
> delivery_rate 46336000bps delivered:30275 busy:4310ms unacked:26
> rcv_space:14480 rcv_ssthresh:64088 notsent:278016 minrtt:0.744
>
> I wonder if fec pseudo-tso code is adding some kind of artifacts,
> maybe with TCP small queue logic.
> (TX completion might be delayed too much on fec driver side)
>
> Can you try
>
> ethtool -K eth0 tso off ?
TSO off, CONFIG_HZ_100=y
root@...ragon:~# iperf -t 10 -i 1 -c 192.168.1.129
------------------------------------------------------------
Client connecting to 192.168.1.129, TCP port 5001
TCP window size: 122 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.12 port 52326 connected with 192.168.1.129 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 11.6 MBytes 97.5 Mbits/sec
[ 3] 1.0- 2.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 2.0- 3.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 3.0- 4.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 4.0- 5.0 sec 11.2 MBytes 94.4 Mbits/sec
[ 3] 5.0- 6.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 6.0- 7.0 sec 11.4 MBytes 95.4 Mbits/sec
[ 3] 7.0- 8.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 8.0- 9.0 sec 11.5 MBytes 96.5 Mbits/sec
[ 3] 9.0-10.0 sec 11.1 MBytes 93.3 Mbits/sec
[ 3] 0.0-10.0 sec 113 MBytes 94.5 Mbits/sec
The figures seems slightly better than tcp_tso_rtt_log = 0 -> Good
> Alternatively I think I mentioned earlier that you could try to reduce
> gso_max_size on a 100Mbit link
>
> ip link set dev eth0 gso_max_size 16384
TSO on, gso_max_size 16384, CONFIG_HZ_100=y
root@...ragon:~# iperf -t 10 -i 1 -c 192.168.1.129
------------------------------------------------------------
Client connecting to 192.168.1.129, TCP port 5001
TCP window size: 101 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.12 port 54548 connected with 192.168.1.129 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 11.4 MBytes 95.4 Mbits/sec
[ 3] 1.0- 2.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 2.0- 3.0 sec 10.9 MBytes 91.2 Mbits/sec
[ 3] 3.0- 4.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 4.0- 5.0 sec 10.6 MBytes 89.1 Mbits/sec
[ 3] 5.0- 6.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 6.0- 7.0 sec 10.9 MBytes 91.2 Mbits/sec
[ 3] 7.0- 8.0 sec 11.0 MBytes 92.3 Mbits/sec
[ 3] 8.0- 9.0 sec 10.8 MBytes 90.2 Mbits/sec
[ 3] 9.0-10.0 sec 10.9 MBytes 91.2 Mbits/sec
[ 3] 0.0-10.0 sec 109 MBytes 91.7 Mbits/sec
The figures are similiar to tcp_tso_rtt_log = 0 -> Good
Powered by blists - more mailing lists