[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ea9578d1-be40-48ab-b9d3-826bb5006756@gmx.net>
Date: Tue, 17 Oct 2023 11:53:13 +0200
From: Stefan Wahren <wahrenst@....net>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Neal Cardwell <ncardwell@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
Fabio Estevam <festevam@...il.com>, linux-imx@....com,
Stefan Wahren <stefan.wahren@...rgebyte.com>,
Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org,
Yuchung Cheng <ycheng@...gle.com>
Subject: Re: iperf performance regression since Linux 5.18
Hi Eric,
Am 16.10.23 um 20:47 schrieb Eric Dumazet:
> On Mon, Oct 16, 2023 at 8:25 PM Stefan Wahren <wahrenst@....net> wrote:
>> Hi Eric,
>>
>> Am 16.10.23 um 12:35 schrieb Eric Dumazet:
>>> On Mon, Oct 16, 2023 at 11:49 AM Eric Dumazet <edumazet@...gle.com> wrote:
>>> Speaking of TSQ, it seems an old change (commit 75eefc6c59fd "tcp:
>>> tsq: add a shortcut in tcp_small_queue_check()")
>>> has been accidentally removed in 2017 (75c119afe14f "tcp: implement
>>> rb-tree based retransmit queue")
>>>
>>> Could you try this fix:
>>>
>>> diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
>>> index 9c8c42c280b7638f0f4d94d68cd2c73e3c6c2bcc..e61a3a381d51b554ec8440928e22a290712f0b6b
>>> 100644
>>> --- a/net/ipv4/tcp_output.c
>>> +++ b/net/ipv4/tcp_output.c
>>> @@ -2542,6 +2542,18 @@ static bool tcp_pacing_check(struct sock *sk)
>>> return true;
>>> }
>>>
>>> +static bool tcp_rtx_queue_empty_or_single_skb(const struct sock *sk)
>>> +{
>>> + const struct rb_node *node = sk->tcp_rtx_queue.rb_node;
>>> +
>>> + /* No skb in the rtx queue. */
>>> + if (!node)
>>> + return true;
>>> +
>>> + /* Only one skb in rtx queue. */
>>> + return !node->rb_left && !node->rb_right;
>>> +}
>>> +
>>> /* TCP Small Queues :
>>> * Control number of packets in qdisc/devices to two packets / or ~1 ms.
>>> * (These limits are doubled for retransmits)
>>> @@ -2579,12 +2591,12 @@ static bool tcp_small_queue_check(struct sock
>>> *sk, const struct sk_buff *skb,
>>> limit += extra_bytes;
>>> }
>>> if (refcount_read(&sk->sk_wmem_alloc) > limit) {
>>> - /* Always send skb if rtx queue is empty.
>>> + /* Always send skb if rtx queue is empty or has one skb.
>>> * No need to wait for TX completion to call us back,
>>> * after softirq/tasklet schedule.
>>> * This helps when TX completions are delayed too much.
>>> */
>>> - if (tcp_rtx_queue_empty(sk))
>>> + if (tcp_rtx_queue_empty_or_single_skb(sk))
>>> return false;
>>>
>>> set_bit(TSQ_THROTTLED, &sk->sk_tsq_flags);
>> This patch applied on top of Linux 6.1.49, TSO on, gso_max_size 65535,
>> CONFIG_HZ_100=y
>>
>> root@...ragon:/boot# iperf -t 10 -i 1 -c 192.168.1.129
>> ------------------------------------------------------------
>> Client connecting to 192.168.1.129, TCP port 5001
>> TCP window size: 192 KByte (default)
>> ------------------------------------------------------------
>> [ 3] local 192.168.1.12 port 59714 connected with 192.168.1.129 port 5001
>> [ ID] Interval Transfer Bandwidth
>> [ 3] 0.0- 1.0 sec 11.5 MBytes 96.5 Mbits/sec
>> [ 3] 1.0- 2.0 sec 11.4 MBytes 95.4 Mbits/sec
>> [ 3] 2.0- 3.0 sec 11.1 MBytes 93.3 Mbits/sec
>> [ 3] 3.0- 4.0 sec 11.2 MBytes 94.4 Mbits/sec
>> [ 3] 4.0- 5.0 sec 11.1 MBytes 93.3 Mbits/sec
>> [ 3] 5.0- 6.0 sec 11.2 MBytes 94.4 Mbits/sec
>> [ 3] 6.0- 7.0 sec 11.2 MBytes 94.4 Mbits/sec
>> [ 3] 7.0- 8.0 sec 11.1 MBytes 93.3 Mbits/sec
>> [ 3] 8.0- 9.0 sec 11.4 MBytes 95.4 Mbits/sec
>> [ 3] 9.0-10.0 sec 11.2 MBytes 94.4 Mbits/sec
>> [ 3] 0.0-10.0 sec 113 MBytes 94.4 Mbits/sec
>>
>> The figures are comparable to disabling TSO -> Good
>>
>> Thanks
> Great. I suspect a very slow TX completion from fec then.
>
> Could you use the following bpftrace program while your iperf is running ?
unfortuntely there is no bpftrace and most of its dependencies on my
platform. I looked at some guides and it seems to have a lot of (build)
dependencies. On a PC this won't be a problem, but on my ARM platform
there is only 1 GB eMMC space left.
Before investing a lot of time to get bpftrace running, is there an
alternative solution?
>
> .bpftrace -e '
> k:__dev_queue_xmit {
> $skb = (struct sk_buff *)arg0;
> if ($skb->fclone == 2) {
> @start[$skb] = nsecs;
> }
> }
> k:__kfree_skb {
> $skb = (struct sk_buff *)arg0;
> if ($skb->fclone == 2 && @start[$skb]) {
> @tx_compl_usecs = hist((nsecs - @start[$skb])/1000);
> delete(@start[$skb]);
> }
> }
> END { clear(@start); }'
Powered by blists - more mailing lists