lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <2e554688-f974-465c-a9a1-50f9044abc15@gmx.net>
Date: Tue, 17 Oct 2023 14:17:08 +0200
From: Stefan Wahren <wahrenst@....net>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Neal Cardwell <ncardwell@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Fabio Estevam <festevam@...il.com>, linux-imx@....com,
 Stefan Wahren <stefan.wahren@...rgebyte.com>,
 Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org,
 Yuchung Cheng <ycheng@...gle.com>
Subject: Re: iperf performance regression since Linux 5.18


Am 17.10.23 um 14:08 schrieb Eric Dumazet:
> On Tue, Oct 17, 2023 at 11:53 AM Stefan Wahren <wahrenst@....net> wrote:
>> Hi Eric,
>>
>> Am 16.10.23 um 20:47 schrieb Eric Dumazet:
>>> On Mon, Oct 16, 2023 at 8:25 PM Stefan Wahren <wahrenst@....net> wrote:
>>>> Hi Eric,
>>>>
>>>> Am 16.10.23 um 12:35 schrieb Eric Dumazet:
>>>>> On Mon, Oct 16, 2023 at 11:49 AM Eric Dumazet <edumazet@...gle.com> wrote:
>>>>> Speaking of TSQ, it seems an old change (commit 75eefc6c59fd "tcp:
>>>>> tsq: add a shortcut in tcp_small_queue_check()")
>>>>> has been accidentally removed in 2017 (75c119afe14f "tcp: implement
>>>>> rb-tree based retransmit queue")
>>>>>
>>>>> Could you try this fix:
>>>>>
>>>>> diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
>>>>> index 9c8c42c280b7638f0f4d94d68cd2c73e3c6c2bcc..e61a3a381d51b554ec8440928e22a290712f0b6b
>>>>> 100644
>>>>> --- a/net/ipv4/tcp_output.c
>>>>> +++ b/net/ipv4/tcp_output.c
>>>>> @@ -2542,6 +2542,18 @@ static bool tcp_pacing_check(struct sock *sk)
>>>>>            return true;
>>>>>     }
>>>>>
>>>>> +static bool tcp_rtx_queue_empty_or_single_skb(const struct sock *sk)
>>>>> +{
>>>>> +       const struct rb_node *node = sk->tcp_rtx_queue.rb_node;
>>>>> +
>>>>> +       /* No skb in the rtx queue. */
>>>>> +       if (!node)
>>>>> +               return true;
>>>>> +
>>>>> +       /* Only one skb in rtx queue. */
>>>>> +       return !node->rb_left && !node->rb_right;
>>>>> +}
>>>>> +
>>>>>     /* TCP Small Queues :
>>>>>      * Control number of packets in qdisc/devices to two packets / or ~1 ms.
>>>>>      * (These limits are doubled for retransmits)
>>>>> @@ -2579,12 +2591,12 @@ static bool tcp_small_queue_check(struct sock
>>>>> *sk, const struct sk_buff *skb,
>>>>>                    limit += extra_bytes;
>>>>>            }
>>>>>            if (refcount_read(&sk->sk_wmem_alloc) > limit) {
>>>>> -               /* Always send skb if rtx queue is empty.
>>>>> +               /* Always send skb if rtx queue is empty or has one skb.
>>>>>                     * No need to wait for TX completion to call us back,
>>>>>                     * after softirq/tasklet schedule.
>>>>>                     * This helps when TX completions are delayed too much.
>>>>>                     */
>>>>> -               if (tcp_rtx_queue_empty(sk))
>>>>> +               if (tcp_rtx_queue_empty_or_single_skb(sk))
>>>>>                            return false;
>>>>>
>>>>>                    set_bit(TSQ_THROTTLED, &sk->sk_tsq_flags);
>>>> This patch applied on top of Linux 6.1.49, TSO on, gso_max_size 65535,
>>>> CONFIG_HZ_100=y
>>>>
>>>> root@...ragon:/boot# iperf -t 10 -i 1 -c 192.168.1.129
>>>> ------------------------------------------------------------
>>>> Client connecting to 192.168.1.129, TCP port 5001
>>>> TCP window size:  192 KByte (default)
>>>> ------------------------------------------------------------
>>>> [  3] local 192.168.1.12 port 59714 connected with 192.168.1.129 port 5001
>>>> [ ID] Interval       Transfer     Bandwidth
>>>> [  3]  0.0- 1.0 sec  11.5 MBytes  96.5 Mbits/sec
>>>> [  3]  1.0- 2.0 sec  11.4 MBytes  95.4 Mbits/sec
>>>> [  3]  2.0- 3.0 sec  11.1 MBytes  93.3 Mbits/sec
>>>> [  3]  3.0- 4.0 sec  11.2 MBytes  94.4 Mbits/sec
>>>> [  3]  4.0- 5.0 sec  11.1 MBytes  93.3 Mbits/sec
>>>> [  3]  5.0- 6.0 sec  11.2 MBytes  94.4 Mbits/sec
>>>> [  3]  6.0- 7.0 sec  11.2 MBytes  94.4 Mbits/sec
>>>> [  3]  7.0- 8.0 sec  11.1 MBytes  93.3 Mbits/sec
>>>> [  3]  8.0- 9.0 sec  11.4 MBytes  95.4 Mbits/sec
>>>> [  3]  9.0-10.0 sec  11.2 MBytes  94.4 Mbits/sec
>>>> [  3]  0.0-10.0 sec   113 MBytes  94.4 Mbits/sec
>>>>
>>>> The figures are comparable to disabling TSO -> Good
>>>>
>>>> Thanks
>>> Great. I suspect a very slow TX completion from fec then.
>>>
>>> Could you use the following bpftrace program while your iperf is running ?
>> unfortuntely there is no bpftrace and most of its dependencies on my
>> platform. I looked at some guides and it seems to have a lot of (build)
>> dependencies. On a PC this won't be a problem, but on my ARM platform
>> there is only 1 GB eMMC space left.
>>
>> Before investing a lot of time to get bpftrace running, is there an
>> alternative solution?
> No worries Stefan, I think we do not have to get precise numbers, I
> will try to provide a debug patch (since you are able to build custom
> kernels)

Great, this would be much easier for me.

>
> In the meantime, I will submit the official TCP patch, as you said it
> was helping a lot.
>
> Just to confirm, have you tried the patch on top of the latest net tree ?

Not yet (only Linux 6.1.49 stable), but i can do this today and come
back to you.

>
> Thanks.
>
>>> .bpftrace -e '
>>> k:__dev_queue_xmit {
>>>    $skb = (struct sk_buff *)arg0;
>>>    if ($skb->fclone == 2) {
>>>     @start[$skb] = nsecs;
>>>    }
>>> }
>>> k:__kfree_skb {
>>>    $skb = (struct sk_buff *)arg0;
>>>    if ($skb->fclone == 2 && @start[$skb]) {
>>>     @tx_compl_usecs = hist((nsecs - @start[$skb])/1000);
>>>     delete(@start[$skb]);
>>>    }
>>> }
>>> END { clear(@start); }'


Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ