lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <2e554688-f974-465c-a9a1-50f9044abc15@gmx.net> Date: Tue, 17 Oct 2023 14:17:08 +0200 From: Stefan Wahren <wahrenst@....net> To: Eric Dumazet <edumazet@...gle.com> Cc: Neal Cardwell <ncardwell@...gle.com>, Jakub Kicinski <kuba@...nel.org>, Fabio Estevam <festevam@...il.com>, linux-imx@....com, Stefan Wahren <stefan.wahren@...rgebyte.com>, Michael Heimpold <mhei@...mpold.de>, netdev@...r.kernel.org, Yuchung Cheng <ycheng@...gle.com> Subject: Re: iperf performance regression since Linux 5.18 Am 17.10.23 um 14:08 schrieb Eric Dumazet: > On Tue, Oct 17, 2023 at 11:53 AM Stefan Wahren <wahrenst@....net> wrote: >> Hi Eric, >> >> Am 16.10.23 um 20:47 schrieb Eric Dumazet: >>> On Mon, Oct 16, 2023 at 8:25 PM Stefan Wahren <wahrenst@....net> wrote: >>>> Hi Eric, >>>> >>>> Am 16.10.23 um 12:35 schrieb Eric Dumazet: >>>>> On Mon, Oct 16, 2023 at 11:49 AM Eric Dumazet <edumazet@...gle.com> wrote: >>>>> Speaking of TSQ, it seems an old change (commit 75eefc6c59fd "tcp: >>>>> tsq: add a shortcut in tcp_small_queue_check()") >>>>> has been accidentally removed in 2017 (75c119afe14f "tcp: implement >>>>> rb-tree based retransmit queue") >>>>> >>>>> Could you try this fix: >>>>> >>>>> diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c >>>>> index 9c8c42c280b7638f0f4d94d68cd2c73e3c6c2bcc..e61a3a381d51b554ec8440928e22a290712f0b6b >>>>> 100644 >>>>> --- a/net/ipv4/tcp_output.c >>>>> +++ b/net/ipv4/tcp_output.c >>>>> @@ -2542,6 +2542,18 @@ static bool tcp_pacing_check(struct sock *sk) >>>>> return true; >>>>> } >>>>> >>>>> +static bool tcp_rtx_queue_empty_or_single_skb(const struct sock *sk) >>>>> +{ >>>>> + const struct rb_node *node = sk->tcp_rtx_queue.rb_node; >>>>> + >>>>> + /* No skb in the rtx queue. */ >>>>> + if (!node) >>>>> + return true; >>>>> + >>>>> + /* Only one skb in rtx queue. */ >>>>> + return !node->rb_left && !node->rb_right; >>>>> +} >>>>> + >>>>> /* TCP Small Queues : >>>>> * Control number of packets in qdisc/devices to two packets / or ~1 ms. >>>>> * (These limits are doubled for retransmits) >>>>> @@ -2579,12 +2591,12 @@ static bool tcp_small_queue_check(struct sock >>>>> *sk, const struct sk_buff *skb, >>>>> limit += extra_bytes; >>>>> } >>>>> if (refcount_read(&sk->sk_wmem_alloc) > limit) { >>>>> - /* Always send skb if rtx queue is empty. >>>>> + /* Always send skb if rtx queue is empty or has one skb. >>>>> * No need to wait for TX completion to call us back, >>>>> * after softirq/tasklet schedule. >>>>> * This helps when TX completions are delayed too much. >>>>> */ >>>>> - if (tcp_rtx_queue_empty(sk)) >>>>> + if (tcp_rtx_queue_empty_or_single_skb(sk)) >>>>> return false; >>>>> >>>>> set_bit(TSQ_THROTTLED, &sk->sk_tsq_flags); >>>> This patch applied on top of Linux 6.1.49, TSO on, gso_max_size 65535, >>>> CONFIG_HZ_100=y >>>> >>>> root@...ragon:/boot# iperf -t 10 -i 1 -c 192.168.1.129 >>>> ------------------------------------------------------------ >>>> Client connecting to 192.168.1.129, TCP port 5001 >>>> TCP window size: 192 KByte (default) >>>> ------------------------------------------------------------ >>>> [ 3] local 192.168.1.12 port 59714 connected with 192.168.1.129 port 5001 >>>> [ ID] Interval Transfer Bandwidth >>>> [ 3] 0.0- 1.0 sec 11.5 MBytes 96.5 Mbits/sec >>>> [ 3] 1.0- 2.0 sec 11.4 MBytes 95.4 Mbits/sec >>>> [ 3] 2.0- 3.0 sec 11.1 MBytes 93.3 Mbits/sec >>>> [ 3] 3.0- 4.0 sec 11.2 MBytes 94.4 Mbits/sec >>>> [ 3] 4.0- 5.0 sec 11.1 MBytes 93.3 Mbits/sec >>>> [ 3] 5.0- 6.0 sec 11.2 MBytes 94.4 Mbits/sec >>>> [ 3] 6.0- 7.0 sec 11.2 MBytes 94.4 Mbits/sec >>>> [ 3] 7.0- 8.0 sec 11.1 MBytes 93.3 Mbits/sec >>>> [ 3] 8.0- 9.0 sec 11.4 MBytes 95.4 Mbits/sec >>>> [ 3] 9.0-10.0 sec 11.2 MBytes 94.4 Mbits/sec >>>> [ 3] 0.0-10.0 sec 113 MBytes 94.4 Mbits/sec >>>> >>>> The figures are comparable to disabling TSO -> Good >>>> >>>> Thanks >>> Great. I suspect a very slow TX completion from fec then. >>> >>> Could you use the following bpftrace program while your iperf is running ? >> unfortuntely there is no bpftrace and most of its dependencies on my >> platform. I looked at some guides and it seems to have a lot of (build) >> dependencies. On a PC this won't be a problem, but on my ARM platform >> there is only 1 GB eMMC space left. >> >> Before investing a lot of time to get bpftrace running, is there an >> alternative solution? > No worries Stefan, I think we do not have to get precise numbers, I > will try to provide a debug patch (since you are able to build custom > kernels) Great, this would be much easier for me. > > In the meantime, I will submit the official TCP patch, as you said it > was helping a lot. > > Just to confirm, have you tried the patch on top of the latest net tree ? Not yet (only Linux 6.1.49 stable), but i can do this today and come back to you. > > Thanks. > >>> .bpftrace -e ' >>> k:__dev_queue_xmit { >>> $skb = (struct sk_buff *)arg0; >>> if ($skb->fclone == 2) { >>> @start[$skb] = nsecs; >>> } >>> } >>> k:__kfree_skb { >>> $skb = (struct sk_buff *)arg0; >>> if ($skb->fclone == 2 && @start[$skb]) { >>> @tx_compl_usecs = hist((nsecs - @start[$skb])/1000); >>> delete(@start[$skb]); >>> } >>> } >>> END { clear(@start); }'
Powered by blists - more mailing lists