netdev - Re: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit time")

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iLpBC5tUqpdSE4jCMKFgUi=mg=2kPZtMaRh-cnBhvauNg@mail.gmail.com>
Date:	Thu, 23 Jun 2016 04:51:10 -0700
From:	Eric Dumazet <edumazet@...gle.com>
To:	"Arjun V." <arjun@...lsio.com>
Cc:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	Hariprasad S <hariprasad@...lsio.com>,
	Casey Leedom <leedom@...lsio.com>,
	Kumar A S <kumaras@...lsio.com>,
	Santosh Rastapur <santosh@...lsio.com>,
	Nirranjan Kirubaharan <nirranjan@...lsio.com>,
	"davem@...emloft.net" <davem@...emloft.net>,
	"ycheng@...gle.com" <ycheng@...gle.com>
Subject: Re: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after
 commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit time")

On Thu, Jun 23, 2016 at 3:08 AM, Arjun V. <arjun@...lsio.com> wrote:
> Hi all,
>
> The following patch introduced a regression in Chelsio cxgb4 driver, causing port failure when running heavy TSO traffic:
>
> commit 10d3be569243def8d92ac3722395ef5a59c504e6
> Author: Eric Dumazet <edumazet@...gle.com>
> Date:   Thu Apr 21 10:55:23 2016 -0700
>
>     tcp-tso: do not split TSO packets at retransmit time
>
>     Linux TCP stack painfully segments all TSO/GSO packets before retransmits.
>
>     This was fine back in the days when TSO/GSO were emerging, with their
>     bugs, but we believe the dark age is over.
>
>     Keeping big packets in write queues, but also in stack traversal
>     has a lot of benefits.
>      - Less memory overhead, because write queues have less skbs
>      - Less cpu overhead at ACK processing.
>      - Better SACK processing, as lot of studies mentioned how
>        awful linux was at this ;)
>      - Less cpu overhead to send the rtx packets
>        (IP stack traversal, netfilter traversal, drivers...)
>      - Better latencies in presence of losses.
>      - Smaller spikes in fq like packet schedulers, as retransmits
>        are not constrained by TCP Small Queues.
>
>     1 % packet losses are common today, and at 100Gbit speeds, this
>     translates to ~80,000 losses per second.
>     Losses are often correlated, and we see many retransmit events
>     leading to 1-MSS train of packets, at the time hosts are already
>     under stress.
>
>     Signed-off-by: Eric Dumazet <edumazet@...gle.com>
>     Acked-by: Yuchung Cheng <ycheng@...gle.com>
>     Signed-off-by: David S. Miller davem@...emloft.net
>
> When the number of TCP retransmissions are quite high, the packet length coming from stack does not seems to be proper, due to which our TSO module gets stuck.
> If I change segs back to 1 in __tcp_retransmit_skb(),  traffic is running fine. Please let us know if we are missing something.
>
> Thanks,
> Arjun.
>

Hmm... I see nothing wrong in TCP stack.

Can you give me more details on the wrong packet length you see ?