netdev - RE: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit time")

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <6DE8042992C3D3488ACEF3F2607097E3C4A82805@nice.asicdesigners.com>
Date:	Fri, 24 Jun 2016 11:38:01 +0000
From:	"Arjun V." <arjun@...lsio.com>
To:	Eric Dumazet <edumazet@...gle.com>
CC:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	Hariprasad S <hariprasad@...lsio.com>,
	Casey Leedom <leedom@...lsio.com>,
	Kumar A S <kumaras@...lsio.com>,
	Santosh Rastapur <santosh@...lsio.com>,
	"Nirranjan Kirubaharan" <nirranjan@...lsio.com>,
	"davem@...emloft.net" <davem@...emloft.net>,
	"ycheng@...gle.com" <ycheng@...gle.com>
Subject: RE: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after
 commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit
 time")

Eric,
We are seeing skb's with length(skb->len) greater than 65536 coming into our ndo_start_xmit() callback routine. 
We can add a check in our eth_xmit() routine to skip those packets, but it will be better if you fix this in kernel.


I have attached pcap file obtained from tcpdump. In the pcap file there are 2 such packets(I used tcpdump filter to extract out those packets).

Let us know if you need anything else.

Thanks,
Arjun.

-----Original Message-----
From: Eric Dumazet [mailto:edumazet@...gle.com] 
Sent: Thursday, June 23, 2016 5:21 PM
To: Arjun V.
Cc: netdev@...r.kernel.org; Hariprasad S; Casey Leedom; Kumar A S; Santosh Rastapur; Nirranjan Kirubaharan; davem@...emloft.net; ycheng@...gle.com
Subject: Re: [REGRESSION, bisect]cxgb4 port failure with TSO traffic after commit 10d3be569243def8("tcp-tso: do not split TSO packets at retransmit time")

On Thu, Jun 23, 2016 at 3:08 AM, Arjun V. <arjun@...lsio.com> wrote:
> Hi all,
>
> The following patch introduced a regression in Chelsio cxgb4 driver, causing port failure when running heavy TSO traffic:
>
> commit 10d3be569243def8d92ac3722395ef5a59c504e6
> Author: Eric Dumazet <edumazet@...gle.com>
> Date:   Thu Apr 21 10:55:23 2016 -0700
>
>     tcp-tso: do not split TSO packets at retransmit time
>
>     Linux TCP stack painfully segments all TSO/GSO packets before retransmits.
>
>     This was fine back in the days when TSO/GSO were emerging, with their
>     bugs, but we believe the dark age is over.
>
>     Keeping big packets in write queues, but also in stack traversal
>     has a lot of benefits.
>      - Less memory overhead, because write queues have less skbs
>      - Less cpu overhead at ACK processing.
>      - Better SACK processing, as lot of studies mentioned how
>        awful linux was at this ;)
>      - Less cpu overhead to send the rtx packets
>        (IP stack traversal, netfilter traversal, drivers...)
>      - Better latencies in presence of losses.
>      - Smaller spikes in fq like packet schedulers, as retransmits
>        are not constrained by TCP Small Queues.
>
>     1 % packet losses are common today, and at 100Gbit speeds, this
>     translates to ~80,000 losses per second.
>     Losses are often correlated, and we see many retransmit events
>     leading to 1-MSS train of packets, at the time hosts are already
>     under stress.
>
>     Signed-off-by: Eric Dumazet <edumazet@...gle.com>
>     Acked-by: Yuchung Cheng <ycheng@...gle.com>
>     Signed-off-by: David S. Miller davem@...emloft.net
>
> When the number of TCP retransmissions are quite high, the packet length coming from stack does not seems to be proper, due to which our TSO module gets stuck.
> If I change segs back to 1 in __tcp_retransmit_skb(),  traffic is running fine. Please let us know if we are missing something.
>
> Thanks,
> Arjun.
>

Hmm... I see nothing wrong in TCP stack.

Can you give me more details on the wrong packet length you see ?

Download attachment "tcpdump.pcap" of type "application/octet-stream" (131116 bytes)