netdev - Re: Slow ramp-up for single-stream TCP throughput on 4.2 kernel.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CADVnQynKmhrpD63sF5aWPYo9f8L3wOMuEyACBnTdkzQQ=D1_uA@mail.gmail.com>
Date:	Sat, 3 Oct 2015 21:20:38 -0400
From:	Neal Cardwell <ncardwell@...gle.com>
To:	Ben Greear <greearb@...delatech.com>
Cc:	netdev <netdev@...r.kernel.org>,
	ath10k <ath10k@...ts.infradead.org>,
	Eric Dumazet <edumazet@...gle.com>
Subject: Re: Slow ramp-up for single-stream TCP throughput on 4.2 kernel.

On Sat, Oct 3, 2015 at 6:46 PM, Ben Greear <greearb@...delatech.com> wrote:
>
>
> On 10/03/2015 09:29 AM, Neal Cardwell wrote:
>>
>> On Fri, Oct 2, 2015 at 8:21 PM, Ben Greear <greearb@...delatech.com>
>> wrote:
>>>
>>> Gah, seems 'cubic' related.  That is the default tcp cong ctrl
>>> I was using (same in 3.17, for that matter).
>>
>>
>> There have been recent changes to CUBIC that may account for this. If
>> you could repeat your test with more instrumentation, eg "nstat", that
>> would be very helpful.
>>
>> nstat > /dev/null
>> # run one test
>> nstat
>>
>> Also, if you could take a sender-side tcpdump trace of the test, that
>> would be very useful (default capture length, grabbing just headers,
>> is fine).
>
>
> Here is nstat output:
>
> [root@...-ota-1 ~]# nstat
> #kernel
> IpInReceives                    14507              0.0
> IpInDelivers                    14507              0.0
> IpOutRequests                   49531              0.0
> TcpActiveOpens                  3                  0.0
> TcpPassiveOpens                 2                  0.0
> TcpInSegs                       14498              0.0
> TcpOutSegs                      50269              0.0
> UdpInDatagrams                  9                  0.0
> UdpOutDatagrams                 1                  0.0
> TcpExtDelayedACKs               43                 0.0
> TcpExtDelayedACKLost            5                  0.0
> TcpExtTCPHPHits                 483                0.0
> TcpExtTCPPureAcks               918                0.0
> TcpExtTCPHPAcks                 12758              0.0
> TcpExtTCPDSACKOldSent           5                  0.0
> TcpExtTCPRcvCoalesce            49                 0.0
> TcpExtTCPAutoCorking            3                  0.0
> TcpExtTCPOrigDataSent           49776              0.0
> TcpExtTCPHystartTrainDetect     1                  0.0
> TcpExtTCPHystartTrainCwnd       16                 0.0
> IpExtInBcastPkts                8                  0.0
> IpExtInOctets                   2934274            0.0
> IpExtOutOctets                  74817312           0.0
> IpExtInBcastOctets              640                0.0
> IpExtInNoECTPkts                14911              0.0
> [root@...-ota-1 ~]#
>
>
> And, you can find the pcap here:
>
> http://www.candelatech.com/downloads/cubic.pcap.bz2
>
> Let me know if you need anything else.

Thanks! This is very useful. It looks like the sender is sending 3
(and later 4) packets every ~1.5ms for the entirety of the trace. 3
packets per burst is usually a hint that this may be related to TSQ.

This slow-and-steady behavior triggers CUBIC's Hystart Train Detection
to enter congestion avoidance at a cwnd of 16, which probably in turn
leads to slow cwnd growth, since the sending is not cwnd-limited, but
probably TSQ-limited, so cwnd does not grow in congestion avoidance
mode. Probably most of the other congestion control modules do better
because they stay in slow-start, which has a more aggressive criterion
for growing cwnd.

So this is probably at root due to the known issue with an interaction
between the ath10k driver and the following change in 3.19:

  605ad7f tcp: refine TSO autosizing

There has been a lot of discussion about how to address the
TSQ-related issues with this driver. For example, you might consider:

  https://patchwork.ozlabs.org/patch/438322/

But I am not sure of the latest status of that effort. Perhaps someone
on the ath10k list will know.

neal
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html