netdev - Re: network stream fairness

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 9 Nov 2015 16:53:02 +0100
From:	Niklas Cassel <niklas.cassel@...s.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	"edumazet@...gle.com" <edumazet@...gle.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: network stream fairness

On 11/09/2015 04:50 PM, Eric Dumazet wrote:
> On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
>> I have a ethernet driver for a 100 Mbps NIC.
>> The NIC has dedicated hardware for offloading.
>> The driver has implemented TSO, GSO and BQL.
>> Since the CPU on the SoC is rather weak, I'd rather
>> not increase the CPU load by turning off offloading.
>>
>> Since commit
>> 605ad7f184b6 ("tcp: refine TSO autosizing")
>>
>> the bandwidth is no longer fair between streams.
>> see output at the end of the mail, where I'm testing with 2 streams.
>>
>>
>> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
>>
>> I can also use vanilla 4.3 and do:
>> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
>> to also get a stable 45 Mbps per stream.
>>
>> My question is, am I supposed to set the BQL limit explicitly?
>> It is possible that I have missed something in my driver,
>> but my understanding is that the TCP stack sets and adjusts
>> the BQL limit automatically.
>>
>>
>> Perhaps the following info might help:
>>
>> After running iperf3 on vanilla 4.3:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 89908
>> limit_max 1879048192
>>
>> After running iperf3 on vanilla 4.3 + BQL explicitly set:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 3000
>> limit_max 3000
>>
>> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
>> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
>> limit 8886
>> limit_max 1879048192
>>
> 
> There is absolutely nothing ensuring fairness among multiple TCP flows.
> 
> One TCP flow can very easily grab whole bandwidth for itself, there are
> numerous descriptions of this phenomena in various TCP studies. 
> 
> This is why we have packet schedulers ;)

Oh.. How stupid of me, I forgot to mention.. all of the measurements were
done with fq_codel.

> 
> tc qdisc replace dev eth0 root fq
> 
> This will probably help : No need to change BQL settings, so that you
> keep minimal latencies for the interactive traffic (like ping)
> 
> 
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html