netdev - Re: network stream fairness

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1447085271.17135.44.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Mon, 09 Nov 2015 08:07:51 -0800
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Niklas Cassel <niklas.cassel@...s.com>
Cc:	"edumazet@...gle.com" <edumazet@...gle.com>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: network stream fairness

On Mon, 2015-11-09 at 16:53 +0100, Niklas Cassel wrote:
> On 11/09/2015 04:50 PM, Eric Dumazet wrote:
> > On Mon, 2015-11-09 at 16:41 +0100, Niklas Cassel wrote:
> >> I have a ethernet driver for a 100 Mbps NIC.
> >> The NIC has dedicated hardware for offloading.
> >> The driver has implemented TSO, GSO and BQL.
> >> Since the CPU on the SoC is rather weak, I'd rather
> >> not increase the CPU load by turning off offloading.
> >>
> >> Since commit
> >> 605ad7f184b6 ("tcp: refine TSO autosizing")
> >>
> >> the bandwidth is no longer fair between streams.
> >> see output at the end of the mail, where I'm testing with 2 streams.
> >>
> >>
> >> If I revert 605ad7f184b6 on 4.3, I get a stable 45 Mbps per stream.
> >>
> >> I can also use vanilla 4.3 and do:
> >> echo 3000 > /sys/class/net/eth0/queues/tx-0/byte_queue_limits/limit_max
> >> to also get a stable 45 Mbps per stream.
> >>
> >> My question is, am I supposed to set the BQL limit explicitly?
> >> It is possible that I have missed something in my driver,
> >> but my understanding is that the TCP stack sets and adjusts
> >> the BQL limit automatically.
> >>
> >>
> >> Perhaps the following info might help:
> >>
> >> After running iperf3 on vanilla 4.3:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 89908
> >> limit_max 1879048192
> >>
> >> After running iperf3 on vanilla 4.3 + BQL explicitly set:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 3000
> >> limit_max 3000
> >>
> >> After running iperf3 on 4.3 + 605ad7f184b6 reverted:
> >> /sys/class/net/eth0/queues/tx-0/byte_queue_limits/
> >> limit 8886
> >> limit_max 1879048192
> >>
> > 
> > There is absolutely nothing ensuring fairness among multiple TCP flows.
> > 
> > One TCP flow can very easily grab whole bandwidth for itself, there are
> > numerous descriptions of this phenomena in various TCP studies. 
> > 
> > This is why we have packet schedulers ;)
> 
> Oh.. How stupid of me, I forgot to mention.. all of the measurements were
> done with fq_codel.

Your numbers suggest a cwnd growth then, which might show a CC bug.

Please run the following when your iper3 runs on regular 4.3 kernel

for i in `seq 1 10`
do
ss -temoi dst 192.168.0.141
sleep 1
done


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html