Message-ID: <CAA93jw7fTUPx+ywKE437OKbgr-d+Exz9Q1izOiQg1yN_HEedXw@mail.gmail.com>
Date:   Tue, 1 May 2018 12:22:30 -0700
From:   Dave Taht <dave.taht@...il.com>
To:     Toke Høiland-Jørgensen <toke@...e.dk>
Cc:     Eric Dumazet <eric.dumazet@...il.com>,
        Cong Wang <xiyou.wangcong@...il.com>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>,
        Cake List <cake@...ts.bufferbloat.net>
Subject: Re: [PATCH net-next v6] Add Common Applications Kept Enhanced (cake) qdisc

On Tue, May 1, 2018 at 11:53 AM, Toke Høiland-Jørgensen <toke@...e.dk> wrote:
> Eric Dumazet <eric.dumazet@...il.com> writes:
>
>> On 04/30/2018 02:27 PM, Dave Taht wrote:
>>
>>> I actually have a tc - bpf based ack filter, during the development of
>>> cake's ack-thinner, that I should submit one of these days. It
>>> proved to be of limited use.
>>>
>>> Probably the biggest mistake we made is by calling this cake feature a
>>> filter. It isn't.
>>>
>>> Maybe we should have called it a "thinner" or something like that? In
>>> order to properly "thin" or "reduce" an ack stream
>>> you have to have a queue to look at and some related state. TC filters
>>> do not operate on queues, qdiscs do. Thus the "ack-filter" here is
>>> deeply embedded into cake's flow isolation and queue structures.
>>
>> A feature eating packets _is_ a filter.
>>
>> Given that a qdisc only sees one direction, I really do not get why
>> ack-filter is so desirable in a packet scheduler.
>
> The ACK filter in CAKE is there to solve a very particular use case:
> Residential internet connections with bandwidths so asymmetric that it
> hurts TCP performance. It is not on by default; but rather meant to be
> configured by users which suffer from this particular ISP brokenness
> (which, sadly, does happen; there are ISPs who believe a 50/1 bandwidth
> ratio is reasonable). For those users, the ACK filter can help.
>
> We certainly do not advise people to turn it on in the general case! As
> you point, in general TCP performance is best improved in the TCP stack...
>
>> You have not provided any numbers to show how useful it is to maintain
>> this code
>
> You mean apart from that in the linked blog post and the paper?
> Here's an executive summary:
>
> On a 30/1 Mbps connection with a bidirectional traffic test (RRUL),
> turning on the ACK filter improves downstream throughput by ~20% and
> upstream throughput by ~12% in conservative mode and ~40% in aggressive
> mode, at the cost of ~5ms of inter-flow latency due to the increased
> congestion.

On a simulated 50/1 comcast connection, I got double the throughput
on a similar test, with no obvious glitches in the tcp flow, in the early stages
of development.

http://blog.cerowrt.org/post/ack_filtering/

I was very, very dubious about the value of ack thinning until that point,
but it was hard to argue with the data.

> In *really* pathological cases, the effect can be a lot more; for
> instance, the ACK filter increases the achievable downstream throughput
> on a link with 100 Kbps in the upstream direction by an order of
> magnitude (from ~2.5 Mbps to ~25 Mbps).
>
>> (probably still broken btw, considering it is changing some skb
>> attributes).
>
> As you may have noticed over the last few iterations, I've actually been
> trying to fix any brokenness. If you could be specific as to what is
> still broken, that would be helpful.
>
> (I'm assuming are referring to the calls to skb_set_inner* - but do you
> mean all of them, or just the ones that set inner == outer?)
>
>> On wifi (or any half duplex medium), you might gain something by
>> carefully sending ACK not too often, but ultimately this should be
>> done by TCP stack, in well controlled environment [1], instead of
>> middle-boxes happily playing/breaking some Internet standards.
>>
>> [1] TCP stack has the estimations of RTT, RWIN, throughput, and might
>> be able to avoid flooding the network with acks under some conditions.
>
> You are quite right that in general, TCP performance is best improved in
> the TCP stack. But home users are not generally in control of that; the
> ACK filter helps in those specific deployments (again, it's off by
> default, and you shouldn't turn it on in the general case!).
>
>> Say RTT is 100ms, and we receive 1 packet every 100 usec (no GRO
>> aggregation) Maybe we do not really _need_ to send 5000 ack per second
>> (or even 10,000 ack per second if a hole needs a repair)
>
> Yes, please do fix that.

:) I really would like to see cake tested at 10GigE and above, and its
performance improved overall. I tend to think we need more queues than 1024
at 40GigE+, and we presently run out of cpu (even unshaped) long
before we hit that point.

>
>> Also on wifi, the queue builds in the driver queues anyway, not in the
>> qdisc.
>
> Actually, we've fixed that (for some drivers); there is now no qdisc on
> WiFi interfaces, and we've integrated FQ-CoDel'ed queueing into the
> stack where it can be effective. But no, running CAKE with an ACK filter
> on a WiFi link is not going to be effective. Don't do that.

I share the belief with eric that thinning acks (either at the wifi qdisc or
in tcp) on wifi interfaces will help - given that the underlying wifi layer
is reliable and does retransmits, and the number of packets that can
fit into a wifi aggregate limited,
you really only need one ack per wifi aggregate per flow to keep the
tcp connection running.

That said, I'd much rather see fq_codel work with more wifi drivers
than pursue this.

>
>> So ACK filtering, if _really_ successful, would need to be
>> modularized.

Heh. I keep hoping ISPs will ship symmetric links and wifi antennas

> I really hope ACK filtering won't be "_really_ successful". Again, it is
> not meant to be a feature that's on everywhere, it's targeting a very
> specific use case that CAKE is optimised for: The home router use case.

Please note that I find cake far more general purpose than just this, the
ease of just slamming:

tc qdisc add dev eth0 root cake bandwidth 50mbit

on a link that needs it is far easier than the equivalent htb +
fq_codel + other filters, and more effective.

That mode is with nat on, some diffserv awareness (more correct than
pfifo_fast), no link layer compensation, no ack-filter, and "just
works".

Certainly the major use case to date has been on home routers. Every
feature in cake was based
on extensive feedback from that part of the field.

>> Please split Cake into a patch series.
>> Presumably putting the ack-filter on a patch of its own.
>
> *sigh* - can do, I guess. Though I'm not sure what that is going to
> accomplish, exactly?
>
> -Toke



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Powered by blists - more mailing lists