Date:   Wed, 13 Nov 2019 19:47:54 +0200
From:   Vladimir Oltean <olteanv@...il.com>
To:     Andrew Lunn <andrew@...n.ch>
Cc:     netdev <netdev@...r.kernel.org>
Subject: Re: Offloading DSA taggers to hardware

Hi Andrew,

On Wed, 13 Nov 2019 at 18:53, Andrew Lunn <andrew@...n.ch> wrote:
>
> Hi Vladimir
>
> I've not seen any hardware that can do this. There is an
> Atheros/Qualcom integrated SoC/Switch where the 'header' is actually
> just a field in the transmit/receive descriptor. There is an out of
> tree driver for it, and the tag driver is very minimal. But clearly
> this only works for integrated systems.
>

What is this Atheros SoC?
It is funny that the topic reminded you of it. Your line of reasoning
probably was: "Atheros pushed this idea so far that they omitted the
DSA frame tag altogether for their own CPU port/DSA master". Which
means that even if they try to use this "offloaded DSA tagger"
abstraction, it would slightly violate the main idea of an offload,
which is the fact that it's optional. What do you think?

> The other 'smart' features i've seen in NICs with respect to DSA is
> being able to do hardware checksums. Freescale FEC for example cannot
> figure out where the IP header is, because of the DSA header, and so
> cannot calculate IP/TCP/UDP checksums. Marvell, and i expect some
> other vendors of both MAC and switch devices, know about these
> headers, and can do checksumming.
>

Of course there are many more benefits that derive from more complete
frame parsing as well, for some reason my mind just stopped at QoS
when I wrote this email.

> I'm not even sure there are any NICs which can do GSO or LRO when
> there is a DSA header involved.
>
> In the direction CPU to switch, i think many of the QoS issues are
> higher up the stack. By the time the tagger is involved, all the queue
> discipline stuff has been done, and it really is time to send the
> frame. In the 'post buffer bloat world', the NICs hardware queue
> should be small, so QoS is not so relevant once you reach the TX
> queue. The real QoS issue i guess is that the slave interfaces have no
> idea they are sharing resources at the lowest level. So a high
> priority frames from slave 1 are not differentiated from best effort
> frames from slave 2. If we were serious about improving QoS, we need a
> meta scheduler across the slaves feeding the master interface in a QoS
> aware way.
>

Qdiscs on the DSA master are a good discussion to be had, but this
wasn't the main thing I wanted to bring up here.

> In the other direction, how much is the NIC really looking at QoS
> information on the receive path? Are you thinking RPS? I'm not sure
> any of the NICs commonly used today with DSA are actually multi-queue
> and do RPS.
>

Actually both DSA master drivers I've been using so far (gianfar,
enetc) register a number of RX queues equal to the number of cores. It
is possible to add ethtool --config-nfc rules to steer certain
priority traffic to its own CPU, but the keys need to be masked
according to where the QoS field in the DSA frame tag overlaps with
what the DSA master thinks it's looking at, aka DMAC, SMAC, EtherType,
etc. It's not pretty.

> Another aspect here might be, what Linux is doing with DSA is probably
> well past the silicon vendors expected use cases. None of the 'vendor
> crap' drivers i've seen for these SOHO class switches have the level
> of integration we have in Linux. We are pushing the limits of the
> host/switch interfaces much more then vendors do, and so silicon
> vendors are not so aware of the limits in these areas? But DSA is
> being successful, vendors are taking more notice of it, and maybe with
> time, the host/switch interface will improve. NICs might start
> supporting GSO/LRO when there is a DSA header involved? Multi-queue
> NICs become more popular in this class of hardware and RPS knows how
> to handle DSA headers. But my guess would be, it will be for a Marvell
> NIC paired with a Marvell Switch, Broadcom NIC paired with a Broadcom
> switch, etc. I doubt there will be cross vendor support.

...Atheros with Atheros... :)

Yes, that's kind of the angle I'm coming from, basically trying to
understand what a correct abstraction from Linux's perspective would
look like, and what is considered too much "tribalism". The DSA model
is attractive even for an integrated system because there is more
modularity in the design, but there are some clear optimizations that
can be made when the master+switch recipe is tightly controlled.

>
>         Andrew

Thanks,
-Vladimir

Powered by blists - more mailing lists