Message-ID: <b5d9f1ff-0b0f-4c97-9d1c-4ba4468ce6e3@lunn.ch>
Date: Wed, 12 Jun 2024 00:03:00 +0200
From: Andrew Lunn <andrew@...n.ch>
To: Siddharth Vadapalli <s-vadapalli@...com>
Cc: davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org,
	pabeni@...hat.com, corbet@....net, rogerq@...nel.org,
	danishanwar@...com, vladimir.oltean@....com, netdev@...r.kernel.org,
	linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
	linux-arm-kernel@...ts.infradead.org, vigneshr@...com,
	misael.lopez@...com, srk@...com
Subject: Re: [RFC PATCH net-next 01/28] docs: networking: ti: add driver doc
 for CPSW Proxy Client

> System Architecture and Implementation Details
> ==============================================
> 
> The CPSW Ethernet Switch has a single Host Port (CPU facing port) through
> which it can receive data from the Host(s) and transmit data to the
> Host(s).

So there is a single host port, but it can support multiple hosts,
each having a subset of the available DMA channels. Maybe it is
explain later, but why call it a _single_ host port? Apart from the
DMA channels, are there other things the hosts are sharing?

> The exchange of data occurs via TX/RX DMA Channels (Hardware
> Queues). These Hardware Queues are a limited resource (8 TX Channels and
> up to 64 RX Flows). If the Operating System on any of the cores is the
> sole user of CPSW then all of these Hardware Queues can be claimed by that
> OS. However, when CPSW has to be shared across the Operating Systems on
> various cores with the aim of enabling Ethernet Functionality for the
> Applications running on different cores, it is necessary to share these
> Hardware Queues in a manner that prevents conflicts. On the control path
> which corresponds to the configuration of CPSW to get it up and running,
> since there is no Integrated Processor within CPSW that can be programmed
> with a startup configuration, either the Operating System or Firmware
> running on one of the cores has to take the responsibility of setting it.
> One option in this case happens to be the Ethernet Switch Firmware (EthFw)
> which is loaded by the Bootloader on a remote core at the same time that
> Linux and other Operating Systems begin booting. EthFw quickly powers on
> and configures CPSW getting the Forwarding Path functional.

At some point, a definition of functional will be needed. How does the
EthFw know what is required? Should Linux care? Can Linux change it?

> Once Linux and
> other Operating Systems on various cores are ready, they can communicate
> with EthFw to obtain details of the Hardware Queues allocated to them to
> exchange data with CPSW.

> With the knowledge of the Hardware Queues that
> have been allocated, Linux can use the DMA APIs to setup these queues
> to exchange data with CPSW.

This might be an important point. You communicate with the CPSW. You
don't communicate transparently through the CPSW to external ports?
There is no mechanism for a host to say, send this packet out port X?
It is the CPSW which decides, based on its address tables? The
destination MAC address decides where a packet goes.

> Setting up the Hardware Queues alone isn't sufficient to exchange data
> with the external network. Consider the following example:
> The ethX interface in userspace which has been created to transmit/receive
> data to/from CPSW has the user-assigned MAC Address of "M". The ping
> command is run with the destination IP of "D". This results in an ARP
> request sent from ethX which is transmitted out of all MAC Ports of CPSW
> since it is a Broadcast request. Assuming that "D" is a valid
> destination IP, the ARP reply is received on one of the MAC Ports which
> is now a Unicast reply with the destination MAC Address of "M". The ALE
> (Address Lookup Engine) in CPSW has learnt that the MAC Address "M"
> corresponds to the Host Port when the ARP request was sent out. So the
> Unicast reply isn't dropped. The challenge however is determining which
> RX DMA Channel (Flow) to send the Unicast reply on. In the case of a
> single Operating System owning all Hardware Queues, sending it on any of
> the RX DMA Channels would have worked. In the current case where the RX
> DMA Channels map to different Hosts (Operating Systems and Applications),
> the mapping between the MAC Address "M" and the RX DMA Channel has to be
> setup to ensure that the correct Host receives the ARP reply. This
> necessitates a method to inform the MAC Address "M" associated with the
> interface ethX to EthFw so that EthFw can setup the MAC Address "M" to
> RX DMA Channel map accordingly.

Why not have EthFW also do learning? The broadcast ARP request tells
you that MAC address M is associated to a TX DMA channel. EthFW should
know the Rx DMA channel which pairs with it, and can program ALE.

That is how a switch works, it learns what MAC address is where, it is
not told.

> At this point, Linux can exchange data with the external network via CPSW,
> but no device on the external network can initiate the communication by
> itself unless it already has the ARP entry for the IP Address of ethX.
> That's because CPSW doesn't support packet replication implying that any
> Broadcast/Multicast packets received on the MAC Ports can only be sent
> on one of the RX DMA Channels.

That sounds broken.

And this is where we need to be very careful. It is hard to build a
generic model when the first device using it is broken. Ethernet
switches have always been able to replicate. Dumb hubs did nothing but
replicate. Address learning, and forwarding out specific ports came
later, but multicast and broadcast was always replicated. IGMP
snooping came later still, which reduced multicast replication.

And your switch cannot do replication....

> So the Broadcast/Multicast packets can
> only be received by one Host. Consider the following example:
> A PC on the network tries to ping the IP Address of ethX. In both of the
> following cases:
> 1. Linux hasn't yet exchanged data with the PC via ethX.
> 2. The MAC Address of ethX has changed.
> the PC sends an ARP request to one of the MAC Ports on CPSW to figure
> out the MAC Address of ethX. Since the ARP request is a Broadcast
> request, it is not possible for CPSW to determine the correct Host,
> since the Broadcast MAC isn't unique to any Host. So CPSW is forced
> to send the Broadcast request to a preconfigured RX DMA Channel which
> in this case happens to be the one mapped to EthFw. Thus, if EthFw
> is aware of the IP Address of ethX, it can generate and send the ARP
> reply containing the MAC Address "M" of ethX that it was informed of.
> With this, the PC can initiate communication with Linux as well.
> 
> Similarly, in the case of Multicast packets, if Linux wishes to receive
> certain Multicast packets, it needs to inform the same to EthFw which
> shall then replicate the Multicast packets it received from CPSW and
> transmit them via alternate means (Shared Memory for example) to Linux.

This all sounds like you are working around broken behaviour, not
something generic.

What i actually think you need to do is hide all the broken
behaviour. Trap all multicast/broadcast to EthFw. It can run a
software bridge, and do learning. It will see the outgoing ARP request
from a host and learn the host MAC address. It can then flood the
packet out the external ports, working around the CSPW brokeness. It
can also program the ALE, so the reply goes straight to the
host. Incoming broadcast and multicast is also trapped to the EthFW
and it can use its software bridge to flood the packet to all the
hosts. It can also perform IGMP snooping, and learn which hosts are
interested in Multicast. 

Your switch then functions as a switch.

And you are then the same as the RealTek and Samsung device. Linux is
just a plain boring host connect to a switch, which somebody else is
managing. No new model needed.

> All data between Linux (Or any Operating System) and EthFw is exchanged
> via the Hardware Mailboxes with the help of the RPMsg framework. Since
> all the resource allocation information comes from EthFw, the
> vendor-specific implementation in the Linux Client is limited to the DMA
> APIs used to setup the Hardware Queues and to transmit/receive data with
> the Ethernet Switch. Therefore, it might be possible to move most of the
> vendor specific implementation to the Switch Configuration Firmware
> (similar to EthFw), to make the Linux Client implementation as generic
> and vendor agnostic as possible. I believe that this series more or less
> does the same, just using custom terminology which can be made generic.

This is actually very similar to what your college is doing:

https://lore.kernel.org/netdev/20240531064006.1223417-1-y-mallik@ti.com/

The only real difference is shared memory vs DMA.

	Andrew

Powered by blists - more mailing lists