Date: Tue, 29 Aug 2023 18:37:07 +0100
From: Daniel Golle <daniel@...rotopia.org>
To: "Russell King (Oracle)" <linux@...linux.org.uk>
Cc: Nicolò Veronese <nicveronese@...il.com>,
	netdev@...r.kernel.org, simonebortolin@...k-gpon.org,
	nanomad@...k-gpon.org, Federico Cappon <dududede371@...il.com>,
	lorenzo@...nel.org, ftp21@...21.eu, pierto88@...k-gpon.org,
	hitech95@...k-gpon.org, davem@...emloft.net, andrew@...n.ch,
	edumazet@...gle.com, hkallweit1@...il.com, kuba@...nel.org,
	pabeni@...hat.com, nbd@....name
Subject: Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed
 single-mac devices (XOR RJ/SFP)

On Tue, Aug 29, 2023 at 04:38:42PM +0100, Russell King (Oracle) wrote:
> On Tue, Aug 29, 2023 at 05:12:48PM +0200, Nicolò Veronese wrote:
> > Hi,
> > 
> > I and some folks in CC are working to properly port all the
> >  functions of a Zyxel ex5601-t0 to OpenWrt.
> > 
> > The manufacturer decided to use a single SerDes connected
> >  to both an SPF cage and an RJ45 phy. A simple GPIO is
> >  used to control a 2 Channel 2:1 MUX to switch the two SGMII pairs
> >  between the RJ45 and the SFP.
> > 
> >   ┌─────┐  ┌──────┐   ┌─────────┐
> >   │     │  │      │   │         │
> >   │     │  │      ├───┤ SFP     │
> >   │     │  │      │   └─────────┘
> >   │     │  │      │
> >   │ MAC ├──┤ MUX  │   ┌─────────┐
> >   │     │  │      │   │         │
> >   │     │  │      │   │ RJ45    │
> >   │     │  │      ├───┤ 2.5G PHY│
> >   │     │  │      │   │         │
> >   └─────┘  └───▲──┘   └─────────┘
> >                │
> >   MUX-GPIO ────┘

Note that most recent MediaTek SoCs have a similar setup built-into the
SoC itself: One MAC can either be internally connected to a built-in
2.5G TP PHY or used with an external PHY or SFP via SerDes ie.
USXGMII/10GBase-R/5GBase-R/2500Base-X/1000Base-X/SGMII **which doesn't
share any pins with the TP PHY**. Hence board manufactorers are likely
to build devices exposing that MAC in both ways, as SFP cage and 2.5G
PHY, because the 2.5G PHY basically comes "for free".

In this case switching the MAC between the two is done using a mux bit
in ETHSYS syscon. However, one should also take care of powering on
and off the 2.5G PHY and maintain the trapdoor in the MDIO bus
allowing the access either built-in PHY or an external PHY at the same
address, so it's a bit more complicated than just a single bit.

> 
> This is do-able in software, but is far from a good idea.
> 
> Yes, it would be possible to "disconnect" the RJ45 PHY from the netdev,
> and switch to the SFP and back again. It would be relatively easy for
> phylink to do that. What phylink would need to do is to keep track of
> the SFP PHY and netdev-native PHY independently, and multiplex between
> the two. It would also have to manage the netdev->phydev pointer.
> Any changes to this must be done under the rtnl lock.
> 
> So technically it's possible. However, there is no notification to
> userspace when such a change may occur. There's also the issue that
> userspace may be in the process of issuing ethtool commands that are
> affecting one of the PHYs. While holding the rtnl lock will block
> those calls, a change between the PHY and e.g. a PHY on the SFP
> would cause the ethtool command to target a different PHY from what
> was the original target.

I can see that the lack of such notification is already an issue even
without the change described above. I've actually struggled with that
just a few days ago:
Some of the SFP+ modules I use for testing expose the built-in
marvell10g PHY via I2C using the RollBall protocol. The protocol
apparently requires a long waiting time between power-on and being
able to access the PHY (something like 25 seconds). So any ethtool
command issued after boot and before the 25 seconds have passed will
have no effect as of today, because the PHY is only being attached
after that. And this is not only a problem when hot-plugging the
module, obviously, but also when having it plugged in already
during boot.

So having userspace notification "some about the PHY has changed"
would already be nice to have, because that'd would an easy way
OpenWrt's netifd could know that it has to re-read supported
capabilities and re-apply ethtool link settings.

> 
> To solve that sanely, every PHY-based ethtool probably needs a way
> to specify which PHY the command is intended for, but then there's
> the question of how userspace users react to that - because it's
> likely more than just modifying the ethtool utility, ethtool
> commands are probably used from many programs.

Maybe just a cookie to make sure "sessions" of ethtool calls are
not broken by a changed from PHY to SFP or vice versa?
Ie. GLINKSETTINGS will should give you a cookie to be used with
SLINKSETTINGS and so one. The cookie may not identify the PHY, but
rather just the previous call to GLINKSETTINGS.

And this could even be optional from perspective of userspace.

> 
> IMHO, it needs a bit of thought beyond "what can we do to support a
> mux".
> 

Another thing which came to my mind is the existing port field in
many ethtool ops which could be either PORT_TP or PORT_FIBRE to
destinguish the TP PHY from the SFP at least for xLINKSETTINGS.

Powered by blists - more mailing lists