Date:   Wed, 26 Jan 2022 18:09:02 +0000
From:   Robert Hancock <robert.hancock@...ian.com>
To:     "temnota.am@...il.com" <temnota.am@...il.com>
CC:     "linux@...linux.org.uk" <linux@...linux.org.uk>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "hkallweit1@...il.com" <hkallweit1@...il.com>,
        "mail@...id-bauer.net" <mail@...id-bauer.net>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "andrew@...n.ch" <andrew@...n.ch>,
        "kuba@...nel.org" <kuba@...nel.org>
Subject: Re: [PATCH net-next v4 0/3] at803x fiber/SFP support

On Wed, 2022-01-26 at 16:56 +0300, Andrey Jr. Melnikov wrote:
> On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote:
> > Add support for 1000Base-X fiber modes to the at803x PHY driver, as
> > well as support for connecting a downstream SFP cage.
> > 
> > Changes since v3:
> > -Renamed some constants with OHM suffix for clarity
> > 
> > Changes since v2:
> > -fixed tabs/spaces issue in one patch
> > 
> > Changes since v1:
> > -moved page selection to config_init so it is handled properly
> > after suspend/resume
> > -added explicit check for empty sfp_support bitmask
> > 
> > Robert Hancock (3):
> >   net: phy: at803x: move page selection fix to config_init
> >   net: phy: at803x: add fiber support
> >   net: phy: at803x: Support downstream SFP cage
> > 
> Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP
> hardware. Optical SFP modules working without problems, but cooper SFP with
> Marvell 88E1111 not work - link is up but no packets sent/recieved via
> interface.

Interesting, apparently we weren't the only ones weird enough to use an AR8031
for an SFP module..

> 
> # dmesg | grep -iE '(dsa|eth|mdio|sfp)'
> [    0.000000] MIPS: machine is Ubiquiti EdgeRouter X SFP
> [    1.349162] gpio-7 (sfp_i2c_clk_gate): hogged as output/high
> [    4.685535] libphy: Fixed MDIO Bus: probed
> [    4.732207] libphy: mdio: probed
> [    4.739151] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
> [    4.768955] mtk_soc_eth 1e100000.ethernet dsa: mediatek frame engine at
> 0xbe100000, irq 20
> [    4.852254] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module
> [    4.973393] mt7530 mdio-bus:00 eth0 (uninitialized): PHY [mt7530-0:00]
> driver [Generic PHY] (irq=POLL)
> [    4.993676] mt7530 mdio-bus:00 eth1 (uninitialized): PHY [mt7530-0:01]
> driver [Generic PHY] (irq=POLL)
> [    5.014054] mt7530 mdio-bus:00 eth2 (uninitialized): PHY [mt7530-0:02]
> driver [Generic PHY] (irq=POLL)
> [    5.034405] mt7530 mdio-bus:00 eth3 (uninitialized): PHY [mt7530-0:03]
> driver [Generic PHY] (irq=POLL)
> [    5.054704] mt7530 mdio-bus:00 eth4 (uninitialized): PHY [mt7530-0:04]
> driver [Generic PHY] (irq=POLL)
> [    5.131850] mt7530 mdio-bus:00 eth5 (uninitialized): PHY [mdio-bus:07]
> driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
> [    5.155828] mt7530 mdio-bus:00: configuring for fixed/rgmii link mode
> [    5.172290] DSA: tree 0 setup
> [    5.179302] mt7530 mdio-bus:00: Link is Up - 1Gbps/Full - flow control off
> [    9.103291] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii
> link mode
> [    9.119230] device dsa entered promiscuous mode
> [    9.128654] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full -
> flow control rx/tx
> [    9.131563] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
> [    9.158926] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
> [   13.306590] device dsa left promiscuous mode
> [   14.587953] libphy: SFP I2C Bus: probed
> [   14.751420] sfp sfp_eth5: Host maximum power 1.0W
> [   14.821296] sfp sfp_eth5: No tx_disable pin: SFP modules will always be
> emitting.
> [   15.691261] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev
> A    sn X201604162293    dc 160422  
> [   15.710686] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not
> function if 1000Base-X not supported
> [   36.440072] mtk_soc_eth 1e100000.ethernet dsa: Link is Down
> [   36.461374] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii
> link mode
> [   36.477856] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full -
> flow control rx/tx
> [   36.495142] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready
> [   36.508246] device dsa entered promiscuous mode
> [   36.517976] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode
> [   36.532794] br-lan: port 1(eth1) entered blocking state
> [   36.543322] br-lan: port 1(eth1) entered disabled state
> [   36.555768] device eth1 entered promiscuous mode
> [   36.592332] mt7530 mdio-bus:00 eth2: configuring for phy/gmii link mode
> [   36.607113] br-lan: port 2(eth2) entered blocking state
> [   36.617651] br-lan: port 2(eth2) entered disabled state
> [   36.630268] device eth2 entered promiscuous mode
> [   36.655233] mt7530 mdio-bus:00 eth3: configuring for phy/gmii link mode
> [   36.669957] br-lan: port 3(eth3) entered blocking state
> [   36.680454] br-lan: port 3(eth3) entered disabled state
> [   36.693351] device eth3 entered promiscuous mode
> [   36.718596] mt7530 mdio-bus:00 eth4: configuring for phy/gmii link mode
> [   36.733435] br-lan: port 4(eth4) entered blocking state
> [   36.743977] br-lan: port 4(eth4) entered disabled state
> [   36.756966] device eth4 entered promiscuous mode
> [   36.779983] mt7530 mdio-bus:00 eth5: configuring for phy/rgmii-rxid link
> mode
> [   36.795073] mt7530 mdio-bus:00 eth5: No phy led trigger registered for
> speed(-1)
> [   36.810051] br-lan: port 5(eth5) entered blocking state
> [   36.810277] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow
> control off
> 
> Link is up - but what mode is selected ?? No packets received from
> interface, remote side report 1000/Full speed negotiated, TX counters
> increased.

I think you're running into the inherent limitation of this setup (which the
"module may not function if 1000Base-X not supported" message is trying to tell
you) - the AR8031 only supports 1000Base-X on this interface, not SGMII. Many
(perhaps most) copper SFP modules use or default to SGMII mode because it
allows the link to work at 100 or 10 Mbps speeds as well as 1 Gbps.

The difference between 1000Base-X and SGMII, at least when at 1000 Mbps speeds,
is mostly in the auto-negotiation, so we've found that in many cases you can
get it to work by using ethtool to disable auto-negotiation and forcing 1000
Mbps full duplex mode on the interface. It seems that (at least from what we
have seen) the SFP module side will often decide to give up on auto-negotiation 
when it sees a link and just run at the proper rate - that doesn't generally
affect the separate negotiation that happens on the copper link to whatever is
connected on the other end. Obviously if the link partner connected on the
copper side doesn't support 1 Gbps it won't work.

I'm not entirely sure why it worked after the module was unplugged and plugged
back in however. But my guess it's related to this issue somehow.

> 
> [   36.820606] br-lan: port 5(eth5) entered disabled state
> [   36.849515] device eth5 entered promiscuous mode
> [   36.862142] br-lan: port 5(eth5) entered blocking state
> [   36.872673] br-lan: port 5(eth5) entered forwarding state
> [   37.260595] mt7530 mdio-bus:00 eth0: configuring for phy/gmii link mode
> [   40.362029] mt7530 mdio-bus:00 eth0: Link is Up - 1Gbps/Full - flow
> control off
> [   40.376656] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [  670.201487] mt7530 mdio-bus:00 eth5: Link is Down
> [  670.211423] br-lan: port 5(eth5) entered disabled state
> [  671.241339] mt7530 mdio-bus:00 eth5: No phy led trigger registered for
> speed(-1)
> [  671.256789] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow
> control off
> [  671.272343] br-lan: port 5(eth5) entered blocking state
> [  671.282821] br-lan: port 5(eth5) entered forwarding state
> [  671.471210] sfp sfp_eth5: module removed
> 
> Now, I'm removed copper SFP, and install optilcal SFP:
> 
> [  672.281451] mt7530 mdio-bus:00 eth5: Link is Down
> [  672.290943] br-lan: port 5(eth5) entered disabled state
> [  688.921868] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Full - flow
> control off
> [  688.936512] br-lan: port 5(eth5) entered blocking state
> [  688.946944] br-lan: port 5(eth5) entered forwarding state
> [  689.591192] sfp sfp_eth5: module Gateray          GR-S1-X3120L-D   rev
> A    sn X201602220961    dc 160229  
> 
> Link is up, 1000/Full, packets traverse in both directions. Now, insert back
> copper SFP:
> 
> [  731.561446] mt7530 mdio-bus:00 eth5: Link is Down
> [  731.570947] br-lan: port 5(eth5) entered disabled state
> [  733.841609] sfp sfp_eth5: module removed
> [  751.321576] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow
> control off
> 
> Whoa, link up with 1Gbps speed.
> 
> [  751.336733] br-lan: port 5(eth5) entered blocking state
> [  751.347167] br-lan: port 5(eth5) entered forwarding state
> [  751.961193] sfp sfp_eth5: module Gateray          GR-S1-RJ         rev
> A    sn X201604162293    dc 160422  
> [  751.980667] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not
> function if 1000Base-X not supported
> [  753.401483] mt7530 mdio-bus:00 eth5: Link is Down
> [  753.410984] br-lan: port 5(eth5) entered disabled state
> [  754.441627] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow
> control off
> [  754.457144] br-lan: port 5(eth5) entered blocking state
> [  754.467619] br-lan: port 5(eth5) entered forwarding state
> 
> Now packets traverse in both directions for copper module too.
> 
> Can someone explain - why copper module not work from boot? And how controll
> 88E1111
> inside SFP ?

That's the other issue with this setup with copper modules - you end up with
the AR8031 PHY device, which is basically acting as an RGMII etc. to 1000Base-X 
converter, and the PHY device inside the module which is the one talking to the
outside world. As Russell mentioned, this sort of dual-PHY setup isn't handled
well in Linux right now - in this AR8031 setup, the kernel talks to the SFP
module enough to identify it, but the driver for the module PHY is not probed.
Even if it was, it can't really be hooked up to the network interface because
the network layer only supports binding one PHY to a device, so there's not
really a way to control it.

There's support for MAC drivers which have an integrated or semi-integrated PCS
layer which handles 1000Base-X or SGMII - some of which look like a PHY in that
they support standard PHY registers and/or have an MDIO interface - but that
essentially is handled internally to phylink and the MAC driver and the "PHY"-
ness isn't really exposed to the rest of the kernel. But in this case, the MAC
driver has no real knowledge of the downstream SFP contraption that's been
attached on its PHY interface, and the only PHY the kernel associates with the
interface is the AR8031.

In the AR8031 case however, this isn't really a big issue because the inherent
limitation of only supporting 1000Base-X is more significant than the
limitations the kernel side is imposing..

-- 
Robert Hancock
Senior Hardware Designer, Calian Advanced Technologies
www.calian.com

Powered by blists - more mailing lists