Message-ID: <20220724173916.lebsvaqveqwus4xj@skbuf>
Date:   Sun, 24 Jul 2022 20:39:16 +0300
From:   Vladimir Oltean <olteanv@...il.com>
To:     Marek Behún <kabel@...nel.org>
Cc:     "Russell King (Oracle)" <linux@...linux.org.uk>,
        Andrew Lunn <andrew@...n.ch>,
        Heiner Kallweit <hkallweit1@...il.com>,
        Alexandre Belloni <alexandre.belloni@...tlin.com>,
        Alvin __ipraga <alsi@...g-olufsen.dk>,
        Andy Shevchenko <andriy.shevchenko@...ux.intel.com>,
        Claudiu Manoil <claudiu.manoil@....com>,
        Daniel Scally <djrscally@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        DENG Qingfang <dqfext@...il.com>,
        Eric Dumazet <edumazet@...gle.com>,
        Florian Fainelli <f.fainelli@...il.com>,
        George McCollister <george.mccollister@...il.com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Hauke Mehrtens <hauke@...ke-m.de>,
        Heikki Krogerus <heikki.krogerus@...ux.intel.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Kurt Kanzenbach <kurt@...utronix.de>,
        Landen Chao <Landen.Chao@...iatek.com>,
        Linus Walleij <linus.walleij@...aro.org>,
        linux-acpi@...r.kernel.org, linux-arm-kernel@...ts.infradead.org,
        linux-mediatek@...ts.infradead.org,
        Matthias Brugger <matthias.bgg@...il.com>,
        netdev@...r.kernel.org, Paolo Abeni <pabeni@...hat.com>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        Sakari Ailus <sakari.ailus@...ux.intel.com>,
        Sean Wang <sean.wang@...iatek.com>,
        UNGLinuxDriver@...rochip.com,
        Vivien Didelot <vivien.didelot@...il.com>,
        Woojung Huh <woojung.huh@...rochip.com>
Subject: Re: [PATCH net-next 3/6] net: dsa: add support for retrieving the
 interface mode

On Sat, Jul 23, 2022 at 07:26:55PM +0200, Marek Behún wrote:
> Does Lynx PCS support 1000base-x with AN?

Yes, that would be the intention.

> Because if so, it may be possible to somehow hack working AN for
> 2500base-x, as I managed it for 88E6393X in the commit I mentioned (by
> configuring 1000base-x and then hacking the PHY speed to 2.5x).

I would need to try and see. For Lynx, to dynamically change from
1000base-x to 2500base-x essentially means to move the SERDES lane from
a PLL that can provide the 1.25 GHz required for 1000base-x to a PLL
that can provide the 3.125 GHz required for 2500base-x. The procedure
itself doesn't involve resetting the PCS, but to be honest with you,
I don't know whether the state of the PCS registers is going to be
preserved across the PLL change. Maybe it isn't, but this is entirely
masked out by the phylink major reconfig process, I don't know.

The alternative to dynamic reconfiguration is to program some bits that
instruct the SoC what to do on power-on reset, and these bits include
the initial SERDES protocols and PLL assignments too. I only tried to
experiment with in-band autoneg in this mode (with the lane being
configured for 2.5G out of reset, rather than dynamically switching it
to 2.5G).

> Anyway, I am now looking at the standards, and it seems that all the X
> and R have K variant: 1000base-kx, 2500base-kx, 5gbase-kr and
> 10gbase-kr. These modes have mandatory clause 73 autonegotiation.

The X in BASE-X stands for 8b/10b coding, the R stands for 64b/66b coding.
Whereas the K stands for bacKplane, i.e. the medium (compare this with
the T in BASE-T, for twisted pair copper cable). Or with 1000BASE-SX and
1000BASE-LX, the S stands for Short wavelength laser and the L for Long
wavelength.

What I'm trying to say, the 'X' in BASE-X doesn't stand for anything
having to do with fiber, I guess 1000BASE-X is just a generic name for
the coding scheme (PCS level) rather than something about the medium
(PMD level). The terminology is pretty much a mess.

> So either we need to add these as different modes of the
> phy_interface_t type, or we need to differentiate whether clause 37 or
> clause 73 AN should be used by another property.
> 
> But since 1000base-x supports clause 37 and 1000base-kx clause 73, the
> one property that we have, managed="in-band-status" is not enough, if
> we keep calling both modes '1000base-x'.
> 
> So maybe we really need to add K variants as separate
> PHY_INTERFACE_MODED_ constants. That way we can keep assuming clause 37
> for 2500base-x, and try to implement it for as much drivers as
> possible, by hacking it up...

Well, for good or bad, 10GBase-KR does have its own phy-mode string,
and Sean Anderson is sending a patch to add 1000base-KX now too.
https://patchwork.kernel.org/project/netdevbpf/patch/20220719235002.1944800-3-sean.anderson@seco.com/
(I still don't understand what that has to do with the topic of his
series, but anyway)

More at the end.

> 
> And I still don't understand this clause 73 AN at all. For example, if
> one PHY supports only up to 2.5g speeds, will it complete AN with
> another PHY that supports up to 10g speeds, if the second PHY will
> (maybe?) try at higher frequency?

Define what you mean by "one PHY supports only up to 2.5G speeds".
My copy of IEEE 802.3-2018 doesn't list in Table 73–4—Technology Ability
Field encoding any signaling mode that is capable of 2.5G, but rather
1000BASE-KX, 10GBASE-KR, 25GBASE-KR and so on. So you'd have to express
your question in terms of bits that are actually advertised through the
Technology Ability field.

Then, clause 73 AN, very much like the clause 28/40 AN of BASE-T (to
which it is most directly comparable) has a priority resolution function,
meaning that if 2 link partners advertise support for multiple
technologies, Table 73–5—Priority Resolution will decide which one of
the commonly advertised technologies gets used.

Side note: contrast this with flow control, which annoyingly was
designed by IEEE to not have a priority resolution, in other words you
don't get a graceful falloff of the resolved pause modes depending on
what you and the link partner advertised, instead you need to
preconfigure both ends if you want to achieve a particular result;
this is IMO as useless as not having AN at all.

There is of course no guarantee that two backplane link partners will
have any technology ability in common, for example one may advertise
only 1000Base-KX and the other only 10GBase-KR. In that case, autoneg
will complete, but the link will simply not come up.

The clause 73 autoneg signaling takes place using a predetermined, low-speed
encoding. The medium transitions to the highest negotiated technology,
and performs clause 74 link training on that medium, only after both
ends agree that clause 73 autoneg has completed. This kind of implies
that they will agree on the frequency being used for the data traffic.

If you're asking whether 2 backplane devices will advertise 10GBase-KR
but one of them supports a data rate of only up to 2.5Gbps over that 10G
link, I think this is vendor-dependent and IEEE doesn't say anything
about it. For example this is where rate adaptation could come into
play, either through flow control, or there could be an extension to
clause 73 similar to what Cisco did with USXGMII, where the lane
operates at 10GBaud but via symbol replication your data rate can
actually be only 2.5Gbps. I'm not aware of real life applications of
rate adaptation over backplane links.


I hinted earlier that clause 73 autoneg is most directly comparable to
BASE-T autoneg (these 2 are even situated at different layers if you
look at the IEEE OSI stack pictures, compared to where clause 37 AN is).
The problem is that the Linux kernel support for new physical technologies
grew organically, and we don't have a structure in place that scales
naturally to all the places in which these technologies may appear in
the stack. For example we have the phy-mode, and this represents the

...

/goes searching for the documentation, I don't want to be making this up/

...

  phy-connection-type:
    description:
      Specifies interface type between the Ethernet device and a physical
      layer (PHY) device.

There you go, pretty vague. What's the Ethernet device, and what's the
PHY device?

For example SGMII connects a MAC to a PHY, but to speak SGMII to reach
to your PHY, you need another PHY that does the parallel GMII to serial
translation for you. So to say that the phy-mode is SGMII, you need to
ignore that the MAC has a PHY too.

10GBase-KR is similar in a way, it can be placed at multiple layers, and
traditionally, where you put it makes a difference to how we describe it
in Linux.

Maybe you have a 10GBase-T PHY chip with a backplane host-side PHY, it
supports clause 73 declaring the 10GBase-KR technology, then it supports
clause 74 link training, the whole shebang. These things exist. How would
you describe this? You'd say the phy-mode is "10gbase-kr", according to
precedent. Would that be the best thing to do, in the spirit of clause 73?
I don't think it would. Essentially what would need to happen as a
consequence of this description is that your PCS would essentially
populate its Technology Ability with a single bit, corresponding to what
you put in phy-mode, because that's how we shoehorned this. Then we'd
say what, that managed = "in-band-status" decides whether to bypass
clause 73 AN or not? I don't think so.

Truth is, a 10G-KR "PCS" (what we mean when we say a PHY integrated into a MAC)
is much more similar to a dedicated 10G-KR PHY, to the point that it's
indistinguishable (what Linux thinks of a phy_device is actually 2 PHYs
back to back, one for the host side and one for the medium side), and it
*needs* to be treated by Linux in the same way regardless of where it's
placed. You *need* to be able to control the backplane PCS' advertisement,
whether to use FEC or not, regardless if it's your medium facing device,
or an in-between device.

The discussion is much, much bigger than this, but in summary, I think
it would be quite short-sighted to expand managed = "in-band-status" for
anything related to clause 73, or for much more than what it means right
now (the problem is, what _does_ it mean and what _doesn't_ it?).

This, plus I think development needs to be driven by someone with real
world needs and a sense for what's practical. I am quite well outside of
the sphere of 10-gig-and-higher networking, I'm just looking from the
peanut gallery, so that won't be me.

Powered by blists - more mailing lists