Date:   Mon, 26 Jul 2021 19:03:16 +0200
From:   Marek Behún <kabel@...nel.org>
To:     "Russell King (Oracle)" <linux@...linux.org.uk>
Cc:     netdev@...r.kernel.org
Subject: Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

On Mon, 26 Jul 2021 17:43:25 +0100
"Russell King (Oracle)" <linux@...linux.org.uk> wrote:

> On Mon, Jul 26, 2021 at 05:52:23PM +0200, Marek Behún wrote:
> > On Fri, 23 Jul 2021 09:05:38 +0100
> > "Russell King (Oracle)" <linux@...linux.org.uk> wrote:
> >   
> > > On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:  
> > > > Hello Russell (and possibly others),
> > > > 
> > > > I discovered that with mvpp2 when switching from gmac (sgmii or
> > > > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > > > change, the link won't come up unless I do
> > > >   ifconfig ethX down
> > > >   ifconfig ethX up
> > > > 
> > > > Can be reproduced on MacchiatoBIN:
> > > > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > > > 2. bring the interfaces up
> > > > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> > > >    10gbase-r
> > > > 4. use ethtool on one of the interfaces to advertise modes only up to
> > > >    2500base-t
> > > > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> > > >    go up in 2500base-x
> > > > 6. use ethtool on the same interface as in step 4 to advertise all
> > > >    supported modes
> > > > 
> > > > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> > > >    !!!
> > > > 8. execute
> > > >      ifconfig ethX down ; ifconfig ethX up
> > > >    on both interfaces. After this, the MACs will successfully link in
> > > >    10gbase-r with the PHYs
> > > > 
> > > > It seems that the mvpp2 driver code needs to make additional stuff when
> > > > chaning mode from gmac to xlg. I haven't yet been able to find out
> > > > what, though.
> > > > 
> > > > BTW I discovered this because I am working on adding support for
> > > > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > > > side.
> > > > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > > > 	sgmii		to	5gbase-r
> > > > 	sgmii		to	10gbase-r
> > > > 	2500base-x	to	5gbase-r
> > > > 	2500base-x	to	10gbase-r
> > > > but also when switching from xlg to different xlg:
> > > > 	5gbase-r	to	10gbase-r
> > > > 	10gbase-r	to	5gbase-r
> > > > 
> > > > Did someone notice this bug? I see that Russell made some changes in
> > > > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > > > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > > > other changes). I haven't tried yet if the switch from gmac to xlg
> > > > worked at some time in the past. But if it did, maybe these changes
> > > > could be the cause?    
> > > 
> > > What are the PHY leds doing when you encounter this bug?
> > >   
> > 
> > Table summary:
> > 
> > 			PHY0/eth0	PHY1/eth1
> > 			green	yellow	green	yellow
> > after boot		ON	OFF	ON	OFF
> > eth0 up			ON	OFF	ON	OFF
> > eth1 up			blink	ON	blink	ON
> > eth0 adv -10g -5g	blink	OFF	blink	OFF
> > eth0 adv +5g *	 	OFF	OFF	OFF	OFF
> > eth0 down		ON	OFF	ON	OFF
> > eth0 up			blink**	OFF	blink** OFF
> > eth1 down		ON	OFF	ON	OFF
> > eth1 up			blink	OFF	blink	OFF
> > 
> >  (*  PHYs are linked now, but MACs are not)
> >  (** blinks only for a while after link, pings do not work,
> >      read my opinion below)
> >  (The last 5 lines basically the same happens if I set it to advertise
> >   10g instead of 5g, but in case of 10g the yellow LED is ON when the
> >   PHYs are linked.)
> > 
> > In words:
> > 
> > After boot, the green LED is ON on both PHYs.
> > 
> > Bringing both interfaces up changes nothing.
> > 
> > Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
> > ON, and the green LEDs OFF, but the green LEDs blinks on activity.
> > (For example when pinging eth0's ipv6 address via eth1.)
> > 
> > Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
> > PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
> > activity.
> > 
> > Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
> > do not link with the PHYs, and there is no blinking on activity, and
> > pings do not work.
> > 
> > Taking one interface (eth0) down and up makes the PHYs link (we are
> > still at 5gbase-t), and the green LEDs blink for a few times because of
> > activity on both PHYs. But the pings do not work. I think this is
> > because the eth0's PHY sent some neighbour discovery packets, and the
> > eth1's PHY received them. But pings do not work because those packets
> > don't go from eth1's PHY to eth1's MAC.
> > 
> > Taking the second interface (eth1) down and up makes the PHYs again
> > link at 5gbase-t, and the green LEDs start blinking on activity. Pings
> > work.  
> 
> Hmm, the reason I asked was because I know the 88x3310 with older
> firmware can get in a bit of a muddle when switching from SGMII back
> to 10G mode. When it does, one of the link LEDs (green I think) blinks
> rapidly while it is muddled, and iirc taking the interface down and
> back up fixes it. There is no workaround for this other than upgrading
> the PHY firmware.
> 
> If they only blink momentarily after establishing link, then that is
> likely just the IPv6 router solicitation packets.
> 
> Hope that helps.
> 

Thanks, I will try different firmwares and let you know. I actually
currently have two different firmware version on the PHYs...

Marek

Powered by blists - more mailing lists