netdev - Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210726164325.GE22278@shell.armlinux.org.uk>
Date:   Mon, 26 Jul 2021 17:43:25 +0100
From:   "Russell King (Oracle)" <linux@...linux.org.uk>
To:     Marek Behún <kabel@...nel.org>
Cc:     netdev@...r.kernel.org
Subject: Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

On Mon, Jul 26, 2021 at 05:52:23PM +0200, Marek Behún wrote:
> On Fri, 23 Jul 2021 09:05:38 +0100
> "Russell King (Oracle)" <linux@...linux.org.uk> wrote:
> 
> > On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:
> > > Hello Russell (and possibly others),
> > > 
> > > I discovered that with mvpp2 when switching from gmac (sgmii or
> > > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > > change, the link won't come up unless I do
> > >   ifconfig ethX down
> > >   ifconfig ethX up
> > > 
> > > Can be reproduced on MacchiatoBIN:
> > > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > > 2. bring the interfaces up
> > > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> > >    10gbase-r
> > > 4. use ethtool on one of the interfaces to advertise modes only up to
> > >    2500base-t
> > > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> > >    go up in 2500base-x
> > > 6. use ethtool on the same interface as in step 4 to advertise all
> > >    supported modes
> > > 
> > > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> > >    !!!
> > > 8. execute
> > >      ifconfig ethX down ; ifconfig ethX up
> > >    on both interfaces. After this, the MACs will successfully link in
> > >    10gbase-r with the PHYs
> > > 
> > > It seems that the mvpp2 driver code needs to make additional stuff when
> > > chaning mode from gmac to xlg. I haven't yet been able to find out
> > > what, though.
> > > 
> > > BTW I discovered this because I am working on adding support for
> > > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > > side.
> > > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > > 	sgmii		to	5gbase-r
> > > 	sgmii		to	10gbase-r
> > > 	2500base-x	to	5gbase-r
> > > 	2500base-x	to	10gbase-r
> > > but also when switching from xlg to different xlg:
> > > 	5gbase-r	to	10gbase-r
> > > 	10gbase-r	to	5gbase-r
> > > 
> > > Did someone notice this bug? I see that Russell made some changes in
> > > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > > other changes). I haven't tried yet if the switch from gmac to xlg
> > > worked at some time in the past. But if it did, maybe these changes
> > > could be the cause?  
> > 
> > What are the PHY leds doing when you encounter this bug?
> > 
> 
> Table summary:
> 
> 			PHY0/eth0	PHY1/eth1
> 			green	yellow	green	yellow
> after boot		ON	OFF	ON	OFF
> eth0 up			ON	OFF	ON	OFF
> eth1 up			blink	ON	blink	ON
> eth0 adv -10g -5g	blink	OFF	blink	OFF
> eth0 adv +5g *	 	OFF	OFF	OFF	OFF
> eth0 down		ON	OFF	ON	OFF
> eth0 up			blink**	OFF	blink** OFF
> eth1 down		ON	OFF	ON	OFF
> eth1 up			blink	OFF	blink	OFF
> 
>  (*  PHYs are linked now, but MACs are not)
>  (** blinks only for a while after link, pings do not work,
>      read my opinion below)
>  (The last 5 lines basically the same happens if I set it to advertise
>   10g instead of 5g, but in case of 10g the yellow LED is ON when the
>   PHYs are linked.)
> 
> In words:
> 
> After boot, the green LED is ON on both PHYs.
> 
> Bringing both interfaces up changes nothing.
> 
> Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
> ON, and the green LEDs OFF, but the green LEDs blinks on activity.
> (For example when pinging eth0's ipv6 address via eth1.)
> 
> Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
> PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
> activity.
> 
> Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
> do not link with the PHYs, and there is no blinking on activity, and
> pings do not work.
> 
> Taking one interface (eth0) down and up makes the PHYs link (we are
> still at 5gbase-t), and the green LEDs blink for a few times because of
> activity on both PHYs. But the pings do not work. I think this is
> because the eth0's PHY sent some neighbour discovery packets, and the
> eth1's PHY received them. But pings do not work because those packets
> don't go from eth1's PHY to eth1's MAC.
> 
> Taking the second interface (eth1) down and up makes the PHYs again
> link at 5gbase-t, and the green LEDs start blinking on activity. Pings
> work.

Hmm, the reason I asked was because I know the 88x3310 with older
firmware can get in a bit of a muddle when switching from SGMII back
to 10G mode. When it does, one of the link LEDs (green I think) blinks
rapidly while it is muddled, and iirc taking the interface down and
back up fixes it. There is no workaround for this other than upgrading
the PHY firmware.

If they only blink momentarily after establishing link, then that is
likely just the IPv6 router solicitation packets.

Hope that helps.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!