netdev - Re: [net-next PATCH 0/2] net: phylink: Fix issue w/ BMC link flap

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <08b79b2c-8078-4180-9b74-7cd03b2b06f7@lunn.ch>
Date: Tue, 22 Apr 2025 18:49:54 +0200
From: Andrew Lunn <andrew@...n.ch>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Alexander Duyck <alexander.duyck@...il.com>, netdev@...r.kernel.org,
	linux@...linux.org.uk, hkallweit1@...il.com, davem@...emloft.net,
	pabeni@...hat.com
Subject: Re: [net-next PATCH 0/2] net: phylink: Fix issue w/ BMC link flap

> > The whole concept of a multi-host NIC is new to me. So i at least need
> > to get up to speed with it. I've no idea if Russell has come across it
> > before, since it is not a SoC concept.
> > 
> > I don't really want to agree to anything until i do have that concept
> > understood. That is part of why i asked about a standard. It is a
> > dense document answering a lot of questions. Without a standard, i
> > need to ask a lot of questions.
> 
> Don't hesitate to ask the questions, your last reply contains no
> question marks :)

O.K. Lets start with the basics. I assume the NIC has a PCIe connector
something like a 4.0 x4? Each of the four hosts in the system
contribute one PCIe lane. So from the host side it looks like a 4.0 x1
NIC?

There are not 4 host MACs connected to a 5 port switch. Rather, each
host gets its own subset of queues, DMA engines etc, for one shared
MAC. Below the MAC you have all the usual PCS, SFP cage, gpios, I2C
bus, and blinky LEDs. Plus you have the BMC connected via an RMII like
interface.

You must have a minimum of firmware on the NIC to get the MAC into a
state the BMC can inject/receive frames, configure the PCS, gpios to
the SFP, enough I2C to figure out what the module is, what quirks are
needed etc.

NC-SI, with Linux controlling the hardware, implies you need to be
able to hand off control of the GPIOs, I2C, PCS to Linux. But with
multi-host, it makes no sense for all 4 hosts to be trying to control
the GPIOs, I2C, PCS, perform SFP firmware upgrade. So it seems more
likely to me, one host gets put in change of everything below the
queues to the MAC. The others just know there is link, nothing more.

This actually circles back to the discussion about fixed-link. The one
host in control of all the lower hardware has the complete
picture. The other 3 maybe just need a fixed link. They don't get to
see what is going on below the MAC, and as a result there is no
ethtool support to change anything, and so no conflicting
configuration? And since they cannot control any of that, they cannot
put the link down. So 3/4 of the problem is solved.

phylink is however not expecting that when phylink_start() is called,
it might or might not have to drive the hardware depending on if it
wins an election to control the hardware. And if it losses, it needs
to ditch all its configuration for a PCS, SPF, etc and swap to a
fixed-link. Do we want to teach phylink all this, or put all phylink
stuff into open(), rather than spread across probe() and open(). Being
in open(), you basically construct a different phylink configuration
depending on if you win the election or not.

Is one host in the position to control the complete media
configuration? Could you split the QSFP into four, each host gets its
own channel, and it gets to choose how to use that channel, different
FEC schemes, bit rates?

	Andrew