Date:   Fri, 29 Jan 2021 16:45:59 +0000
From:   Vadim Pasternak <vadimp@...dia.com>
To:     Andrew Lunn <andrew@...n.ch>, Jiri Pirko <jiri@...nulli.us>
CC:     David Ahern <dsahern@...il.com>, Jakub Kicinski <kuba@...nel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "jacob.e.keller@...el.com" <jacob.e.keller@...el.com>,
        Roopa Prabhu <roopa@...dia.com>, mlxsw <mlxsw@...dia.com>
Subject: RE: [patch net-next RFC 00/10] introduce line card support for
 modular switch



> -----Original Message-----
> From: Andrew Lunn <andrew@...n.ch>
> Sent: Friday, January 29, 2021 5:50 PM
> To: Jiri Pirko <jiri@...nulli.us>
> Cc: David Ahern <dsahern@...il.com>; Jakub Kicinski <kuba@...nel.org>;
> netdev@...r.kernel.org; davem@...emloft.net; jacob.e.keller@...el.com;
> Roopa Prabhu <roopa@...dia.com>; mlxsw <mlxsw@...dia.com>; Vadim
> Pasternak <vadimp@...dia.com>
> Subject: Re: [patch net-next RFC 00/10] introduce line card support for
> modular switch
> 
> On Fri, Jan 29, 2021 at 08:20:15AM +0100, Jiri Pirko wrote:
> > Thu, Jan 28, 2021 at 03:17:13PM CET, andrew@...n.ch wrote:
> > >On Thu, Jan 28, 2021 at 09:14:34AM +0100, Jiri Pirko wrote:
> > >> Wed, Jan 27, 2021 at 03:14:34PM CET, andrew@...n.ch wrote:
> > >> >> >There are Linux standard APIs for controlling the power to
> > >> >> >devices, the regulator API. So i assume mlxreg-pm will make use
> > >> >> >of that. There are also standard APIs for thermal management,
> > >> >> >which again, mlxreg-pm should be using. The regulator API
> > >> >> >allows you to find regulators by name. So just define a
> > >> >> >sensible naming convention, and the switch driver can lookup the
> regulator, and turn it on/off as needed.
> > >> >>
> > >> >>
> > >> >> I don't think it would apply. The thing is, i2c driver has a
> > >> >> channel to the linecard eeprom, from where it can read info
> > >> >> about the linecard. The i2c driver also knows when the linecard is
> plugged in, unlike mlxsw.
> > >> >> It acts as a standalone driver. Mlxsw has no way to directly
> > >> >> find if the card was plugged in (unpowered) and which type it is.
> > >> >>
> > >> >> Not sure how to "embed" it. I don't think any existing API could help.
> > >> >> Basicall mlxsw would have to register a callback to the i2c
> > >> >> driver called every time card is inserted to do auto-provision.
> > >> >> Now consider a case when there are multiple instances of the
> > >> >> ASIC on the system. How to assemble a relationship between mlxsw
> > >> >> instance and i2c driver instance?
> > >> >
> > >> >You have that knowledge already, otherwise you cannot solve this
> > >>
> > >> No I don't have it. I'm not sure why do you say so. The mlxsw and
> > >> i2c driver act independently.
> > >
> > >Ah, so you just export some information in /sys from the i2c driver?
> > >And you expect the poor user to look at the values, and copy paste
> > >them to the correct mlxsw instance? 50/50 guess if you have two
> > >switches, and hope they don't make a typO?
> >
> > Which values are you talking about here exactly?
> 
> The i2c driver tells you what line card is actually inserted.
> Hopefully it interprets the EEPROM and gives the user a nice string. You then
> need to use this string to provision the switch, so it knows what line card has
> been inserted. Or the user can pre-prevision, make a guess as to what card will
> actually be inserted sometime in the future, tell the switch, and hope that
> actually happens.

Hi Andrew,

mlxsw I2C driver is BMC side driver. Its purpose to provide hwmon,
thermal, QSFP info for the chassis management at BMC side.
It works on top of PRM interface and it is associated with the chip I2C
slave device.
It doesn't aware of system topology, it knows nothing about system I2C
tree, what is EEPROM, where it located and so on. This is not a scope of
this driver.

Platform line card driver is aware of line card I2C topology, its
responsibility is to detect line card basic hardware type, create I2C
topology (mux), connect all the necessary I2C devices, like hotswap
devices, voltage and power regulators devices, iio/a2d devices and line
card EEPROMs, creates LED instances for LED located on a line card, exposes
line card related attributes, like CPLD and FPGA versions, reset causes,
required powered through line card hwmon interface.

> 
> > >> >I still don't actually get this use case. Why would i want to
> > >> >manually provision?
> > >>
> > >> Because user might want to see the system with all netdevices,
> > >> configure them, change the linecard if they got broken and all
> > >> config, like bridge, tc, etc will stay on the netdevices. Again,
> > >> this is the same we do for split port. This is important
> > >> requirement, user don't want to see netdevices come and go when he
> > >> is plugging/unplugging cables. Linecards are the same in this
> > >> matter. Basically is is a "splitter module", replacing the "splitter cable"
> > >
> > >So, what is the real use case here? Why might the user want to do
> > >this?
> > >
> > >Is it: The magic smoke has escaped. The user takes a spare switch,
> > >and wants to put it on her desk to configure it where she has a comfy
> > >chair and piece and quiet, unlike in the data centre, which is very
> > >noise, only has hard plastic chair, no coffee allowed. She makes her
> > >best guess at the configuration, up/downs the interfaces, reboots, to
> > >make sure it is permanent, and only then moves to the data centre to
> > >swap the dead router for the new one, and fix up whatever
> > >configuration errors there are, while sat on the hard chair?
> > >
> > >So this feature is about comfy chair vs hard chair?
> >
> > I don't really get the question, but configuring switch w/o any
> > linecard and plug the linecards in later on is definitelly a usecase.
> 
> It is a requirement, not a use case.
> 
> A use case is the big picture, what is the user doing, at the big picture level. In
> the somewhat absurd example given above, the user case is, the router chassis
> has died, but they think the line cards are O.K. They want to do as much
> configuration and testing as possible before going into the data center to
> actually replace the chassis.  By reusing the existing line cards, they reduce the
> risk of getting the cables plugged into the wrong port.
> 
> From the use case, you can derive the requirements. In order to test that ifup --
> all puts the IP addresses in the right places, it needs to have the netdevs with
> the correct names. Either they need line cards in the router, or they need to be
> able to fake line cards. There is also a requirement that the line cards are easy
> to exchange. They do not need to be fully hot-plugable, since one router is
> dead, the other can be powered off. But ideally you want simple thumb
> screws, not a Philips screwdriver or an allan key. There is also a requirement
> that this provision is persistent, since the user is likely to reboot the system in
> order to test the configurations files actually work at boot time. Either the
> switch driver needs to write the information to FLASH, or user space needs to
> tell it on every boot, a systemd service file or similar.
> 
> But since you have not been able to answer my question, i wonder if
> everything is backwards around here. Your architecture is broken, you cannot
> easily determine what line card is inserted, so you need a workaround,
> provisioning. But provision might actually be a useful feature, so lets try to sell
> the feature, and gloss over that the architecture is broken.
> 
> So, i would like to see the architecture fixed first. The switch driver somehow
> talks to the i2c driver to find out what card is in the slot, and configures itself.
> My guess is, every other switch does this, this is what the user expects as a
> base feature, it is what we want Linux to do by default.
> 
> You can later add provisioning, where if the slot is empty, you can fake a line
> card, to fulfil the use cases. And when the slot is actually filled, you can verify
> what is plugged in matches what was expected, and be very noise if not.
> 
> 	  Andrew

Powered by blists - more mailing lists