Message-ID: <20240607002536.GI19897@nvidia.com>
Date: Thu, 6 Jun 2024 21:25:36 -0300
From: Jason Gunthorpe <jgg@...dia.com>
To: Dan Williams <dan.j.williams@...el.com>
Cc: Jakub Kicinski <kuba@...nel.org>, David Ahern <dsahern@...nel.org>,
	Jonathan Corbet <corbet@....net>, Itay Avraham <itayavr@...dia.com>,
	Leon Romanovsky <leon@...nel.org>, linux-doc@...r.kernel.org,
	linux-rdma@...r.kernel.org, netdev@...r.kernel.org,
	Paolo Abeni <pabeni@...hat.com>, Saeed Mahameed <saeedm@...dia.com>,
	Tariq Toukan <tariqt@...dia.com>,
	Andy Gospodarek <andrew.gospodarek@...adcom.com>,
	Aron Silverton <aron.silverton@...cle.com>,
	Christoph Hellwig <hch@...radead.org>, Jiri Pirko <jiri@...dia.com>,
	Leonid Bloch <lbloch@...dia.com>,
	Leon Romanovsky <leonro@...dia.com>, linux-cxl@...r.kernel.org,
	patches@...ts.linux.dev
Subject: Re: [PATCH 0/8] Introduce fwctl subystem

On Thu, Jun 06, 2024 at 10:24:46AM -0700, Dan Williams wrote:
> Jason Gunthorpe wrote:
> [..]
> > > I am warming to your assertion that there is a wide array of
> > > vendor-specific configuration and debug that are not an efficient use of
> > > upstream's time to wrap in a shared Linux ABI. I want to explore fwctl
> > > for CXL for that use case, I personally don't want to marshal a Linux
> > > command to each vendor's slightly different backend CXL toggles.
> > 
> > Personally I think this idea to marshal/unmarshal everything in the
> > kernel is often misguided. If it is truely obvious and actually shared
> > multi-vendor capability then by all means go and do it.
> > 
> > But if you are spending weeks/months fighting about uAPI because all
> > the vendors are so different, it isn't obvious what is "generic" then
> > you've probably already lost. The very worst outcome is a per-device
> > uAPI masquerading as an obfuscated "generic" uAPI that wasted ages of
> > peoples time to argue out.
> 
> Certainly once you have gotten to the "months of arguing" point it begs the
> question "was there really any generic benefit to reap in the first
> place?"

Indeed, but I've seen, and participated, in these things many times :)

> That said, *some* grappling, especially when muliple vendors hit the
> list with the similar feature at the same time, has yielded
> collaboration in the past. 

Absolutely! But we have also frequently done that retroactively, like
see three examples and then consolidate the common APIs. The challenge
is uAPI. Since we can't change uAPI people like to rush to make it
future proof without examples. Broadly I lean towards waiting until we
have several examples to build a standard uAPI and let the examples
evolve on their own.

If there is value in the commonality then people will change over.

> This gets back to the unspoken conceit of the kernel restriction that I
> mentioned earlier. At some point the kernel restriction begets a cynical
> in-tree workaround or an out-of-tree workaround which either way means
> upstream Linux loses.

Right.. The kernel just don't have the power to say no to the
industry. Things will just go OOT and it is really our community that
suffers in the long run. As I said, you can't lead with NO.

IHMO there has to be a really high quality reason to keep support for
HW that people have built out of the kernel. Especially start ups and
other more vulnerable companies. I don't think Linux maintainers
should be choosing industry winners and losers. I sometimes feel I
have a minority opinion here though :(

> > > So, document what each subsystem's stance towards fwctl is,
> > > like maybe a distro only wants fwctl to front publicly documented vendor
> > > commands, or maybe private vendor commands ok, but only with a
> > > constrained set of Command Effects (I potentially see CXL here). 
> > 
> > I wouldn't say subsystem here, but techonology. I think it is
> > reasonable that a CXL fwctl driver have some kconfig tunables like you
> > already have. This idea works alot better if the underlying thing is
> > already standards based.
> 
> True, I worry about these technologies that cross upstream maintainer
> boundaries. When you have a composable switch that enables net, block,
> and/or mem use cases, which upstream maintainer policy applies to the
> fwctl posture of that thing?

fwctl is intended to sit on its own. I think it is even a bad
architecture direction that Linux has N different ways to flash FW on
devices, N different ways to read diagnostics, etc all because each
subsystem went on its own. With fwctl I'd like to see a greater
consolidation of not re-inventing the low level of fw interaction
differently in each and every subsystem.

Like you mentioned CXL has its own way to program flash. How many ways
does Linux have to update device flash now? :(

So, if you have a real multi-function device fwctl should be the
central place to operate the shared PCI function and the FW
interface. There may be some duplication in subsystems, but that is a
side effect of our sub-system siloed development model (software
architecture tends to follow org chart, after all)

Jason

Powered by blists - more mailing lists