Message-ID: <20240605135911.GT19897@nvidia.com>
Date: Wed, 5 Jun 2024 10:59:11 -0300
From: Jason Gunthorpe <jgg@...dia.com>
To: Dan Williams <dan.j.williams@...el.com>
Cc: Jakub Kicinski <kuba@...nel.org>, David Ahern <dsahern@...nel.org>,
	Jonathan Corbet <corbet@....net>, Itay Avraham <itayavr@...dia.com>,
	Leon Romanovsky <leon@...nel.org>, linux-doc@...r.kernel.org,
	linux-rdma@...r.kernel.org, netdev@...r.kernel.org,
	Paolo Abeni <pabeni@...hat.com>, Saeed Mahameed <saeedm@...dia.com>,
	Tariq Toukan <tariqt@...dia.com>,
	Andy Gospodarek <andrew.gospodarek@...adcom.com>,
	Aron Silverton <aron.silverton@...cle.com>,
	Christoph Hellwig <hch@...radead.org>, Jiri Pirko <jiri@...dia.com>,
	Leonid Bloch <lbloch@...dia.com>,
	Leon Romanovsky <leonro@...dia.com>, linux-cxl@...r.kernel.org,
	patches@...ts.linux.dev
Subject: Re: [PATCH 0/8] Introduce fwctl subystem

On Tue, Jun 04, 2024 at 04:56:57PM -0700, Dan Williams wrote:
> Jakub Kicinski wrote:
> [..]
> > I don't begrudge anyone building proprietary options, but leave
> > upstream out of it.
> 
> So I am of 2 minds here. In general, how is upstream benefited by
> requiring every vendor command to be wrapped by a Linux command?

People actually can use upstream :)

Amazingly there is inherit benefit to people being able to use the
software we produce.

> 3 years on from that recommendation it seems no vendor has even needed
> that level of distribution help. I.e. checking a few distro kernels
> (Fedora, openSUSE) shows no uptake for CONFIG_CXL_MEM_RAW_COMMANDS=y in
> their debug builds. I can only assume that locally compiled custom
> kernel binaries are filling the need.

My strong advice would be to be careful about this. Android-ism where
nobody runs the upstream kernel is a real thing. For something
emerging like CXL there is a real risk that the hyperscale folks will
go off and do their own OOT stuff and in-tree CXL will be something
usuable but inferior. I've seen this happen enough times..

If people come and say we need X and the maintainer says no, they
don't just give up and stop doing X, the go and do X anyhow out of
tree. This has become especially true now that the center of business
activity in server-Linux is driven by the hyperscale crowd that don't
care much about upstream. Linux maintainer's don't actually have the
power to force the industry to do things, though people do keep
trying.. Maintainers can only lead, and productive leading is not done
with a NO.

You will start to see this pain in maybe 5-10 years if CXL starts to
be something deployed in an enterprise RedHat/Dell/etc sort of
environment. Then that missing X becomes a critical issue because it
turns out the hyperscale folks long since figured out it is really
important but didn't do anything to enable it upstream.

There is merit in upstream being something people can and do actually
use, not just an ivory tower of architectural perfection. There is
merit in bringing code into the community instead of forcing things to
be OOT.

For instance the thread you linked where there was talk of needing the
signal integrity data is a great example. Sure some of that is
manufacturing time, but also if you deploy a million interfaces in a
datacenter, then yes, there will be need to collect SI information
from live systems and do some analysis on it. You wouldn't believe how
much physically broken HW leaks out into data centers and needs
manufacturing level debugging techniques to properly root cause :(

> userpace-to-device-firmware tunnel?" to at least get all the various
> concerns documented in one place, and provide guidance for how device
> vendors should navigate this space across subsystems. 

This is my effort here. If we document the expectations there is a
much better chance that a standard body or device manufacturer can
implement their interfaces in a way that works with the OS. There is a
much higher chance they will attract CVEs and be forced to fix it if
the security expectations are clearly laid out. You had a good
observation in one of those links about how they are not OS
people. Let's help them do better.

Shunt the less robust stuff to fwctl and then people can also make
their own security choices, don't enable or load the fwctl modules and
you get more protection. It is closer to your
CONFIG_CXL_MEM_RAW_COMMANDS=y but at runtime.

I think I captured most of your commentary below here in patch 6.

>   Effects Log". In that "trust Command Effects" scenario the kernel still
>   has no idea what the command is actually doing, but it can at least
>   assert that the device does not claim that the command changes the
>   contents of system-memory. Now, you might say, "the device can just
>   lie", but that betrays a conceit of the kernel restriction. A device
>   could lie that a Linux wrapped command when passed certain payloads does
>   not in turn proxy to a restricted command.

Yeah, we have to trust the device. If the device is hostile toward the
OS then there are already big problems. We need to allow for
unintentional defects in the devices, but we don't need to be
paranoid.

IMHO a command effects report, in conjunction with a robust OS centric
defintion is something we can trust in.

> * Introspection / validation: Subsystem community needs to be able to
>   audit behavior after the fact.
> 
>   To me this means even if the kernel is letting a command through based
>   on the stated Command Effect of "Configuration Change after Cold Reset"
>   upstream community has a need to be able to read the vendor
>   specification for that command. I.e. commands might be vendor-specific,
>   but never vendor-private. I see this as similar to the requirement for
>   open source userspace for sophisticated accelerators.

I'm less hard on this. As long as reasonable open userspace exists I
think it is fine to let other stuff through too. I can appreciate the
DRM stance on this, but IMHO, there is meaningfully more value for open
source in trying get an open Vulkan implementation vs blocking users
from reading their vendor'd diagnostic SI values.

I don't think we should get into some kind of extremism and insist
that every single bit must be documented/standardized or Linux won't
support it.

This is why I envision fwctl as not being suitable for actual
datapath/performance stuff.

> * Collaboration: open standards support open driver maintenance.
> 
>   Without standards we end up with awkward situations like Confidential
>   Computing where every vendor races to implement the same functionality
>   in arbitrarily different and vendor specific ways.

Standard are important. Linux is not a standards body. Linux
maintainers can only advise, not force, the industry to make
standards. At a certain point Linux's job is to implement software to
support what people have built. CC is a sad example where the industry
did not get together enough, but still Linux will support the CC mess.

>   For CXL devices, and I believe the devices fwctl is targeting, there
>   are a whole class of commands for vendor specific configuration and
>   debug. Commands that the kernel really need not worry about.

Right.

>   Some subsystems may want to allow high-performance science experiments
>   like what NVMe allows, but it seems worth asking the question if
>   standardizing device configuration and debug is really the best use of
>   upstream's limited time?

>From what I've been seeing it looks like a significant waste of
time. For example there is minimal industry value in standardizing
values stored in a device's boot time flash configuration. If some
common software wants to access really generic configuration (like
SRIOV enable) then sure there is merit, but that is really the
minority.

Jason

Powered by blists - more mailing lists