Message-ID: <20250409161049.GM1778492@nvidia.com>
Date: Wed, 9 Apr 2025 13:10:49 -0300
From: Jason Gunthorpe <jgg@...dia.com>
To: Jamal Hadi Salim <jhs@...atatu.com>
Cc: Leon Romanovsky <leon@...nel.org>,
	Nikolay Aleksandrov <nikolay@...abrica.net>,
	Linux Kernel Network Developers <netdev@...r.kernel.org>,
	Shrijeet Mukherjee <shrijeet@...abrica.net>,
	alex.badea@...sight.com, eric.davis@...adcom.com, rip.sohan@....com,
	David Ahern <dsahern@...nel.org>, bmt@...ich.ibm.com,
	roland@...abrica.net, Winston Liu <winston.liu@...sight.com>,
	dan.mihailescu@...sight.com, kheib@...hat.com,
	parth.v.parikh@...sight.com, davem@...hat.com, ian.ziemba@....com,
	andrew.tauferner@...nelisnetworks.com, welch@....com,
	rakhahari.bhunia@...sight.com, kingshuk.mandal@...sight.com,
	linux-rdma@...r.kernel.org, Jakub Kicinski <kuba@...nel.org>,
	Paolo Abeni <pabeni@...hat.com>
Subject: Re: Netlink vs ioctl WAS(Re: [RFC PATCH 00/13] Ultra Ethernet driver
 introduction

On Tue, Apr 08, 2025 at 10:16:45AM -0400, Jamal Hadi Salim wrote:
> > > I cant imagine a commonly used utility like iproute2/tc being
> > > invoked with "when using broadcom then use foo=x bar=y" apply but
> > > when using intel use "goo=x-1 and gah=y-2".
> >
> > Right, it doesn't make sense for a tool like iproute, but we aren't
> > building anything remotely like iproute.
> >
> 
> My point was on the API. I dont know enough so pardon my ignorance. My
> basic assumption is there is common cross-vendor tooling and that
> deployments may have to be multi-vendor. If that assumption is wrong
> then then my concern is not valid.
> If my assumption is correct, whatever provisioning app is involved it
> needs to keep track of the multiple vendor interfacing - which means
> the code will have to understand different semantics across vendors.

It is like DRM and other places. There is only one userspace
implementation, coded into a library that all actual implementations
use.

For example, one of the ioctls is 'alloc pd'. All user applications
will link to libibverb.so and invoke ibv_alloc_pd().

ib_create_pd() under the covers has detected what kind of kernel
driver is present and will load an appropriate helper library, lets's
say libmlx5.so. So it calls mlx5_alloc_pd() which knows how to talk to
the kernel mlx5 side.

It is the responsibility of libibverbs.so/libmlx5.so to present a
standardized library call interface that is largely perscribed by the
IBTA specification.

For DRM this is similar to how libmesa/etc present a standardized
Vulkan/OpenGL library call interface but the kernel ioctls are all
very device specific.

There is no use case, or interest, in making it easy for anyone to
invoke the ioctls without using the single userspace library.

DRM/RDMA are all building things like this in pursuit of maximum
performance. We cannot afford to put an abstraction layer in the
kernel. Instead it is abstracted in userspace code with a
userspace/kernel split driver architecture.

> > For instance, if the user is running a MPI application and the vendor
> > makes standard open source MPI 5% faster with some unique HW
> > innovation should anyone actually care about the "common path" deep,
> > deep below MPI?
> 
> I would say they shouldnt care because the customer gets to benefit.
> But on the flip side, again, that is counting on the goodwill of the
> vendor.

In this space we have sophisticated large customers, it is not good
will. The open source stuff appears because the customers demand it,
so long as that is true I feel pretty comfortable with things.

Jason

Powered by blists - more mailing lists