Message-ID: <20250417133156.GG823903@nvidia.com>
Date: Thu, 17 Apr 2025 10:31:56 -0300
From: Jason Gunthorpe <jgg@...dia.com>
To: Sean Hefty <shefty@...dia.com>
Cc: "Ziemba, Ian" <ian.ziemba@....com>,
	Bernard Metzler <BMT@...ich.ibm.com>,
	Roland Dreier <roland@...abrica.net>,
	Nikolay Aleksandrov <nikolay@...abrica.net>,
	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"shrijeet@...abrica.net" <shrijeet@...abrica.net>,
	"alex.badea@...sight.com" <alex.badea@...sight.com>,
	"eric.davis@...adcom.com" <eric.davis@...adcom.com>,
	"rip.sohan@....com" <rip.sohan@....com>,
	"dsahern@...nel.org" <dsahern@...nel.org>,
	"winston.liu@...sight.com" <winston.liu@...sight.com>,
	"dan.mihailescu@...sight.com" <dan.mihailescu@...sight.com>,
	Kamal Heib <kheib@...hat.com>,
	"parth.v.parikh@...sight.com" <parth.v.parikh@...sight.com>,
	Dave Miller <davem@...hat.com>,
	"andrew.tauferner@...nelisnetworks.com" <andrew.tauferner@...nelisnetworks.com>,
	"welch@....com" <welch@....com>,
	"rakhahari.bhunia@...sight.com" <rakhahari.bhunia@...sight.com>,
	"kingshuk.mandal@...sight.com" <kingshuk.mandal@...sight.com>,
	"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
	"kuba@...nel.org" <kuba@...nel.org>,
	Paolo Abeni <pabeni@...hat.com>
Subject: Re: [RFC PATCH 00/13] Ultra Ethernet driver introduction

On Thu, Apr 17, 2025 at 02:59:58AM +0000, Sean Hefty wrote:
> > I think the "Relative Addressing" Ian described is just a PD pointing to a single
> > job and all MRs within the PD linked to a single job. Is there more than that?
> 
> Relative / absolute addressing is in regard to the endpoint address.
> I.e. the equivalent of the QPN.
> 
> With relative addressing, the QPN is relative to the job ID.  So
> QPN=5 for job=2 and QPN=5 for job=3 may or may not be the same HW
> resource.  A HW QP may still belong to multiple jobs, if supported
> by the vendor.

Yes, but I think the key distinction is that everything is relative
to, or contained with in the job key so we only have ony job key and
every single object touched by a packet must be within that job. That
is the same security model as PD if the PD has 1 job.

> As an example, assigning MRs to jobs allows the server to setup RMA
> buffers with access restricted to that job.
> 
> I have no idea how the receiver plans to enable sending back a response.

Or get access to the new job id, which seems like a more important
question for the OS. I think I understand that there must be some
privileged entity that grants fine grained access to jobs, but I have
not seen any detail on how that would actually work inside the OS to
cover all these cases.

Does this all-listening process have to do some kind of DBUS operation
to request access to a job and get back a job FD? Something else? Does
anyone have a plan in mind?

MPI seems to have a more obvious design where the launcher could be
privileged and pass a job FD to its children. The global MPI scheduler
could allocate the network-global job ids. Un priv processes never
request a job on the fly.

> The second feature is called scalable
> endpoints.  A scalable endpoint has multiple receive queues, which
> are directly addressable by the peer.  Different jobs could target
> different receive queues.

That's just a new queue with different addressing rules. If the new
queue is created inside a new PD from it's endpoint are we OK then?

> I've gone back and forth between separating and combining the
> 'security key' and job objects.  Today I opted for separate, more
> focused objects.  Tomorrow, who knows?  Job is where addressing
> information goes.

I don't know about combining, but it seems like security key and
addressing are sub objects of the top level job? Is there any reason
to share a security key with two jobs???

> A separate security key made more sense to me when I considered
> applying it to an RC QP.  Additionally, an MPI/AI job may require
> multiple job objects, one for each IP address.  (Imagine a system
> connected to separate networks, such that the job ID value cannot be
> global).  A single security key can be used with all job instances.

I haven't heard any definition of how the job id is actually matched.

If you are talking about permitting on-the-wire job ids that alias and
map to different OS level job security domains then the HW must be
doing a full (src IP, dst IP, job key) -> Job Context search on every
packet to disambiguate?

That seems like something a latency focused HPC NIC would not want to do.

If you are not doing full searching based on all allowed src IPs then
you can't have separate networks with separate job id spaces either.

But even then, managing the number space seem very hard. If a MPI
scheduler is assiging on-the-wire job ids from src/dst IP pairs within
its cluster then nothing else can assign job IDs from that pool or it
will conflict.

Jason

Powered by blists - more mailing lists