Date:   Tue, 09 Feb 2021 01:39:09 -0800
From:   Saeed Mahameed <saeed@...nel.org>
To:     Jakub Kicinski <kuba@...nel.org>
Cc:     Yishai Hadas <yishaih@...dia.com>, netdev@...r.kernel.org,
        davem@...emloft.net, parav@...dia.com
Subject: Re: [PATCH net-next 0/2] devlink: Add port function attribute to
 enable/disable roce

On Wed, 2021-02-03 at 13:26 -0800, Jakub Kicinski wrote:
> On Wed, 03 Feb 2021 11:22:44 -0800 Saeed Mahameed wrote:
> > On Wed, 2021-02-03 at 10:51 -0800, Jakub Kicinski wrote:
> > > On Tue, 02 Feb 2021 20:13:48 -0800 Saeed Mahameed wrote:  
> > > > yes, user in this case is the admin, who controls the
> > > > provisioned
> > > > network function SF/VFs.. by turning off this knob it allows to
> > > > create
> > > > more of that resource in case the user/admin is limited by
> > > > memory.  
> > > 
> > > Ah, so in case of the SmartNIC this extra memory is allocated on
> > > the
> > > control system, not where the function resides?
> > 
> > most of the memeory are actually allocated from where the function
> > resides, some are on the management system but it is not as
> > critical.
> > SFs for now can only be probed on the management system, so the
> > main
> > issue will be on the SmartNIC side for now.
> 
> Why not leave the decision whether to allocate that memory or not to
> the SF itself? If user never binds the RDMA driver to the SF they
> clearly don't care for RDMA. No extra knobs needed.
> 

Sorry about the late response.

But FW is already setup for RDMA weather SW wants it or not for this
specific function.
a system admin may want to deploy a leaner function to the client. we
can disable RoCE in FW before we even instantiate this function.

> > > My next question is regarding the behavior on the target system -
> > > what
> > > does "that user" see? Can we expect they will understand that the
> > > limitation was imposed by the admin and not due to some
> > > initialization
> > > failure or SW incompatibility?
> > 
> > the whole thing works with only real HW capabilities, there is no
> > synthetic SW capabilities. 
> > 
> > when mlx5 instance driver loads, it doesn't assume anything about
> > underlying HW, and it queries for the advertised FW capability
> > according to the HW spec before it enables a feature.
> > 
> > so this patch adds the ability for admin to enforce a specific HW
> > cap
> > "off" for a VF/SF hca slice.
> > 
> > > > RAW eth QP, i think you already know this one, it is a very
> > > > thin
> > > > layer
> > > > that doesn't require the whole rdma stack.  
> > > 
> > > Sorry for asking a leading question. You know how we'll feel
> > > about
> > > that one, do we need to talk this out or can we save ourselves
> > > the
> > > battle? :S  
> > 
> > I know, I know :/
> > 
> > So, there is no rdma bit/cap in HW.. to disable non-RoCE commands
> > we
> > will have to disable etherent capability. 
> 
> It's your driver, you can make it do what you need to. Why does 
> the RDMA driver bind successfully to a non-RoCE Ethernet device 
> in the first place?
> 

because RDMA people are greedy :), and they can create QPs that are not
RoCE..

this patch is only basic enable/disable RoCE feature on a specific
device slice, how rdma driver initializes itself is not related to this
patch at all, and it is just how rdma works, with or without this
patch.

> > The user interface here has no synthetic semantics, all knobs will
> > eventually be mapped to real HW/FW capabilities to get disabled.
> > 
> > the whole feature is about allowing admin to ship network functions
> > with different capabilities that are actually enforced by FW/HW.. 
> > so the user of the VF will see, RDMA/ETH only cards or both.
> 
> RDMA-only, ETH-only, RDMA+ETH makes sense to me. Having an ETH-only
> device also exposed though rdma subsystem does not.

But this has nothing to do with this patch. this is the rdma driver
implementation.

Powered by blists - more mailing lists