netdev - Re: [RFC] current devlink extension plan for NICs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ea8a8434b1db2b692489edfd4abbc2274a77228c.camel@mellanox.com>
Date:   Fri, 27 Mar 2020 19:45:53 +0000
From:   Saeed Mahameed <saeedm@...lanox.com>
To:     "sridhar.samudrala@...el.com" <sridhar.samudrala@...el.com>,
        "kuba@...nel.org" <kuba@...nel.org>
CC:     Aya Levin <ayal@...lanox.com>,
        "andrew.gospodarek@...adcom.com" <andrew.gospodarek@...adcom.com>,
        "sburla@...vell.com" <sburla@...vell.com>,
        "jiri@...nulli.us" <jiri@...nulli.us>,
        Tariq Toukan <tariqt@...lanox.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Vlad Buslov <vladbu@...lanox.com>,
        "lihong.yang@...el.com" <lihong.yang@...el.com>,
        Ido Schimmel <idosch@...lanox.com>,
        "jgg@...pe.ca" <jgg@...pe.ca>,
        "fmanlunas@...vell.com" <fmanlunas@...vell.com>,
        "oss-drivers@...ronome.com" <oss-drivers@...ronome.com>,
        "leon@...nel.org" <leon@...nel.org>,
        Parav Pandit <parav@...lanox.com>,
        "grygorii.strashko@...com" <grygorii.strashko@...com>,
        "michael.chan@...adcom.com" <michael.chan@...adcom.com>,
        Alex Vesker <valex@...lanox.com>,
        "snelson@...sando.io" <snelson@...sando.io>,
        "linyunsheng@...wei.com" <linyunsheng@...wei.com>,
        "magnus.karlsson@...el.com" <magnus.karlsson@...el.com>,
        "dchickles@...vell.com" <dchickles@...vell.com>,
        "jacob.e.keller@...el.com" <jacob.e.keller@...el.com>,
        Moshe Shemesh <moshe@...lanox.com>,
        Mark Zhang <markz@...lanox.com>,
        "aelior@...vell.com" <aelior@...vell.com>,
        Yuval Avnery <yuvalav@...lanox.com>,
        "drivers@...sando.io" <drivers@...sando.io>,
        mlxsw <mlxsw@...lanox.com>,
        "GR-everest-linux-l2@...vell.com" <GR-everest-linux-l2@...vell.com>,
        Yevgeny Kliteynik <kliteyn@...lanox.com>,
        "vikas.gupta@...adcom.com" <vikas.gupta@...adcom.com>,
        Eran Ben Elisha <eranbe@...lanox.com>
Subject: Re: [RFC] current devlink extension plan for NICs

On Fri, 2020-03-27 at 12:10 -0700, Jakub Kicinski wrote:
> On Fri, 27 Mar 2020 11:49:10 -0700 Samudrala, Sridhar wrote:
> > On 3/27/2020 9:38 AM, Jakub Kicinski wrote:
> > > On Fri, 27 Mar 2020 08:47:36 +0100 Jiri Pirko wrote:  
> > > > > So the queues, interrupts, and other resources are also part
> > > > > of the slice then?  
> > > > 
> > > > Yep, that seems to make sense.
> > > >  
> > > > > How do slice parameters like rate apply to NVMe?  
> > > > 
> > > > Not really.
> > > >  
> > > > > Are ports always ethernet? and slices also cover endpoints
> > > > > with
> > > > > transport stack offloaded to the NIC?  
> > > > 
> > > > devlink_port now can be either "ethernet" or "infiniband".
> > > > Perhaps,
> > > > there can be port type "nve" which would contain only some of
> > > > the
> > > > config options and would not have a representor "netdev/ibdev"
> > > > linked.
> > > > I don't know.  
> > > 
> > > I honestly find it hard to understand what that slice abstraction
> > > is,
> > > and which things belong to slices and which to PCI ports (or why
> > > we even
> > > have them).  
> > 
> > Looks like slice is a new term for sub function and we can consider
> > this
> > as a VMDQ VSI(intel terminology) or even a Queue group of a VSI.
> > 
> > Today we expose VMDQ VSI via offloaded MACVLAN. This mechanism
> > should 
> > allow us to expose it as a separate netdev.
> 
> Kinda. Looks like with the new APIs you guys will definitely be able
> to
> expose VMDQ as a full(er) device, and if memory serves me well that's
> what you wanted initially.
> 

VMDQ is just a steering based isolated set of rx tx rings pointed at by
a dumb steering rule in the HW .. i am not sure we can just wrap them
in their own vendor specific netdev and just call it a slice..

from what i understand, a real slice is a full isolated HW pipeline
with its own HW resources and HW based isolation, a slice rings/hw
resources can never be shared between different slices, just like a vf,
but without the pcie virtual function back-end..

Why would you need a devlink slice instance for something that has only
rx/tx rings attributes ? if we are going with such design, then i guess
a simple rdma app with a pair of QPs can call itself a slice .. 

We need a clear-cut definition of what a Sub-function slice is.. this
RFC doesn't seem to address that clearly.

> But the sub-functions are just a subset of slices, PF and VFs also
> have a slice associated with them.. And all those things have a port,
> too.
> 

PFs/VFs, might have more than one port sometimes .. 

> > > With devices like NFP and Mellanox CX3 which have one PCI PF
> > > maybe it
> > > would have made sense to have a slice that covers multiple ports,
> > > but
> > > it seems the proposal is to have port to slice mapping be 1:1.
> > > And rate
> > > in those devices should still be per port not per slice.
> > > 
> > > But this keeps coming back, and since you guys are doing all the
> > > work,
> > > if you really really need it..