Message-ID: <20200330123623.634739de@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date:   Mon, 30 Mar 2020 12:36:23 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     Parav Pandit <parav@...lanox.com>
Cc:     Jiri Pirko <jiri@...nulli.us>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        Yuval Avnery <yuvalav@...lanox.com>,
        "jgg@...pe.ca" <jgg@...pe.ca>,
        Saeed Mahameed <saeedm@...lanox.com>,
        "leon@...nel.org" <leon@...nel.org>,
        "andrew.gospodarek@...adcom.com" <andrew.gospodarek@...adcom.com>,
        "michael.chan@...adcom.com" <michael.chan@...adcom.com>,
        Moshe Shemesh <moshe@...lanox.com>,
        Aya Levin <ayal@...lanox.com>,
        Eran Ben Elisha <eranbe@...lanox.com>,
        Vlad Buslov <vladbu@...lanox.com>,
        Yevgeny Kliteynik <kliteyn@...lanox.com>,
        "dchickles@...vell.com" <dchickles@...vell.com>,
        "sburla@...vell.com" <sburla@...vell.com>,
        "fmanlunas@...vell.com" <fmanlunas@...vell.com>,
        Tariq Toukan <tariqt@...lanox.com>,
        "oss-drivers@...ronome.com" <oss-drivers@...ronome.com>,
        "snelson@...sando.io" <snelson@...sando.io>,
        "drivers@...sando.io" <drivers@...sando.io>,
        "aelior@...vell.com" <aelior@...vell.com>,
        "GR-everest-linux-l2@...vell.com" <GR-everest-linux-l2@...vell.com>,
        "grygorii.strashko@...com" <grygorii.strashko@...com>,
        mlxsw <mlxsw@...lanox.com>, Ido Schimmel <idosch@...lanox.com>,
        Mark Zhang <markz@...lanox.com>,
        "jacob.e.keller@...el.com" <jacob.e.keller@...el.com>,
        Alex Vesker <valex@...lanox.com>,
        "linyunsheng@...wei.com" <linyunsheng@...wei.com>,
        "lihong.yang@...el.com" <lihong.yang@...el.com>,
        "vikas.gupta@...adcom.com" <vikas.gupta@...adcom.com>,
        "magnus.karlsson@...el.com" <magnus.karlsson@...el.com>
Subject: Re: [RFC] current devlink extension plan for NICs

On Mon, 30 Mar 2020 07:48:39 +0000 Parav Pandit wrote:
> On 3/27/2020 10:08 PM, Jakub Kicinski wrote:
> > On Fri, 27 Mar 2020 08:47:36 +0100 Jiri Pirko wrote:  
> >>> So the queues, interrupts, and other resources are also part 
> >>> of the slice then?    
> >>
> >> Yep, that seems to make sense.
> >>  
> >>> How do slice parameters like rate apply to NVMe?    
> >>
> >> Not really.
> >>  
> >>> Are ports always ethernet? and slices also cover endpoints with
> >>> transport stack offloaded to the NIC?    
> >>
> >> devlink_port now can be either "ethernet" or "infiniband". Perhaps,
> >> there can be port type "nve" which would contain only some of the
> >> config options and would not have a representor "netdev/ibdev" linked.
> >> I don't know.  
> > 
> > I honestly find it hard to understand what that slice abstraction is,
> > and which things belong to slices and which to PCI ports (or why we even
> > have them).
> >   
> In an alternative, devlink port can be overloaded/retrofit to do all
> things that slice desires to do.

I wouldn't say retrofitted, in my mind port has always been a port of 
a device.

Jiri explained to me that to Mellanox port is port of a eswitch, not
port of a device. While to me (/Netronome) it was any way to send or
receive data to/from the device.

Now I understand why to you nvme doesn't fit the port abstraction.

> For that matter representor netdev can be overloaded/extended to do what
> slice desire to do (instead of devlink port).

Right, in my mental model representor _is_ a port of the eswitch, so
repr would not make sense to me.

> Can you please explain why you think devlink port should be overloaded
> instead of netdev or any other kernel object?
> Do you have an example of such overloaded functionality of a kernel object?
> Like why macvlan and vlan drivers are not combined to in single driver
> object? Why teaming and bonding driver are combined in single driver
> object?...

I think it's not overloading, but the fact that we started with
different definitions. We (me and you) tried adding the PCIe ports
around the same time, I guess we should have dug into the details
right away.

> User should be able to create, configure, deploy, delete a 'portion of
> the device' with/without eswitch.

Right, to me ports are of the device, not eswitch.

> We shouldn't be starting with restrictive/narrow view of devlink port.
> 
> Internally with Jiri and others, we also explored the possibility to
> have 'mgmtvf', 'mgmtpf',  'mgmtsf' port flavours by overloading port to
> do all things as that of slice.
> It wasn't elegant enough. Why not create right object?

We just need clear definitions of what goes where. We already have
params etc. hanging off the ports, including irq/sriov stuff. But in
slice model those don't belong there :S

In fact very little belongs to the port in that model. So why have
PCI ports in the first place?

> Additionally devlink port object doesn't go through the same state
> machine as that what slice has to go through.
> So its weird that some devlink port has state machine and some doesn't.

You mean for VFs? I think you can add the states to the API.

> > With devices like NFP and Mellanox CX3 which have one PCI PF maybe it
> > would have made sense to have a slice that covers multiple ports, but
> > it seems the proposal is to have port to slice mapping be 1:1. And rate
> > in those devices should still be per port not per slice.
> >   
> Slice can have multiple ports. slice object doesn't restrict it. User
> can always split the port for a device, if device support it.

Okay, so slices are not 1:1 with ports, then?  Is it any:any?

> > But this keeps coming back, and since you guys are doing all the work,
> > if you really really need it..

Powered by blists - more mailing lists