Message-ID: <AM0PR05MB4866687F67EE56BEBB0D9B3AD1B90@AM0PR05MB4866.eurprd05.prod.outlook.com>
Date:   Tue, 19 May 2020 05:17:27 +0000
From:   Parav Pandit <parav@...lanox.com>
To:     Jacob Keller <jacob.e.keller@...el.com>,
        Jiri Pirko <jiri@...nulli.us>
CC:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "kuba@...nel.org" <kuba@...nel.org>,
        Yuval Avnery <yuvalav@...lanox.com>,
        "jgg@...pe.ca" <jgg@...pe.ca>,
        Saeed Mahameed <saeedm@...lanox.com>,
        "leon@...nel.org" <leon@...nel.org>,
        "andrew.gospodarek@...adcom.com" <andrew.gospodarek@...adcom.com>,
        "michael.chan@...adcom.com" <michael.chan@...adcom.com>,
        Moshe Shemesh <moshe@...lanox.com>,
        Aya Levin <ayal@...lanox.com>,
        Eran Ben Elisha <eranbe@...lanox.com>,
        Vlad Buslov <vladbu@...lanox.com>,
        Yevgeny Kliteynik <kliteyn@...lanox.com>,
        "dchickles@...vell.com" <dchickles@...vell.com>,
        "sburla@...vell.com" <sburla@...vell.com>,
        "fmanlunas@...vell.com" <fmanlunas@...vell.com>,
        Tariq Toukan <tariqt@...lanox.com>,
        "oss-drivers@...ronome.com" <oss-drivers@...ronome.com>,
        "snelson@...sando.io" <snelson@...sando.io>,
        "drivers@...sando.io" <drivers@...sando.io>,
        "aelior@...vell.com" <aelior@...vell.com>,
        "GR-everest-linux-l2@...vell.com" <GR-everest-linux-l2@...vell.com>,
        "grygorii.strashko@...com" <grygorii.strashko@...com>,
        mlxsw <mlxsw@...lanox.com>, Ido Schimmel <idosch@...lanox.com>,
        Mark Zhang <markz@...lanox.com>,
        Alex Vesker <valex@...lanox.com>,
        "linyunsheng@...wei.com" <linyunsheng@...wei.com>,
        "lihong.yang@...el.com" <lihong.yang@...el.com>,
        "vikas.gupta@...adcom.com" <vikas.gupta@...adcom.com>,
        "sridhar.samudrala@...el.com" <sridhar.samudrala@...el.com>
Subject: RE: [RFC v2] current devlink extension plan for NICs

Hi Jake,

> From: netdev-owner@...r.kernel.org <netdev-owner@...r.kernel.org> On
> Behalf Of Jacob Keller
> 
> 
> On 5/17/2020 11:52 PM, Jiri Pirko wrote:
> > Fri, May 15, 2020 at 11:36:19PM CEST, jacob.e.keller@...el.com wrote:
> >>
> >>
> >> On 5/15/2020 2:30 AM, Jiri Pirko wrote:
> >>> Fri, May 15, 2020 at 01:52:54AM CEST, jacob.e.keller@...el.com wrote:
> >>>>> $ devlink port add pci/0000.06.00.0/100 flavour pcisf pfnum 1
> >>>>> sfnum 10
> >>>>>
> >>>>
> >>>> Can you clarify what sfnum means here? and why is it different from
> >>>> the index? I get that the index is a unique number that identifies
> >>>> the port regardless of type, so sfnum must be some sort of hardware
> >>>> internal identifier?
> >>>
> >>> Basically pfnum, sfnum and vfnum could overlap. Index is unique
> >>> within all groups together.
> >>>
> >>
> >> Right. Index is just an identifier for which port this is.
> >>
> 
> Ok, so whether or not a driver uses this internally is an implementation
> detail that doesn't matter to the interface.
> 
> 
> >>>
> >>>>
> >>>> When looking at this with colleagues, there was a lot of confusion
> >>>> about the difference between the index and the sfnum.
> >>>
> >>> No confusion about index and pfnum/vfnum? They behave the same.
> >>> Index is just a port handle.
> >>>
> >>
> >> I'm less confused about the difference between index and these
> >> "nums", and more so questioning what pfnum/vfnum/sfnum represent?
> Are
> >> they similar to the vf ID that we have in the legacy SRIOV functions?
> >> I.e. a hardware index?
> >>
> >> I don't think in general users necessarily care which "index" they
> >> get upfront. They obviously very much care about the index once it's
> >> selected. I do believe the interfaces should start with the
> >> capability for the index to be selected automatically at creation
> >> (with the optional capability to select a specific index if desired, as shown
> here).
> >>
> >> I do not think most users want to care about what to pick for this
> >> number. (Just as they would not want to pick a number for the port
> >> index either).
> >
> > I see your point. However I don't think it is always the right
> > scenario. The "nums" are used for naming of the netdevices, both the
> > eswitch port representor and the actual SF (in case of SF).
> >
> > I think that in lot of usecases is more convenient for user to select
> > the "num" on the cmdline.
> >
> 
> Agreed, based on the below statements. Basically "let users specify or get it
> automatically chosen", just like with the port identifier and with the region
> numbers now.
> 
> 
> Thanks for the explanations!
> 
> >>
> >>>> Obviously this is a TODO, but how does this differ from the current
> >>>> port_split and port_unsplit?
> >>>
> >>> Does not have anything to do with port splitting. This is about
> >>> creating a "child PF" from the section above.
> >>>
> >>
> >> Hmm. Ok so this is about internal connections in the switch, then?
> >
> > Yes. Take the smartnic as an example. On the smartnic cpu, the eswitch
> > management is being done. There's devlink instance with all eswitch
> > port visible as devlink ports. One PF-type devlink port per host. That
> > are the "child PFs".
> >
> > Now from perspective of the host, there are 2 scenarios:
> > 1) have the "simple dumb" PF, which just exposes 1 netdev for host to
> >    run traffic over. smartnic cpu manages the VFs/SFs and sees the
> >    devlink ports for them. This is 1 level switch - merged switch
> >
> > 2) PF manages a sub-switch/nested-switch. The devlink/devlink ports are
> >    created on the host and the devlink ports for SFs/VFs are created
> >    there. This is multi-level eswitch. Each "child PF" on a parent
> >    manages a nested switch. And could in theory have other PF child with
> >    another nested switch.
> >
> 
> Ok. So in the smart NIC CPU, we'd see the primary PF and some child PFs,
> and in the host system we'd see a "primary PF" that is the other end of the
> associated Child PF, and might be able to manage its own subswitch.
> 
> Ok this is making more sense now.
> 
> I think I had imagined that was what subfuntions were. But really
> subfunctions are a bit different, they're more similar to expanded VFs?
>
 
1. Sub functions are more light weight than VFs because,
2. They share the same PCI device (BAR, IRQs) as that of PF/VF on which it is deployed.
3. Unlike VFs which are enabled/disabled in bulk, subfunctions are created, deployed in unit of 1.

Since this RFC content is overwhelming, I expanded the SF plumbing details more in [1] in previous RFC version.
You can replace 'devlink slice' with 'devlink port func' in [1].

[1] https://marc.info/?l=linux-netdev&m=158555928517777&w=2

Powered by blists - more mailing lists