Date:   Mon, 18 May 2020 14:05:45 -0700
From:   Jacob Keller <jacob.e.keller@...el.com>
To:     Jiri Pirko <jiri@...nulli.us>
Cc:     netdev@...r.kernel.org, davem@...emloft.net, kuba@...nel.org,
        parav@...lanox.com, yuvalav@...lanox.com, jgg@...pe.ca,
        saeedm@...lanox.com, leon@...nel.org,
        andrew.gospodarek@...adcom.com, michael.chan@...adcom.com,
        moshe@...lanox.com, ayal@...lanox.com, eranbe@...lanox.com,
        vladbu@...lanox.com, kliteyn@...lanox.com, dchickles@...vell.com,
        sburla@...vell.com, fmanlunas@...vell.com, tariqt@...lanox.com,
        oss-drivers@...ronome.com, snelson@...sando.io,
        drivers@...sando.io, aelior@...vell.com,
        GR-everest-linux-l2@...vell.com, grygorii.strashko@...com,
        mlxsw@...lanox.com, idosch@...lanox.com, markz@...lanox.com,
        valex@...lanox.com, linyunsheng@...wei.com, lihong.yang@...el.com,
        vikas.gupta@...adcom.com, sridhar.samudrala@...el.com
Subject: Re: [RFC v2] current devlink extension plan for NICs



On 5/17/2020 11:52 PM, Jiri Pirko wrote:
> Fri, May 15, 2020 at 11:36:19PM CEST, jacob.e.keller@...el.com wrote:
>>
>>
>> On 5/15/2020 2:30 AM, Jiri Pirko wrote:
>>> Fri, May 15, 2020 at 01:52:54AM CEST, jacob.e.keller@...el.com wrote:
>>>>> $ devlink port add pci/0000.06.00.0/100 flavour pcisf pfnum 1 sfnum 10
>>>>>
>>>>
>>>> Can you clarify what sfnum means here? and why is it different from the
>>>> index? I get that the index is a unique number that identifies the port
>>>> regardless of type, so sfnum must be some sort of hardware internal
>>>> identifier?
>>>
>>> Basically pfnum, sfnum and vfnum could overlap. Index is unique within
>>> all groups together.
>>>
>>
>> Right. Index is just an identifier for which port this is.
>>

Ok, so whether or not a driver uses this internally is an implementation
detail that doesn't matter to the interface.


>>>
>>>>
>>>> When looking at this with colleagues, there was a lot of confusion about
>>>> the difference between the index and the sfnum.
>>>
>>> No confusion about index and pfnum/vfnum? They behave the same.
>>> Index is just a port handle.
>>>
>>
>> I'm less confused about the difference between index and these "nums",
>> and more so questioning what pfnum/vfnum/sfnum represent? Are they
>> similar to the vf ID that we have in the legacy SRIOV functions? I.e. a
>> hardware index?
>>
>> I don't think in general users necessarily care which "index" they get
>> upfront. They obviously very much care about the index once it's
>> selected. I do believe the interfaces should start with the capability
>> for the index to be selected automatically at creation (with the
>> optional capability to select a specific index if desired, as shown here).
>>
>> I do not think most users want to care about what to pick for this
>> number. (Just as they would not want to pick a number for the port index
>> either).
> 
> I see your point. However I don't think it is always the right
> scenario. The "nums" are used for naming of the netdevices, both the
> eswitch port representor and the actual SF (in case of SF).
> 
> I think that in lot of usecases is more convenient for user to select
> the "num" on the cmdline.
> 

Agreed, based on the below statements. Basically "let users specify or
get it automatically chosen", just like with the port identifier and
with the region numbers now.


Thanks for the explanations!

>>
>>>> Obviously this is a TODO, but how does this differ from the current
>>>> port_split and port_unsplit?
>>>
>>> Does not have anything to do with port splitting. This is about creating
>>> a "child PF" from the section above.
>>>
>>
>> Hmm. Ok so this is about internal connections in the switch, then?
> 
> Yes. Take the smartnic as an example. On the smartnic cpu, the
> eswitch management is being done. There's devlink instance with all
> eswitch port visible as devlink ports. One PF-type devlink port per
> host. That are the "child PFs".
> 
> Now from perspective of the host, there are 2 scenarios:
> 1) have the "simple dumb" PF, which just exposes 1 netdev for host to
>    run traffic over. smartnic cpu manages the VFs/SFs and sees the
>    devlink ports for them. This is 1 level switch - merged switch
> 
> 2) PF manages a sub-switch/nested-switch. The devlink/devlink ports are
>    created on the host and the devlink ports for SFs/VFs are created
>    there. This is multi-level eswitch. Each "child PF" on a parent
>    manages a nested switch. And could in theory have other PF child with
>    another nested switch.
> 

Ok. So in the smart NIC CPU, we'd see the primary PF and some child PFs,
and in the host system we'd see a "primary PF" that is the other end of
the associated Child PF, and might be able to manage its own subswitch.

Ok this is making more sense now.

I think I had imagined that was what subfuntions were. But really
subfunctions are a bit different, they're more similar to expanded VFs?

Thanks,
Jake

Powered by blists - more mailing lists