Message-ID: <20200519051910.GA4655@nanopsycho>
Date:   Tue, 19 May 2020 07:19:10 +0200
From:   Jiri Pirko <jiri@...nulli.us>
To:     Jacob Keller <jacob.e.keller@...el.com>
Cc:     netdev@...r.kernel.org, davem@...emloft.net, kuba@...nel.org,
        parav@...lanox.com, yuvalav@...lanox.com, jgg@...pe.ca,
        saeedm@...lanox.com, leon@...nel.org,
        andrew.gospodarek@...adcom.com, michael.chan@...adcom.com,
        moshe@...lanox.com, ayal@...lanox.com, eranbe@...lanox.com,
        vladbu@...lanox.com, kliteyn@...lanox.com, dchickles@...vell.com,
        sburla@...vell.com, fmanlunas@...vell.com, tariqt@...lanox.com,
        oss-drivers@...ronome.com, snelson@...sando.io,
        drivers@...sando.io, aelior@...vell.com,
        GR-everest-linux-l2@...vell.com, grygorii.strashko@...com,
        mlxsw@...lanox.com, idosch@...lanox.com, markz@...lanox.com,
        valex@...lanox.com, linyunsheng@...wei.com, lihong.yang@...el.com,
        vikas.gupta@...adcom.com, sridhar.samudrala@...el.com
Subject: Re: [RFC v2] current devlink extension plan for NICs

Mon, May 18, 2020 at 11:05:45PM CEST, jacob.e.keller@...el.com wrote:
>
>
>On 5/17/2020 11:52 PM, Jiri Pirko wrote:
>> Fri, May 15, 2020 at 11:36:19PM CEST, jacob.e.keller@...el.com wrote:
>>>
>>>
>>> On 5/15/2020 2:30 AM, Jiri Pirko wrote:
>>>> Fri, May 15, 2020 at 01:52:54AM CEST, jacob.e.keller@...el.com wrote:
>>>>>> $ devlink port add pci/0000.06.00.0/100 flavour pcisf pfnum 1 sfnum 10
>>>>>>
>>>>>
>>>>> Can you clarify what sfnum means here? and why is it different from the
>>>>> index? I get that the index is a unique number that identifies the port
>>>>> regardless of type, so sfnum must be some sort of hardware internal
>>>>> identifier?
>>>>
>>>> Basically pfnum, sfnum and vfnum could overlap. Index is unique within
>>>> all groups together.
>>>>
>>>
>>> Right. Index is just an identifier for which port this is.
>>>
>
>Ok, so whether or not a driver uses this internally is an implementation
>detail that doesn't matter to the interface.
>
>
>>>>
>>>>>
>>>>> When looking at this with colleagues, there was a lot of confusion about
>>>>> the difference between the index and the sfnum.
>>>>
>>>> No confusion about index and pfnum/vfnum? They behave the same.
>>>> Index is just a port handle.
>>>>
>>>
>>> I'm less confused about the difference between index and these "nums",
>>> and more so questioning what pfnum/vfnum/sfnum represent? Are they
>>> similar to the vf ID that we have in the legacy SRIOV functions? I.e. a
>>> hardware index?
>>>
>>> I don't think in general users necessarily care which "index" they get
>>> upfront. They obviously very much care about the index once it's
>>> selected. I do believe the interfaces should start with the capability
>>> for the index to be selected automatically at creation (with the
>>> optional capability to select a specific index if desired, as shown here).
>>>
>>> I do not think most users want to care about what to pick for this
>>> number. (Just as they would not want to pick a number for the port index
>>> either).
>> 
>> I see your point. However I don't think it is always the right
>> scenario. The "nums" are used for naming of the netdevices, both the
>> eswitch port representor and the actual SF (in case of SF).
>> 
>> I think that in lot of usecases is more convenient for user to select
>> the "num" on the cmdline.
>> 
>
>Agreed, based on the below statements. Basically "let users specify or
>get it automatically chosen", just like with the port identifier and
>with the region numbers now.
>
>
>Thanks for the explanations!
>
>>>
>>>>> Obviously this is a TODO, but how does this differ from the current
>>>>> port_split and port_unsplit?
>>>>
>>>> Does not have anything to do with port splitting. This is about creating
>>>> a "child PF" from the section above.
>>>>
>>>
>>> Hmm. Ok so this is about internal connections in the switch, then?
>> 
>> Yes. Take the smartnic as an example. On the smartnic cpu, the
>> eswitch management is being done. There's devlink instance with all
>> eswitch port visible as devlink ports. One PF-type devlink port per
>> host. That are the "child PFs".
>> 
>> Now from perspective of the host, there are 2 scenarios:
>> 1) have the "simple dumb" PF, which just exposes 1 netdev for host to
>>    run traffic over. smartnic cpu manages the VFs/SFs and sees the
>>    devlink ports for them. This is 1 level switch - merged switch
>> 
>> 2) PF manages a sub-switch/nested-switch. The devlink/devlink ports are
>>    created on the host and the devlink ports for SFs/VFs are created
>>    there. This is multi-level eswitch. Each "child PF" on a parent
>>    manages a nested switch. And could in theory have other PF child with
>>    another nested switch.
>> 
>
>Ok. So in the smart NIC CPU, we'd see the primary PF and some child PFs,
>and in the host system we'd see a "primary PF" that is the other end of
>the associated Child PF, and might be able to manage its own subswitch.
>
>Ok this is making more sense now.
>
>I think I had imagined that was what subfuntions were. But really
>subfunctions are a bit different, they're more similar to expanded VFs?

Yeah, they are basically VFs without separate pci BDF. They reside on a
BDF of the PF they are created on. Basically a lightweight VFs.


>
>Thanks,
>Jake

Powered by blists - more mailing lists