Message-ID: <20200504113419.GA14398@nanopsycho.orion>
Date:   Mon, 4 May 2020 13:42:52 +0200
From:   Jiri Pirko <jiri@...nulli.us>
To:     "Samudrala, Sridhar" <sridhar.samudrala@...el.com>
Cc:     netdev@...r.kernel.org, davem@...emloft.net, kuba@...nel.org,
        parav@...lanox.com, yuvalav@...lanox.com, jgg@...pe.ca,
        saeedm@...lanox.com, leon@...nel.org,
        andrew.gospodarek@...adcom.com, michael.chan@...adcom.com,
        moshe@...lanox.com, ayal@...lanox.com, eranbe@...lanox.com,
        vladbu@...lanox.com, kliteyn@...lanox.com, dchickles@...vell.com,
        sburla@...vell.com, fmanlunas@...vell.com, tariqt@...lanox.com,
        oss-drivers@...ronome.com, snelson@...sando.io,
        drivers@...sando.io, aelior@...vell.com,
        GR-everest-linux-l2@...vell.com, grygorii.strashko@...com,
        mlxsw@...lanox.com, idosch@...lanox.com, markz@...lanox.com,
        jacob.e.keller@...el.com, valex@...lanox.com,
        linyunsheng@...wei.com, lihong.yang@...el.com,
        vikas.gupta@...adcom.com
Subject: Re: [RFC v2] current devlink extension plan for NICs

Mon, May 04, 2020 at 04:12:18AM CEST, sridhar.samudrala@...el.com wrote:
>
>
>On 5/1/2020 2:14 AM, Jiri Pirko wrote:
>> Hi all.
>> 
>> First, I would like to apologize for very long email. But I think it
>> would be beneficial to the see the whole picture, where we are going.
>> 
>> Currently we are working internally on several features with
>> need of extension of the current devlink infrastructure. I took a stab
>> at putting it all together in a single txt file, inlined below.
>> 
>> Most of the stuff is based on a new port sub-object called "func"
>> (called "slice" previously" and "subdev" originally in Yuval's patchsets
>> sent some while ago).
>> 
>> The text describes how things should behave and provides a draft
>> of user facing console input/outputs. I think it is important to clear
>> that up before we go in and implement the devlink core and
>> driver pieces.
>> 
>> I would like to ask you to read this and comment. Especially, I would
>> like to ask vendors if what is described fits the needs of your
>> NIC/e-switch.
>> 
>> Please note that something is already implemented, but most of this
>> isn't (see "what needs to be implemented" section).
>> 
>> v1->v2
>> - mainly move from separate slice object into port/func subobject
>> - couple of small fixes here and there
>> 
>
><snip>
>
>> 
>> 
>> 
>> ==================================================================
>> ||                                                              ||
>> ||             Port func user cmdline API draft                 ||
>> ||                                                              ||
>> ==================================================================
>> 
>> Note that some of the "devlink port" attributes may be forgotten or misordered.
>> 
>> Funcs are created as sub-objects of ports where it makes sense to have them
>> The driver takes care of that. The "func" is a handle to configure "the other
>> side of the wire". The original port object has port leve properties,
>> the new "func" sub-object on the other hand has device level properties".
>> 
>> This is example for the HOST A from the example above:
>> 
>> $ devlink port show
>> pci/0000:06:00.0/0: flavour physical pfnum 0 type eth netdev enp6s0f0np1
>> pci/0000:06:00.0/1: flavour physical pfnum 1 type eth netdev enp6s0f0np2
>> pci/0000:06:00.0/2: flavour pcipf pfnum 2 type eth netdev enp6s0pf2
>>                      func: hw_addr 10:22:33:44:55:66 state active
>> pci/0000:06:00.0/3: flavour pcivf pfnum 2 vfnum 0 type eth netdev enp6s0pf2vf0
>>                      func: hw_addr 10:22:33:44:55:77 state active
>> pci/0000:06:00.0/4: flavour pcivf pfnum 0 vfnum 0 type eth netdev enp6s0pf0vf0
>>                      func: hw_addr 10:22:33:44:55:88 state active
>> pci/0000:06:00.0/5: flavour pcisf pfnum 0 sfnum 1 type eth netdev enp6s0pf0sf1
>>                      func: hw_addr 10:22:33:44:55:99 state active
>> pci/0000:06:00.0/6: flavour pcivf pfnum 1 vfnum 2 type nvme
>>                      func: state active
>
>
>I am trying to understand how the current implementation of 'devlink port' is
>being refactored to support this new model.
>
>Today 'devlink port show' on a system with 2 port mlx5 NIC with 1 VFs created
>on each PF shows
>
>pci/0000:af:00.0/1: type eth netdev enp175s0f0np0 flavour physical port 0
>pci/0000:af:00.1/1: type eth netdev enp175s0f1np1 flavour physical port 1
>pci/0000:af:00.2/1: type eth netdev enp175s0f2np0 flavour virtual port 0
>pci/0000:af:08.2/1: type eth netdev enp175s8f2np0 flavour virtual port 0

The representor instances are not present here. They should be added.

The output would look like this:

pci/0000:af:00.0/1: type eth netdev enp175s0f0np0 flavour physical port 0 pfnum 0
pci/0000:af:00.0/2: type eth netdev enp175s0f1np1 flavour physical port 1 pfnum 1
pci/0000:af:00.0/3: type eth netdev enp175s0f0pf0vf0 flavour pcivf pfnum 0 vfnum 0
                      func: hw_addr 10:22:33:44:55:66 state active
pci/0000:af:00.0/4: type eth netdev enp175s0f1pf1vf0 flavour pcivf pfnum 1 vfnum 0
                      func: hw_addr 10:22:33:44:55:77 state active

pci/0000:af:00.2/1: type eth netdev enp175s0f2np0 flavour virtual port 0
pci/0000:af:08.2/1: type eth netdev enp175s8f2np0 flavour virtual port 0


Handle pci/0000:af:00.1 is alias for pci/0000:af:00.0. It's the same
instance of devlink object.




>
>
>Can you tell me how this will be represented in the new model?
>
>It looks like you are assigning a pfnum to physical port as well as PCI PF.
>However, i am little confused as both pfnum 0 and pfnum 1 which seem to be 2
>physical ports have the same bus/dev/func 06:00.0 and also the VF ports.

Two ports of the same nic can be under 1 PF or each can be a separate
PF. In both cases, there should be 1 devlink instances shared for both.
In my example, I mixed this up a bit, sorry about that. It should be one
of following:
1) Same PF for both ports:
  pci/0000:06:00.0/0: flavour physical pfnum 0 type eth netdev enp6s0f0np1
  pci/0000:06:00.0/1: flavour physical pfnum 0 type eth netdev enp6s0f0np2
2) Separate per-port PF:
  pci/0000:06:00.0/0: flavour physical pfnum 0 type eth netdev enp6s0f0np1
  pci/0000:06:00.0/1: flavour physical pfnum 1 type eth netdev enp6s0f1np2

  and pci/0000:06:00.1 is alias for pci/0000:06:00.0


Will fix. Thanks!

>
>Thanks
>Sridhar

Powered by blists - more mailing lists