Date:   Tue, 8 Jun 2021 11:27:51 +0800
From:   Yunsheng Lin <linyunsheng@...wei.com>
To:     Parav Pandit <parav@...dia.com>,
        "dsahern@...il.com" <dsahern@...il.com>,
        "stephen@...workplumber.org" <stephen@...workplumber.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:     Jiri Pirko <jiri@...dia.com>,
        "moyufeng@...wei.com" <moyufeng@...wei.com>,
        "linuxarm@...neuler.org" <linuxarm@...neuler.org>
Subject: Re: [PATCH RESEND iproute2-next] devlink: Add optional controller
 user input

On 2021/6/7 19:12, Parav Pandit wrote:
>> From: Yunsheng Lin <linyunsheng@...wei.com>
>> Sent: Monday, June 7, 2021 4:27 PM
>>

[..]

>>>
>>>> 2. each PF's devlink instance has three types of port, which is
>>>>    FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and
>> FLAVOUR_PCI_VF(supposing I
>>>> understand
>>>>    port flavour correctly).
>>>>
>>> FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on
>> switchdev device.
>>
>> If devlink instance or eswitch is in DEVLINK_ESWITCH_MODE_LEGACY mode,
>> the FLAVOUR_PCI_{PF,VF,SF} port instance does not need to created?
> No. in eswitch legacy, there are no representor netdevice or devlink ports.

It seems each devlink port instance corresponds to a netdevice.
More specificly, the devlink instance is created in the
struct pci_driver' probe function of a pci function, a devlink
port instance is created and registered to that devlink instance
when a netdev of that pci function is created?

As in diagram [1], the devlink port instance(flavour FLAVOUR_PHYSICAL)
for ctrl-0-pf0 is created when the netdev of ctrl-0-pf0 is created in
the host of smartNIC, the devlink port instance(flavour FLAVOUR_VIRTUAL)
for ctrl-0-pf0vfN is created when the netdev of ctrl-0-pf0vfN is created
in the host of smartNIC, right?

When eswitch mode is set to DEVLINK_ESWITCH_MODE_SWITCHDEV, the representor
netdev for PF/VF in "controller_num=1" is created in the host of smartNIC,
so is the devlink port instance(FLAVOUR_PCI_{PF,VF,SF}) corresponding to that
representor netdev just created in the host of smartNIC? More specificly,
devlink port instance(flavour FLAVOUR_PCI_PF) for ctrl-1-pf0 and devlink port
instance (flavour FLAVOUR_PCI_VF)for ctrl-1-pf0vfN?

When "controller_num=1" is plugged to a server, the server host creates
devlink instance and devlink port instance in the host of server as
similar as the ctrl-0-pf0 and ctrl-0-pf0vfN in the host of smartNIC?

> 
>>
>>>
>>>> If I understand above correctly, all ports in the same devlink
>>>> instance should have the same controller id, right? If yes, why not
>>>> put the controller id in the devlink instance?
>>> Need not be. All PCI_{PF,VF,SF} can have controller id different for
>> different controllers.
>>
>> The point is that two VF from different PF may be in the different host, all VF
>> of a specific PF need to be in the same host, right?
>> otherwise it may break PCI enumeration process?
>>
> Sure. VFs belong to PF, PF belong to controller, controller is plugged into a host root complex.
> 
>> If yes, as PCI_{PF,VF,SF} belongs to eswitch (representor) side on switchdev
>> device(which means PCI_{PF,VF,SF} port instance is in the same host, as the
>> host corresponding to "controller_num=0" in diagram [1]), so it seems all the
>> PCI_{PF,VF,SF} of a specific PF should have the same controller id, 
> Yes.
> 
>> and using
>> a controller id of the devlink instance in "controller_num=0" in diagram [1]
>> seems enough?
> Yes.
> 
>>
>>> Usually each multi-host is a different controller.
>>> Refer to this diagram [1] and detailed description.
>>
>> devlink instance does not exist in the host corresponding to
>> "controller_num=1" in diagram [1]?
> Devlink instance do exist for controller=1 related PCI PF,VF,SF devices when those functions are plugged in the host.

> 
>> Or devlink instance does exist in the host corresponding to
>> "controller_num=1", but the mode of that devlink instance is
>> DEVLINK_ESWITCH_MODE_LEGACY in diagram [1]?
> As you can see that eswitch is located only on controller=0.
> This eswitch is serving PF, VF, SFs of controller=1 + controlloler=0 as well.

How do we decide where eswitch is located? through some fw/hw
configuration?

It seems if the eswitch is enabled on "controller=1", that is
a nested eswitch too, which you mentioned below?

>>
>> Also, eswitch mode can only be set on the devlink instance corresponding to
>> PF, but not for VF/SF(supposing that VF/SF could have it's own devlink
>> instance too), right?
> Yes. Eswitch can be located on the VF too. Mlx5 driver doesn't have it yet on VF.
> This may be some nested eswitch in future. I do not know when.
> 
>> by the network/sysadmin.
>>> While devlink instance of a given PF,VF,SF is managed by the user of such
>> function itself.
>>
>> 'devlink port function' means "struct devlink_port", right?
> 'function' is the object managing the function connected on the otherside of this port.
> This includes its hw_addr, rate, state, operational state.

Does "other side of this port" means the pci function that is most
likely have been passed through to a VM?

"devlink port" without the "function" represents the representor
netdev on the host where eswitch is located?

> 
>> It seems 'devlink port function' in the host is representing a VF when devlink
>> instance of that VF is in the VM?
> Right.
>>
>>> For example when a VF is mapped to a VM, devlink instance of this VF
>> resides in the VM managed by the guest VM.
>>
>> Does the user in VM really care about devlink info or configuration when
>> network/sysadmin has configured the VF through 'devlink port function'
>> in the host?
> Yes. devlink instance offers many knobs in uniform way on PF, VF, SF.
> They are in use in mlx5 for devlink params, reload, net ns.

"net ns" refer to "net namespace", right?
I am not sure how devlink instance related to net namespace yet.
I thought devlink is not limited to networking, it can be used in
other pcie device other than ethernet device?

> 
>> which devlink info or configuration does user need to query or configure in a
>> VM?
> Usually not much.
> Few examples that mlx5 users do with devlink instance of VF in a VM are, devlink params, devlink reload, board info. Health reporters, health recovery to name few.
> 

Powered by blists - more mailing lists