Message-ID: <PH0PR12MB5481EA2EB1B78BC7DD92FD19DC379@PH0PR12MB5481.namprd12.prod.outlook.com>
Date:   Tue, 8 Jun 2021 05:26:11 +0000
From:   Parav Pandit <parav@...dia.com>
To:     Yunsheng Lin <linyunsheng@...wei.com>,
        "dsahern@...il.com" <dsahern@...il.com>,
        "stephen@...workplumber.org" <stephen@...workplumber.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:     Jiri Pirko <jiri@...dia.com>,
        "moyufeng@...wei.com" <moyufeng@...wei.com>,
        "linuxarm@...neuler.org" <linuxarm@...neuler.org>
Subject: RE: [PATCH RESEND iproute2-next] devlink: Add optional controller
 user input



> From: Yunsheng Lin <linyunsheng@...wei.com>
> Sent: Tuesday, June 8, 2021 8:58 AM
> 
> On 2021/6/7 19:12, Parav Pandit wrote:
> >> From: Yunsheng Lin <linyunsheng@...wei.com>
> >> Sent: Monday, June 7, 2021 4:27 PM
> >>
> 
> [..]
> 
> >>>
> >>>> 2. each PF's devlink instance has three types of port, which is
> >>>>    FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and
> >> FLAVOUR_PCI_VF(supposing I
> >>>> understand
> >>>>    port flavour correctly).
> >>>>
> >>> FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on
> >> switchdev device.
> >>
> >> If devlink instance or eswitch is in DEVLINK_ESWITCH_MODE_LEGACY
> >> mode, the FLAVOUR_PCI_{PF,VF,SF} port instance does not need to
> created?
> > No. in eswitch legacy, there are no representor netdevice or devlink ports.
> 
> It seems each devlink port instance corresponds to a netdevice.
> More specificly, the devlink instance is created in the struct pci_driver' probe
> function of a pci function, a devlink port instance is created and registered to
> that devlink instance when a netdev of that pci function is created?
> 
Yes.

> As in diagram [1], the devlink port instance(flavour FLAVOUR_PHYSICAL) for
> ctrl-0-pf0 is created when the netdev of ctrl-0-pf0 is created in the host of
> smartNIC, the devlink port instance(flavour FLAVOUR_VIRTUAL) for ctrl-0-
> pf0vfN is created when the netdev of ctrl-0-pf0vfN is created in the host of
> smartNIC, right?
> 
Ctrl-0-pf0vfN, ctrl-0-pf0 ports are eswitch ports. They are created where there is eswitch.
Usually in smartnic where eswitch is located.

> When eswitch mode is set to DEVLINK_ESWITCH_MODE_SWITCHDEV, the
> representor netdev for PF/VF in "controller_num=1" is created in the host of
> smartNIC, so is the devlink port instance(FLAVOUR_PCI_{PF,VF,SF})
> corresponding to that representor netdev just created in the host of
> smartNIC? More specificly, devlink port instance(flavour FLAVOUR_PCI_PF)
> for ctrl-1-pf0 and devlink port instance (flavour FLAVOUR_PCI_VF)for ctrl-1-
> pf0vfN?
> 
I do not know what you mean by "host of smartnic". But as clarified in diagram and above,
They are created on the eswitch device.

> When "controller_num=1" is plugged to a server, the server host creates
> devlink instance and devlink port instance in the host of server as similar as
> the ctrl-0-pf0 and ctrl-0-pf0vfN in the host of smartNIC?
Since eswitch is not in the controller=1, controller 1 is just hosting PCI PF, VF, SF functions.
Either it has PHYSICAL or VIRTUAL ports along with its netdevice and devlink instance.

> >> Or devlink instance does exist in the host corresponding to
> >> "controller_num=1", but the mode of that devlink instance is
> >> DEVLINK_ESWITCH_MODE_LEGACY in diagram [1]?
> > As you can see that eswitch is located only on controller=0.
> > This eswitch is serving PF, VF, SFs of controller=1 + controlloler=0 as well.
> 
> How do we decide where eswitch is located? through some fw/hw
> configuration?
This is usually decided by the hardware.

> 
> It seems if the eswitch is enabled on "controller=1", that is a nested eswitch
> too, which you mentioned below?
Yes.

> 
> >>
> >> Also, eswitch mode can only be set on the devlink instance
> >> corresponding to PF, but not for VF/SF(supposing that VF/SF could
> >> have it's own devlink instance too), right?
> > Yes. Eswitch can be located on the VF too. Mlx5 driver doesn't have it yet
> on VF.
> > This may be some nested eswitch in future. I do not know when.
> >
> >> by the network/sysadmin.
> >>> While devlink instance of a given PF,VF,SF is managed by the user of
> >>> such
> >> function itself.
> >>
> >> 'devlink port function' means "struct devlink_port", right?
> > 'function' is the object managing the function connected on the otherside
> of this port.
> > This includes its hw_addr, rate, state, operational state.
> 
> Does "other side of this port" means the pci function that is most likely have
> been passed through to a VM?
> 
Yes.

> "devlink port" without the "function" represents the representor netdev on
> the host where eswitch is located?
> 
Yes.

> >
> >> It seems 'devlink port function' in the host is representing a VF
> >> when devlink instance of that VF is in the VM?
> > Right.
> >>
> >>> For example when a VF is mapped to a VM, devlink instance of this VF
> >> resides in the VM managed by the guest VM.
> >>
> >> Does the user in VM really care about devlink info or configuration
> >> when network/sysadmin has configured the VF through 'devlink port
> function'
> >> in the host?
> > Yes. devlink instance offers many knobs in uniform way on PF, VF, SF.
> > They are in use in mlx5 for devlink params, reload, net ns.
> 
> "net ns" refer to "net namespace", right?
Yes.

> I am not sure how devlink instance related to net namespace yet.
> I thought devlink is not limited to networking, it can be used in other pcie
> device other than ethernet device?
It can be. And it can be used for networking devices too and hence it is attached to single net ns.

Powered by blists - more mailing lists