Message-ID: <PH0PR12MB5481FB8528A90E34FA3578C1DC389@PH0PR12MB5481.namprd12.prod.outlook.com>
Date:   Mon, 7 Jun 2021 11:12:42 +0000
From:   Parav Pandit <parav@...dia.com>
To:     Yunsheng Lin <linyunsheng@...wei.com>,
        "dsahern@...il.com" <dsahern@...il.com>,
        "stephen@...workplumber.org" <stephen@...workplumber.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
CC:     Jiri Pirko <jiri@...dia.com>,
        "moyufeng@...wei.com" <moyufeng@...wei.com>,
        "linuxarm@...neuler.org" <linuxarm@...neuler.org>
Subject: RE: [PATCH RESEND iproute2-next] devlink: Add optional controller
 user input



> From: Yunsheng Lin <linyunsheng@...wei.com>
> Sent: Monday, June 7, 2021 4:27 PM
> 
> On 2021/6/7 14:10, Parav Pandit wrote:
> >> From: Yunsheng Lin <linyunsheng@...wei.com>
> >> Sent: Monday, June 7, 2021 9:01 AM
> >>
> >> On 2021/6/6 15:10, Parav Pandit wrote:
> >>> Hi Yunsheng,
> >>>
> >>>> From: Yunsheng Lin <linyunsheng@...wei.com>
> >>>> Sent: Friday, June 4, 2021 7:05 AM
> >>>>
> >>>> On 2021/6/3 19:19, Parav Pandit wrote:
> >>>>> A user optionally provides the external controller number when
> >>>>> user wants to create devlink port for the external controller.
> >>>>
> >>>> Hi, Parav
> >>>>    I was planing to use controller id to solve the devlink instance
> >>>> representing problem for multi-function which shares common
> >>>> resource in the same ASIC, see [1].
> >>>>
> >>>> It seems the controller id used here is to differentiate the
> >>>> function used in different host?
> >>>>
> >>> That’s correct. Controller holds one or more PCI functions (PF,VF,SF).
> >>
> >> I am not sure I understand the exact usage of controller and why
> >> controller id is in "devlink_port_*_attrs".
> >>
> >> Let's consider a simplified case where there is two PF(supposing both
> >> have VF enabled), and each PF has different controller and each PF
> >> corresponds to a different physical port(Or it is about multi-host
> >> case multi PF may sharing the same physical port?):
> > Typically single host with two PFs have their own physical ports.
> > Multi-host with two PFs, on each host they share respective physical ports.
> >
> > Single host:
> > Pf0.physical_port = p0
> > Pf1.physical_port = p1.
> >
> > Multi-host (two) host setup
> > H1.pf0.phyical_port = p0.
> > H1.pf1.phyical_port = p1.
> > H2.pf0.phyical_port = p0.
> > H2.pf1.phyical_port = p1.
> 
> Multi-host (two) host setup with separate physical port for each host:
> H1.pf0.phyical_port = p0
> H2.pf0.phyical_port = p1
> 
> Does above use case make sense for mlx, it seems a common case for our
> internal use.
It is mlx5 use case too.

> 
> >
> >> 1. I suppose each PF has it's devlink instance for mlx case(I suppose each
> >>    VF can not have it's own devlink instance for VF shares the same physical
> >>    port with PF, right?).
> > VF and SF ports are of flavour VIRTUAL.
> 
> Which devlink instance does the flavour VIRTUAL port instance for VF and SF
> is registered to?
To the devlink instance of the VF/SF.

> Does it mean VF has it's own devlink instance in VM when it is passed a VM,
> and flavour VIRTUAL port instance for that VF is registered to that devlink
> instance in the VM too？
Yes.

> 
> Even in the same host as PF, the VF also has it's own devlink instance?
Yes.

> 
> >
> >> 2. each PF's devlink instance has three types of port, which is
> >>    FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and
> FLAVOUR_PCI_VF(supposing I
> >> understand
> >>    port flavour correctly).
> >>
> > FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on
> switchdev device.
> 
> If devlink instance or eswitch is in DEVLINK_ESWITCH_MODE_LEGACY mode,
> the FLAVOUR_PCI_{PF,VF,SF} port instance does not need to created?
No. in eswitch legacy, there are no representor netdevice or devlink ports.

> 
> >
> >> If I understand above correctly, all ports in the same devlink
> >> instance should have the same controller id, right? If yes, why not
> >> put the controller id in the devlink instance?
> > Need not be. All PCI_{PF,VF,SF} can have controller id different for
> different controllers.
> 
> The point is that two VF from different PF may be in the different host, all VF
> of a specific PF need to be in the same host, right?
> otherwise it may break PCI enumeration process?
> 
Sure. VFs belong to PF, PF belong to controller, controller is plugged into a host root complex.

> If yes, as PCI_{PF,VF,SF} belongs to eswitch (representor) side on switchdev
> device(which means PCI_{PF,VF,SF} port instance is in the same host, as the
> host corresponding to "controller_num=0" in diagram [1]), so it seems all the
> PCI_{PF,VF,SF} of a specific PF should have the same controller id, 
Yes.

> and using
> a controller id of the devlink instance in "controller_num=0" in diagram [1]
> seems enough?
Yes.

> 
> > Usually each multi-host is a different controller.
> > Refer to this diagram [1] and detailed description.
> 
> devlink instance does not exist in the host corresponding to
> "controller_num=1" in diagram [1]?
Devlink instance do exist for controller=1 related PCI PF,VF,SF devices when those functions are plugged in the host.

> Or devlink instance does exist in the host corresponding to
> "controller_num=1", but the mode of that devlink instance is
> DEVLINK_ESWITCH_MODE_LEGACY in diagram [1]?
As you can see that eswitch is located only on controller=0.
This eswitch is serving PF, VF, SFs of controller=1 + controlloler=0 as well.
> 
> Also, eswitch mode can only be set on the devlink instance corresponding to
> PF, but not for VF/SF(supposing that VF/SF could have it's own devlink
> instance too), right?
Yes. Eswitch can be located on the VF too. Mlx5 driver doesn't have it yet on VF.
This may be some nested eswitch in future. I do not know when.

> by the network/sysadmin.
> > While devlink instance of a given PF,VF,SF is managed by the user of such
> function itself.
> 
> 'devlink port function' means "struct devlink_port", right?
'function' is the object managing the function connected on the otherside of this port.
This includes its hw_addr, rate, state, operational state.

> It seems 'devlink port function' in the host is representing a VF when devlink
> instance of that VF is in the VM?
Right.
> 
> > For example when a VF is mapped to a VM, devlink instance of this VF
> resides in the VM managed by the guest VM.
> 
> Does the user in VM really care about devlink info or configuration when
> network/sysadmin has configured the VF through 'devlink port function'
> in the host?
Yes. devlink instance offers many knobs in uniform way on PF, VF, SF.
They are in use in mlx5 for devlink params, reload, net ns.

> which devlink info or configuration does user need to query or configure in a
> VM?
Usually not much.
Few examples that mlx5 users do with devlink instance of VF in a VM are, devlink params, devlink reload, board info. Health reporters, health recovery to name few.

Powered by blists - more mailing lists