Message-ID: <20210609094036.7557bd83@kicinski-fedora-PC1C0HJN.hsd1.ca.comcast.net>
Date:   Wed, 9 Jun 2021 09:40:36 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     Yunsheng Lin <linyunsheng@...wei.com>
Cc:     moyufeng <moyufeng@...wei.com>,
        Jakub Kicinski <jakub.kicinski@...ronome.com>,
        Jiri Pirko <jiri@...nulli.us>,
        Parav Pandit <parav@...lanox.com>,
        Or Gerlitz <gerlitz.or@...il.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "michal.lkml@...kovi.net" <michal.lkml@...kovi.net>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        Jiri Pirko <jiri@...lanox.com>,
        Salil Mehta <salil.mehta@...wei.com>,
        "lipeng (Y)" <lipeng321@...wei.com>,
        Guangbin Huang <huangguangbin2@...wei.com>,
        <shenjian15@...wei.com>, "chenhao (DY)" <chenhao288@...ilicon.com>,
        Jiaran Zhang <zhangjiaran@...wei.com>,
        "linuxarm@...neuler.org" <linuxarm@...neuler.org>
Subject: Re: [RFC net-next 0/8] Introducing subdev bus and devlink extension

On Wed, 9 Jun 2021 17:16:06 +0800 Yunsheng Lin wrote:
> On 2021/6/9 1:29, Jakub Kicinski wrote:
> > On Tue, 8 Jun 2021 20:10:37 +0800 Yunsheng Lin wrote:  
> >> Afer discussion with Parav in other thread, I undersood it was the current
> >> practice, but I am not sure I understand why it is current *best* practice.
> >>
> >> If we allow all PF of a ASCI to register to the same devlink instance, does
> >> it not make sense that all VF under one PF also register to the same devlink
> >> instance that it's PF is registering to when they are in the same host?
> >>
> >> For eswitch legacy mode, whether VF and PF are the same host or not, the VF
> >> can also provide the serial number of a ASIC to register to the devlink instance,
> >> if that devlink instance does not exist yet, just create that devlink instance
> >> according to the serial number, just like PF does.
> >>
> >> For eswitch DEVLINK_ESWITCH_MODE_SWITCHDEV mode, the flavour type for devlink
> >> port instance representing the netdev of VF function is FLAVOUR_VIRTUAL, the
> >> flavour type for devlink port instance representing the representor netdev of
> >> VF is FLAVOUR_PCI_VF, which are different type, so they can register to the same
> >> devlink instance even when both of the devlink port instance is in the same host?
> >>
> >> Is there any reason why VF use its own devlink instance?  
> > 
> > Primary use case for VFs is virtual environments where guest isn't
> > trusted, so tying the VF to the main devlink instance, over which guest
> > should have no control is counter productive.  
> 
> The security is mainly about VF using in container case, right?
> Because VF using in VM, it is different host, it means a different devlink
> instance for VF, so there is no security issue for VF using in VM case?
> But it might not be the case for VF using in container?

How do you differentiate from the device perspective VF being assigned
to the host vs VM? Presumably PFs and VFs have a similar API to talk to
the FW, if VF can "join" the devlink instance of the PF that'd suggest
to me it has access to privileged FW commands.

> Also I read about the devlink disscusion betwwen you and jiri in [1]:
> "I think we agree that all objects of an ASIC should be under one
> devlink instance, the question remains whether both ends of the pipe
> for PCI devices (subdevs or not) should appear under ports or does the
> "far end" (from ASICs perspective)/"host end" get its own category."
> 
> I am not sure if there is already any conclusion about the latter part
> (I did not find the conclusion in that thread)?
> 
> "far end" (from ASICs perspective)/"host end" means PF/VF, right?
> Which seems to correspond to port flavor of FLAVOUR_PHYSICAL and
> FLAVOUR_VIRTUAL if we try to represent PF/VF using devlink port
> instance?

No, no, PHYSICAL is a physical port on the adapter, like an SFP port.
There wasn't any conclusion to that discussion. Mellanox views devlink
ports as eswitch ports, I view them as device ports which is hard to
reconcile.

> It seems the conclusion is very important to our disscusion in this
> thread, as we are trying to represent PF/VF as devlink port instance
> in this thread(at least that is what I think, hns3 does not support
> eswitch SWITCHDEV mode yet).
> 
> Also, there is a "switch_id" concept from jiri's example, which seems
> to be not implemented yet?
> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
> 
> 1. https://lore.kernel.org/netdev/20190304164007.7cef8af9@cakuba.netronome.com/t/
> 
> >> I am not sure I understand what does it mean by "devlink instances with
> >> multiple names"?
> >>
> >> Does that mean whenever a devlink port instance is registered to a devlink
> >> instance, that devlink instance get a new name according to the PCI device
> >> which the just registered devlink port instance corresponds to?  
> > 
> > Not devlink port, new PCI device. Multiple ports may reside on the same
> > PCI function, some ports don't have a function (e.g. Ethernet ports).  
> 
> Multiple ports on the same mainly PCI function means subfunction from mlx,
> right?

Not necessarily, there are older devices out there (older NFPs, mlx4)
which have one PF which is logically divided by the driver to service
multiple ports.

> “some ports don't have a function (e.g. Ethernet ports)” does not seem
> exist yet? For now devlink port instance of FLAVOUR_PHYSICAL represents
> both PF and Ethernet ports?

It does. I think Mellanox cards are incapable of divorcing PFs from
Ethernet ports, but the NFP driver represents the Ethernet port/SFP 
as one netdev and devlink port (PHYSICAL) and the host port by another
netdev and devlink port (PCI_PF). Which allows forwarding frames between
PFs and between Ethernet ports directly (again, something not supported
efficiently by simpler cards, but supported by NFPs).

Powered by blists - more mailing lists