netdev - RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <AM0PR05MB48668E29F1811F496602079ED17B0@AM0PR05MB4866.eurprd05.prod.outlook.com>
Date:   Fri, 8 Nov 2019 01:49:09 +0000
From:   Parav Pandit <parav@...lanox.com>
To:     Jakub Kicinski <jakub.kicinski@...ronome.com>
CC:     "alex.williamson@...hat.com" <alex.williamson@...hat.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Saeed Mahameed <saeedm@...lanox.com>,
        "kwankhede@...dia.com" <kwankhede@...dia.com>,
        "leon@...nel.org" <leon@...nel.org>,
        "cohuck@...hat.com" <cohuck@...hat.com>,
        Jiri Pirko <jiri@...lanox.com>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        Or Gerlitz <gerlitz.or@...il.com>
Subject: RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support



> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@...ronome.com>
> Sent: Thursday, November 7, 2019 7:16 PM
> To: Parav Pandit <parav@...lanox.com>
> Cc: alex.williamson@...hat.com; davem@...emloft.net;
> kvm@...r.kernel.org; netdev@...r.kernel.org; Saeed Mahameed
> <saeedm@...lanox.com>; kwankhede@...dia.com; leon@...nel.org;
> cohuck@...hat.com; Jiri Pirko <jiri@...lanox.com>; linux-
> rdma@...r.kernel.org; Or Gerlitz <gerlitz.or@...il.com>
> Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
> 
> On Thu, 7 Nov 2019 20:52:29 +0000, Parav Pandit wrote:
> > > On Thu,  7 Nov 2019 10:04:48 -0600, Parav Pandit wrote:
> > > > Mellanox sub function capability allows users to create several
> > > > hundreds of networking and/or rdma devices without depending on
> > > > PCI SR-
> > > IOV support.
> > >
> > > You call the new port type "sub function" but the devlink port flavour is
> mdev.
> > >
> > Sub function is the internal driver structure. The abstract entity at user and
> stack level is mdev.
> > Hence the port flavour is mdev.
> 
> FWIW I agree mdev as flavour seems like the right choice.
>
Ok.
 
> > > As I'm sure you remember you nacked my patches exposing NFP's PCI
> > > sub functions which are just regions of the BAR without any mdev
> > > capability. Am I in the clear to repost those now? Jiri?
> > >
> > For sure I didn't nack it. :-)
> 
> Well, maybe the word "nack" wasn't exactly used :)
> 
> > What I remember discussing offline/mailing list is
> > (a) exposing mdev/sub fuctions as devlink sub ports is not so good
> > abstraction
> > (b) user creating/deleting eswitch sub ports would be hard to fit in
> > the whole usage model
> 
> Okay, so I can repost the "basic" sub functions?
> 
I think so. Would you like post on top of this series as port flavour etc would come by default?
Also there is vfio/mdev dependency exist in this series...

> > > > Overview:
> > > > ---------
> > > > Mellanox ConnectX sub functions are exposed to user as a mediated
> > > > device (mdev) [2] as discussed in RFC [3] and further during
> > > > netdevconf0x13 at [4].
> > > >
> > > > mlx5 mediated device (mdev) enables users to create multiple
> > > > netdevices and/or RDMA devices from single PCI function.
> > > >
> > > > Each mdev maps to a mlx5 sub function.
> > > > mlx5 sub function is similar to PCI VF. However it doesn't have
> > > > its own PCI function and MSI-X vectors.
> > > >
> > > > mlx5 mdevs share common PCI resources such as PCI BAR region,
> > > > MSI-X interrupts.
> > > >
> > > > Each mdev has its own window in the PCI BAR region, which is
> > > > accessible only to that mdev and applications using it.
> > > >
> > > > Each mlx5 sub function has its own resource namespace for RDMA
> resources.
> > > >
> > > > mdevs are supported when eswitch mode of the devlink instance is
> > > > in switchdev mode described in devlink documentation [5].
> > >
> > > So presumably the mdevs don't spawn their own devlink instance
> > > today, but once mapped via VIRTIO to a VM they will create one?
> > >
> > mdev doesn't spawn the devlink instance today when mdev is created by
> user, like PCI.
> > When PCI bus driver enumerates and creates PCI device, there isn't a
> devlink instance for it.
> >
> > But, mdev's devlink instance is created when mlx5_core driver binds to the
> mdev device.
> > (again similar to PCI, when mlx5_core driver binds to PCI, its devlink
> instance is created ).
> >
> > I should have put the example in patch-15 which creates/deletes devlink
> instance of mdev.
> > I will revise the commit log of patch-15 to include that.
> > Good point.
> 
> Thanks.
> 
> > > It could be useful to specify.
> > >
> > Yes, its certainly useful. I missed to put the example in commit log of
> patch-15.
> >
> > > > Network side:
> > > > - By default the netdevice and the rdma device of mlx5 mdev cannot
> > > > send or receive any packets over the network or to any other mlx5
> mdev.
> > >
> > > Does this mean the frames don't fall back to the repr by default?
> > Probably I wasn't clear.
> > What I wanted to say is, that frames transmitted by mdev's netdevice and
> rdma devices don't go to network.
> > These frames goes to representor device.
> > User must configure representor to send/receive/steer traffic to mdev.
> 
> 👍