Date:   Fri, 8 Nov 2019 13:34:35 -0700
From:   Alex Williamson <alex.williamson@...hat.com>
To:     Jason Gunthorpe <jgg@...pe.ca>
Cc:     Jakub Kicinski <jakub.kicinski@...ronome.com>,
        Parav Pandit <parav@...lanox.com>,
        Jiri Pirko <jiri@...nulli.us>,
        David M <david.m.ertman@...el.com>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Saeed Mahameed <saeedm@...lanox.com>,
        "kwankhede@...dia.com" <kwankhede@...dia.com>,
        "leon@...nel.org" <leon@...nel.org>,
        "cohuck@...hat.com" <cohuck@...hat.com>,
        Jiri Pirko <jiri@...lanox.com>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        Or Gerlitz <gerlitz.or@...il.com>
Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

On Fri, 8 Nov 2019 16:12:53 -0400
Jason Gunthorpe <jgg@...pe.ca> wrote:

> On Fri, Nov 08, 2019 at 11:12:38AM -0800, Jakub Kicinski wrote:
> > On Fri, 8 Nov 2019 15:40:22 +0000, Parav Pandit wrote:  
> > > > The new intel driver has been having a very similar discussion about how to
> > > > model their 'multi function device' ie to bind RDMA and other drivers to a
> > > > shared PCI function, and I think that discussion settled on adding a new bus?
> > > > 
> > > > Really these things are all very similar, it would be nice to have a clear
> > > > methodology on how to use the device core if a single PCI device is split by
> > > > software into multiple different functional units and attached to different
> > > > driver instances.
> > > > 
> > > > Currently there is alot of hacking in this area.. And a consistent scheme
> > > > might resolve the ugliness with the dma_ops wrappers.
> > > > 
> > > > We already have the 'mfd' stuff to support splitting platform devices, maybe
> > > > we need to create a 'pci-mfd' to support splitting PCI devices?
> > > > 
> > > > I'm not really clear how mfd and mdev relate, I always thought mdev was
> > > > strongly linked to vfio.
> > > >  
> > >
> > > Mdev at beginning was strongly linked to vfio, but as I mentioned
> > > above it is addressing more use case.
> > > 
> > > I observed that discussion, but was not sure of extending mdev further.
> > > 
> > > One way to do for Intel drivers to do is after series [9].
> > > Where PCI driver says, MDEV_CLASS_ID_I40_FOO
> > > RDMA driver mdev_register_driver(), matches on it and does the probe().  
> > 
> > Yup, FWIW to me the benefit of reusing mdevs for the Intel case vs
> > muddying the purpose of mdevs is not a clear trade off.  
> 
> IMHO, mdev has amdev_parent_ops structure clearly intended to link it
> to vfio, so using a mdev for something not related to vfio seems like
> a poor choice.

Unless there's some opposition, I'm intended to queue this for v5.5:

https://www.spinics.net/lists/kvm/msg199613.html

mdev has started out as tied to vfio, but at it's core, it's just a
device life cycle infrastructure with callbacks between bus drivers
and vendor devices.  If virtio is on the wrong path with the above
series, please speak up.  Thanks,

Alex

 
> I suppose this series is the start and we will eventually see the
> mlx5's mdev_parent_ops filled in to support vfio - but *right now*
> this looks identical to the problem most of the RDMA capable net
> drivers have splitting into a 'core' and a 'function'
> 
> > IMHO MFD should be of more natural use for Intel, since it's about
> > providing different functionality rather than virtual slices of the
> > same device.  
> 
> I don't think the 'different functionality' should matter much. 
> 
> Generally these multi-function drivers are build some some common
> 'core' language like queues interrupts, BAR space, etc and then these
> common things can be specialized into netdev, rdma, scsi, etc. So we
> see a general rough design with a core layer managing the raw HW then
> drivers on top of that (including netdev) using that API.
> 
> The actual layering doesn't come through in the driver model,
> generally people put all the core stuff in with the netdev and then
> try and shuffle the netdev around as the 'handle' for that core API.
> 
> These SFs are pretty similar in that the core physical driver
> continues to provide some software API support to the SF children (at
> least for mlx it is a small API)
> 
> For instance mdev has no generic way to learn the BAR struct
> resources, so there is some extra API around the side that does this -
> in this series it is done by hackily co-opting the drvdata to
> something owned by the struct device instead of the device_driver and
> using that to access the API surface on 'struct mlx5_sf *', which
> includes the BAR info and so forth.
> 
> This is probably the main difference from MFD. At least the few
> drivers I looked at, did not try and expose an SW API from the 'core'
> to the 'part', everything was usual generic driver resource stuff.
> 
> Jason

Powered by blists - more mailing lists