Date:   Thu, 21 Nov 2019 12:53:21 +0800
From:   Jason Wang <jasowang@...hat.com>
To:     Jason Gunthorpe <jgg@...pe.ca>,
        "Michael S. Tsirkin" <mst@...hat.com>
Cc:     Parav Pandit <parav@...lanox.com>,
        Jeff Kirsher <jeffrey.t.kirsher@...el.com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        Dave Ertman <david.m.ertman@...el.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "nhorman@...hat.com" <nhorman@...hat.com>,
        "sassmann@...hat.com" <sassmann@...hat.com>,
        Kiran Patil <kiran.patil@...el.com>,
        Alex Williamson <alex.williamson@...hat.com>,
        "Bie, Tiwei" <tiwei.bie@...el.com>
Subject: Re: [net-next v2 1/1] virtual-bus: Implementation of Virtual Bus


On 2019/11/21 上午12:45, Jason Gunthorpe wrote:
> On Wed, Nov 20, 2019 at 09:57:17AM -0500, Michael S. Tsirkin wrote:
>> On Wed, Nov 20, 2019 at 10:30:54AM -0400, Jason Gunthorpe wrote:
>>> On Wed, Nov 20, 2019 at 08:43:20AM -0500, Michael S. Tsirkin wrote:
>>>> On Wed, Nov 20, 2019 at 09:03:19AM -0400, Jason Gunthorpe wrote:
>>>>> On Wed, Nov 20, 2019 at 02:38:08AM -0500, Michael S. Tsirkin wrote:
>>>>>>>> I don't think that extends as far as actively encouraging userspace
>>>>>>>> drivers poking at hardware in a vendor specific way.
>>>>>>> Yes, it does, if you can implement your user space requirements using
>>>>>>> vfio then why do you need a kernel driver?
>>>>>> People's requirements differ. You are happy with just pass through a VF
>>>>>> you can already use it. Case closed. There are enough people who have
>>>>>> a fixed userspace that people have built virtio accelerators,
>>>>>> now there's value in supporting that, and a vendor specific
>>>>>> userspace blob is not supporting that requirement.
>>>>> I have no idea what you are trying to explain here. I'm not advocating
>>>>> for vfio pass through.
>>>> You seem to come from an RDMA background, used to userspace linking to
>>>> vendor libraries to do basic things like push bits out on the network,
>>>> because users live on the performance edge and rebuild their
>>>> userspace often anyway.
>>>>
>>>> Lots of people are not like that, they would rather have the
>>>> vendor-specific driver live in the kernel, with userspace being
>>>> portable, thank you very much.
>>> You are actually proposing a very RDMA like approach with a split
>>> kernel/user driver design. Maybe the virtio user driver will turn out
>>> to be 'portable'.
>>>
>>> Based on the last 20 years of experience, the kernel component has
>>> proven to be the larger burden and drag than the userspace part. I
>>> think the high interest in DPDK, SPDK and others show this is a common
>>> principle.
>> And I guess the interest in BPF shows the opposite?
> There is room for both, I wouldn't discount either approach entirely
> out of hand.
>
>>> At the very least for new approaches like this it makes alot of sense
>>> to have a user space driver until enough HW is available that a
>>> proper, well thought out kernel side can be built.
>> But hardware is available, driver has been posted by Intel.
>> Have you looked at that?
> I'm not sure pointing at that driver is so helpful, it is very small
> and mostly just reflects virtio ops into some undocumented register
> pokes.


What do you expect to see then? The IFC driver is sufficient for 
demonstrating the design and implementation of the framework that is a 
vDPA driver. If you care about a better management API for mdev, we can 
discuss but it should be another topic which should not block this series.


>
> There is no explanation at all for the large scale architecture
> choices:


Most of the parts have been explained more or less in the cover letter.


>   - Why vfio


In cover letter it explains that userspace driver + vhost mdev is the 
goal. And VFIO is the most popular interface for developing userspace 
drivers. Having vendor specific userspace driver framework is possible 
but would be a pain for management and qemu.


>   - Why mdev without providing a device IOMMU


This is a question for mdev not directly related to the series . Either 
bus IOMMU or device IOMMU (as vGPU already did) is supported.


>   - Why use GUID lifecycle management for singlton function PF/VF
>     drivers


It was just because it's the only existed interface right now, and 
management has been taught to use this interface.


>   - Why not use devlink


Technically it's possible. But for explanation, it's just because I 
don't get any question before start the draft the new version. I can add 
this in the cover letter of next version.


>   - Why not use vfio-pci with a userspace driver


In cover letter, it explains that the series is for kernel virtio driver.


>
> These are legitimate questions and answers like "because we like it
> this way"


Where are stuffs like this?


>   or "this is how the drivers are written today" isn't very
> satisfying at all.


If you are talking about devlink + mdev. I would say for now, you're 
welcome to develop devlink based lifecycle for mdev.  But if you want to 
discuss devlink support for each type of devices, it's obvious not the 
correct place.


>
>>> For instance, this VFIO based approach might be very suitable to the
>>> intel VF based ICF driver, but we don't yet have an example of non-VF
>>> HW that might not be well suited to VFIO.


What's the reason that causes your HW not suited to VFIO? Mdev had 
already supported device IOMMU partially, let's just improve it if it 
doesn't meet your requirement. Or are there any fundamental barriers there?


>> I don't think we should keep moving the goalposts like this.
> It is ABI, it should be done as best we can as we have to live with it
> for a long time. Right now HW is just starting to come to market with
> VDPA and it feels rushed to design a whole subsystem style ABI around
> one, quite simplistic, driver example.


Well, I know there could be some special features in your hardware, 
let's just discuss here to seek a solution instead of keep saying "your 
framework does not fit our case" without any real details.


>
>> If people write drivers and find some infrastruture useful,
>> and it looks more or less generic on the outset, then I don't
>> see why it's a bad idea to merge it.
> Because it is userspace ABI, caution is always justified when defining
> new ABI.


Well, if you read vhost-mdev patch, you will see it doesn't invent any 
userspace ABI. VFIO ABI is completely followed there.

Thanks


>
> Jason
>

Powered by blists - more mailing lists