netdev - Re: [pull request][net-next V10 00/14] Add mlx5 subfunction support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 25 Jan 2021 19:13:55 -0400
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Edwin Peer <edwin.peer@...adcom.com>
CC:     Parav Pandit <parav@...dia.com>, Saeed Mahameed <saeed@...nel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        netdev <netdev@...r.kernel.org>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        Alexander Duyck <alexander.duyck@...il.com>,
        Sridhar Samudrala <sridhar.samudrala@...el.com>,
        David Ahern <dsahern@...nel.org>,
        Kiran Patil <kiran.patil@...el.com>,
        Jacob Keller <jacob.e.keller@...el.com>,
        "Ertman, David M" <david.m.ertman@...el.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Saeed Mahameed <saeedm@...dia.com>
Subject: Re: [pull request][net-next V10 00/14] Add mlx5 subfunction support

On Mon, Jan 25, 2021 at 01:23:04PM -0800, Edwin Peer wrote:
> On Mon, Jan 25, 2021 at 12:41 PM Jason Gunthorpe <jgg@...dia.com> wrote:
> 
> > > That's an implementation decision. Nothing mandates that the state has
> > > to physically exist in the same structure, only that reads and writes
> > > are appropriately responded to.
> >
> > Yes, PCI does mandate this, you can't store the data on the other side
> > of the PCI link, and if you can't cross the PCI link that only leaves
> > on die/package memory resources.
> 
> Going off device was not what I was suggesting at all. I meant the
> data doesn't necessarily need to be stored in the same physical
> layout.

It doesn't change anything, every writable bit must still be stored
on-die SRAM. You can compute the minimum by summing all writable and
read-reporting bits in the standard SRIOV config space.

Every bit used for SRIOV is a bit that couldn't be used to improve
device performance.

> > > Right, but presumably it still needs to be at least a page. And,
> > > nothing says your device's VF BAR protocol can't be equally simple.
> >
> > Having VFs that are not self-contained would require significant
> > changing of current infrastructure, if we are going to change things
> > then let's fix everything instead of some half measure.
> 
> I don't understand what you mean by self-contained. 

Self-contained means you can pass the VF to a VM with vfio and run a
driver on it. A VF that only has a write-only doorbell page probably
cannot be self contained.

> In practice, there will be some kind of configuration channel too,
> but this doesn't necessarily need a lot of room either 

I don't know of any device that can run without configuration, even in
a VF case.

So this all costs SRAM too.

> > The actual complexity inside the kernel is small and the user
> > experience to manage them through devlink is dramatically better than
> > SRIOV. I think it is a win even if there isn't any HW savings.
> 
> I'm not sure I agree with respect to user experience. Users are
> familiar with SR-IOV.

Sort of, SRIOV is a very bad fit for these sophisticated devices, and
no, users are not familiar with the weird intricate details of SR-IOV
in the context of very sophisticated reconfigurable HW like we are
seeing now.

Look at the other series about MSI-X reconfiguration for some colour
on where SRIOV runs into limits due to its specific design.

> Now you impose a complementary model for accomplishing the same goal
> (without solving all the problems, as per the previous discussion,
> so we'll need to reinvent it again later).  

I'm not sure what you are referring to.

> It's not easier for vendors either. Now we need to get users onto new
> drivers to exploit it, with all the distribution lags that entails
> (where existing drivers would work for SR-IOV). 

Compatability with existing drivers in a VM is a vendor
choice. Drivers can do a lot in a scalable way in hypervisor SW to
present whateve programming interface makes sense to the VM. Intel is
showing this approach in their IDXD SIOV ADI driver.

> Some vendors will support it, some won't, further adding to user
> confusion.

Such is the nature of all things, some vendors supported SRIOV and
other didn't too.

Jason