linux-kernel - Re: Enabling peer to peer device transactions for PCIe devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <bedaa7a2-e42d-da83-5c2b-9d639b0397c5@deltatee.com>
Date:   Tue, 6 Dec 2016 14:47:04 -0700
From:   Logan Gunthorpe <logang@...tatee.com>
To:     Jason Gunthorpe <jgunthorpe@...idianresearch.com>
Cc:     Stephen Bates <sbates@...thlin.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Haggai Eran <haggaie@...lanox.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "linux-nvdimm@...1.01.org" <linux-nvdimm@...1.01.org>,
        "christian.koenig@....com" <christian.koenig@....com>,
        "Suravee.Suthikulpanit@....com" <suravee.suthikulpanit@....com>,
        "John.Bridgman@....com" <john.bridgman@....com>,
        "Alexander.Deucher@....com" <alexander.deucher@....com>,
        "Linux-media@...r.kernel.org" <linux-media@...r.kernel.org>,
        "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
        Max Gurtovoy <maxg@...lanox.com>,
        "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "serguei.sagalovitch@....com" <serguei.sagalovitch@....com>,
        "Paul.Blinzer@....com" <paul.blinzer@....com>,
        "Felix.Kuehling@....com" <felix.kuehling@....com>,
        "ben.sander@....com" <ben.sander@....com>
Subject: Re: Enabling peer to peer device transactions for PCIe devices

Hey,

> Okay, so clearly this needs a kernel side NVMe specific allocator
> and locking so users don't step on each other..

Yup, ideally. That's why device dax isn't ideal for this application: it
doesn't provide any way to prevent users from stepping on each other.

> Or as Christoph says some kind of general mechanism to get these
> bounce buffers..

Yeah, I imagine a general allocate from BAR/region system would be very
useful.

> Ah, I see.
> 
> As a first draft I'd stick with some kind of API built into the
> /dev/nvmeX that backs the filesystem. The user app would fstat the
> target file, open /dev/block/MAJOR(st_dev):MINOR(st_dev), do some
> ioctl to get a CMB mmap, and then proceed from there..
> 
> When that is all working kernel-side, it would make sense to look at a
> more general mechanism that could be used unprivileged??

That makes a lot of sense to me. I suggested mmapping the char device
because it's really easy, but I can see that an ioctl on the block
device does seem more general and device agnostic.

> This is similar to the GPU issues too.. On NVMe you don't need to pin
> the pages, you just need to lock that VMA so it doesn't get freed from
> the NVMe CMB allocator while the IO is running...
> Probably in the long run the get_user_pages is going to have to be
> pushed down into drivers.. Future MMU coherent IO hardware also does
> not need the pinning or other overheads.

Yup. Yup.

Logan