linux-kernel - Re: Requirement to get BAR pci_bus

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CABe79T4yopn4FJqSv+igvekNBZOgmySxyoxFfFzQP=grd9p=vg@mail.gmail.com>
Date:   Fri, 15 Jun 2018 11:41:47 +0530
From:   Srinath Mannam <srinath.mannam@...adcom.com>
To:     "Walker, Benjamin" <benjamin.walker@...el.com>
Cc:     "alex.williamson@...hat.com" <alex.williamson@...hat.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "vikram.prakash@...adcom.com" <vikram.prakash@...adcom.com>,
        "okaya@...eaurora.org" <okaya@...eaurora.org>,
        "hch@....de" <hch@....de>,
        "abhishek.shah@...adcom.com" <abhishek.shah@...adcom.com>,
        "linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>,
        "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "linux-pci-owner@...r.kernel.org" <linux-pci-owner@...r.kernel.org>,
        "bhelgaas@...gle.com" <bhelgaas@...gle.com>
Subject: Re: Requirement to get BAR pci_bus_address in user space

Thank you very much Alex and Ben for all the details.

Please find my comments in lined..


On Fri, Jun 15, 2018 at 1:34 AM, Walker, Benjamin
<benjamin.walker@...el.com> wrote:
> On Thu, 2018-06-14 at 08:50 -0600, Alex Williamson wrote:
>> On Thu, 14 Jun 2018 16:18:15 +0530
>> Srinath Mannam <srinath.mannam@...adcom.com> wrote:
>>
>> > Hi Sinan Kaya,
>> >
>> > Here are the details,
>> >
>> > The issue is, For CMB cards SQs are allocated inside device BAR memory
>> > which is different from normal cards.
>> > In Normal cards SQ memory allocated at host side.
>> > In both the cases physical address of CQ memory is programmed in NVMe
>> > controller register.
>> > This method works for normal cards because CQ memory is at host side.
>> > But in CMB cards pci bus address equivalent to CQ memory needs to program.
>> >
>> > More details are in the patch: nvme-pci: Use PCI bus address for
>> > data/queues in CMB.
>> >
>> > With the above patch issue is fixed in the NVMe kernel driver, But
>> > similar fix is required in SPDK library also.
>> > So, We need a mechanism to get pci_bus_address in user space libraries
>> > to address this issue.
>>
>> I don't understand the CQ vs CMB, but I think I gather that there's some
Sorry, It was mistake, I meant SQ not CQ.
>> sort of buffer that's allocated from within the devices MMIO BAR and
>> some programming of the device needs to reference that buffer.
>> Wouldn't you therefore use the vfio type1 IOMMU MAP_DMA ioctl to map
>> the BAR into the IOVA address space and you can then use the IOVA +
>> offset into the BAR for the device to reference the buffer?  It seems
>> this is the same way we'd setup a peer-to-peer mapping, but we're using
>> it for the device to reference itself effectively.  Thanks,
>
> SPDK already does this today. It's capable of placing submission and completion
> queues in the device's controller memory buffer (CMB) and will do all of the
> right things for both uio (look up physical address) and vfio (send
> IOMMU_MAP_DMA ioctl to set a iova and use that). It additionally already
> supports using the CMB as a target for peer-to-peer data transfers.
>
> This was just confirmed to work on the most recent SPDK release. If you are not
> on the most recent release, I recommend trying that first. If there is still
> some issue, please do take it to the SPDK mailing list.
>
We are using SPDK version 18.04.
Is fix available in 18.04? Please point me to the patch with which fix
was added.

> Thanks,
> Ben