linux-kernel - Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210930220446.GF964074@nvidia.com>
Date:   Thu, 30 Sep 2021 19:04:46 -0300
From:   Jason Gunthorpe <jgg@...dia.com>
To:     Jean-Philippe Brucker <jean-philippe@...aro.org>
Cc:     "Tian, Kevin" <kevin.tian@...el.com>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "jasowang@...hat.com" <jasowang@...hat.com>,
        "kwankhede@...dia.com" <kwankhede@...dia.com>,
        "hch@....de" <hch@....de>, "Jiang, Dave" <dave.jiang@...el.com>,
        "Raj, Ashok" <ashok.raj@...el.com>,
        "corbet@....net" <corbet@....net>,
        "parav@...lanox.com" <parav@...lanox.com>,
        Alex Williamson <alex.williamson@...hat.com>,
        "lkml@...ux.net" <lkml@...ux.net>,
        "david@...son.dropbear.id.au" <david@...son.dropbear.id.au>,
        "dwmw2@...radead.org" <dwmw2@...radead.org>,
        "Tian, Jun J" <jun.j.tian@...el.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "lushenming@...wei.com" <lushenming@...wei.com>,
        "pbonzini@...hat.com" <pbonzini@...hat.com>,
        "robin.murphy@....com" <robin.murphy@....com>
Subject: Re: [RFC 10/20] iommu/iommufd: Add IOMMU_DEVICE_GET_INFO

On Thu, Sep 30, 2021 at 11:33:13AM +0100, Jean-Philippe Brucker wrote:
> On Thu, Sep 30, 2021 at 08:30:42AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe
> > > Sent: Wednesday, September 29, 2021 8:37 PM
> > > 
> > > On Wed, Sep 29, 2021 at 08:48:28AM +0000, Tian, Kevin wrote:
> > > 
> > > > ARM:
> > > >     - set to snoop format if IOMMU_CACHE
> > > >     - set to nonsnoop format if !IOMMU_CACHE
> > > > (in both cases TLP snoop bit is ignored?)
> > > 
> > > Where do you see this? I couldn't even find this functionality in the
> > > ARM HW manual??
> > 
> > Honestly speaking I'm getting confused by the complex attribute
> > transformation control (default, replace, combine, input, output, etc.)
> > in SMMU manual. Above was my impression after last check, but now
> > I cannot find necessary info to build the same picture (except below 
> > code). :/
> > 
> > > 
> > > What I saw is ARM linking the IOMMU_CACHE to a IO PTE bit that causes
> > > the cache coherence to be disabled, which is not ignoring no snoop.
> > 
> > My impression was that snoop is one way of implementing cache
> > coherency and now since the PTE can explicitly specify cache coherency 
> > like below:
> > 
> >                 else if (prot & IOMMU_CACHE)
> >                         pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
> >                 else
> >                         pte |= ARM_LPAE_PTE_MEMATTR_NC;
> > 
> > This setting in concept overrides the snoop attribute from the device thus
> > make it sort of ignored?
> 
> To make sure we're talking about the same thing: "the snoop attribute from
> the device" is the "No snoop" attribute in the PCI TLP, right?
> 
> The PTE flags define whether the memory access is cache-coherent or not.
> * WB is cacheable (short for write-back cacheable. Doesn't matter here
>   what OI or RWA mean.)
> * NC is non-cacheable.
> 
>          | Normal PCI access | No_snoop PCI access
>   PTE WB | Cacheable         | Non-cacheable
>   PTE NC | Non-cacheable     | Non-cacheable
> 
> Cacheable memory access participate in cache coherency. Non-cacheable
> accesses go directly to memory, do not cause cache allocation.

This table is what I was thinking after reading through the ARM docs.

> On Arm cache coherency is configured through PTE attributes. I don't think
> PCI No_snoop should be used because it's not necessarily supported
> throughout the system and, as far as I understand, software can't discover
> whether it is.

The usage of no-snoop is a behavior of a device. A generic PCI driver
should be able to program the device to generate no-snoop TLPs and
ideally rely on an arch specific API in the OS to trigger the required
cache maintenance.

It doesn't make much sense for a portable driver to rely on a
non-portable IO PTE flag to control coherency, since that is not a
standards based approach.

That said, Linux doesn't have a generic DMA API to support
no-snoop. The few GPUs drivers that use this stuff just hardwired
wbsync on Intel..

What I don't really understand is why ARM, with an IOMMU that supports
PTE WB, has devices where dev_is_dma_coherent() == false ? 

Is it the case that DMA from those devices ignores the IO PTE's
cachable mode?

Jason