Date:   Thu, 3 Aug 2017 01:31:53 -0700
From:   "Raj, Ashok" <ashok.raj@...el.com>
To:     Casey Leedom <leedom@...lsio.com>
Cc:     Ding Tianhong <dingtianhong@...wei.com>,
        Alexander Duyck <alexander.duyck@...il.com>,
        Alex Williamson <alex.williamson@...hat.com>,
        Sinan Kaya <okaya@...eaurora.org>,
        "bhelgaas@...gle.com" <bhelgaas@...gle.com>,
        "helgaas@...nel.org" <helgaas@...nel.org>,
        Michael Werner <werner@...lsio.com>,
        Ganesh GR <ganeshgr@...lsio.com>,
        "asit.k.mallick@...el.com" <asit.k.mallick@...el.com>,
        "patrick.j.cramer@...el.com" <patrick.j.cramer@...el.com>,
        "Suravee.Suthikulpanit@....com" <Suravee.Suthikulpanit@....com>,
        "Bob.Shaw@....com" <Bob.Shaw@....com>,
        "l.stach@...gutronix.de" <l.stach@...gutronix.de>,
        "amira@...lanox.com" <amira@...lanox.com>,
        "gabriele.paoloni@...wei.com" <gabriele.paoloni@...wei.com>,
        "David.Laight@...lab.com" <David.Laight@...lab.com>,
        "jeffrey.t.kirsher@...el.com" <jeffrey.t.kirsher@...el.com>,
        "catalin.marinas@....com" <catalin.marinas@....com>,
        "will.deacon@....com" <will.deacon@....com>,
        "mark.rutland@....com" <mark.rutland@....com>,
        "robin.murphy@....com" <robin.murphy@....com>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linuxarm@...wei.com" <linuxarm@...wei.com>, ashok.raj@...el.com
Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported

Hi Casey

On Wed, Aug 02, 2017 at 05:53:52PM +0000, Casey Leedom wrote:
>   Okay, here you go.  As you can tell, it's almost a trivial copy of the
> cxgb4 patch.
>  
>   By the way, I realized that we have yet another hole which is likely not
> to be fixable.  If we're dealing with a problematic Root Complex, and we
> instantiate Virtual Functions and attach them to a Virtual Machine along
> with an NVMe device which can deal with Relaxed Ordering TLPs, the VF driver
> in the VM will be able to tell that it shouldn't attempt to send RO TLPs to
> the RC because it will see the state of its own PCIe Capability Device
> Control[Relaxed Ordering Enable] (a copy of the setting in the VF's
> corresponding PF), but it won't be able to change that and send non-RO TLPs
> to the RC, and RO TLPs to the NVMe device.  Oh well.

I don't understand this completely.. So your driver would know not to send RO
TLP's to root complex. But you want to send RO to the NVMe device? This is the
peer-2-peer case correct?

The issue in the current patchset is that we device to turn off the device 
capability for all devices in the hierarchy so one would expect that 
the NVMe also would have RO turned off i suppose. 

The other approach is to not turn off the device capabilty, but let the
driver do the right thing. i.e for transactions towards system memory vs. 
peer-2-peer? But since we wanted to take a big hammer approach because
some platforms there can be data-corruption and we can't let trust guest
drivers to do the right thing. This isn't something we can fix in this 
current version.

One possible approach is to provide a strict flag, where we use this heavy 
hammer approach only on platforms that have a serious implication, and the 
other is we let the driver do the right thing depending on the platform.

Worst case if the driver doesn't do the right thing, you would see perf issues
but nothing bad would happen. It would allow you to select when to turn on 
RO and when to turn it off.

Cheers,
Ashok

Powered by blists - more mailing lists