Date:   Wed, 16 Nov 2016 07:58:20 -0700
From:   Myron Stowe <myron.stowe@...il.com>
To:     xlpang@...hat.com
Cc:     iommu@...ts.linux-foundation.org, Joerg Roedel <joro@...tes.org>,
        Don Brace <don.brace@...rosemi.com>,
        Dave Young <dyoung@...hat.com>, kexec@...ts.infradead.org,
        LKML <linux-kernel@...r.kernel.org>,
        Myron Stowe <myron.stowe@...hat.com>,
        David Woodhouse <dwmw2@...radead.org>
Subject: Re: [PATCH] iommu/vt-d: Flush old iotlb for kdump when the device
 gets context mapped

On Wed, Nov 16, 2016 at 2:13 AM, Xunlei Pang <xpang@...hat.com> wrote:
> Ccing David
> On 2016/11/16 at 17:02, Xunlei Pang wrote:
>> We met the DMAR fault both on hpsa P420i and P421 SmartArray controllers
>> under kdump, it can be steadily reproduced on several different machines,
>> the dmesg log is like:
>> HP HPSA Driver (v 3.4.16-0)
>> hpsa 0000:02:00.0: using doorbell to reset controller
>> hpsa 0000:02:00.0: board ready after hard reset.
>> hpsa 0000:02:00.0: Waiting for controller to respond to no-op
>> DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]
>> DMAR: Setting identity map for device 0000:02:00.0 [0xf4000 - 0xf4fff]
>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6e000 - 0xbdf6efff]
>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf6f000 - 0xbdf7efff]
>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf7f000 - 0xbdf82fff]
>> DMAR: Setting identity map for device 0000:02:00.0 [0xbdf83000 - 0xbdf84fff]
>> DMAR: DRHD: handling fault status reg 2
>> DMAR: [DMA Read] Request device [02:00.0] fault addr fffff000 [fault reason 06] PTE Read access is not set
>> hpsa 0000:02:00.0: controller message 03:00 timed out
>> hpsa 0000:02:00.0: no-op failed; re-trying
>>
>> After some debugging, we found that the corresponding pte entry value
>> is correct, and the value of the iommu caching mode is 0, the fault is
>> probably due to the old iotlb cache of the in-flight DMA.
>>
>> Thus need to flush the old iotlb after context mapping is setup for the
>> device, where the device is supposed to finish reset at its driver probe
>> stage and no in-flight DMA exists hereafter.
>>
>> With this patch, all our problematic machines can survive the kdump tests.
>>
>> CC: Myron Stowe <myron.stowe@...hat.com>
>> CC: Don Brace <don.brace@...rosemi.com>
>> CC: Baoquan He <bhe@...hat.com>
>> CC: Dave Young <dyoung@...hat.com>
>> Tested-by: Joseph Szczypek <jszczype@...hat.com>
>> Signed-off-by: Xunlei Pang <xlpang@...hat.com>
>> ---
>>  drivers/iommu/intel-iommu.c | 11 +++++++++--
>>  1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
>> index 3965e73..eb79288 100644
>> --- a/drivers/iommu/intel-iommu.c
>> +++ b/drivers/iommu/intel-iommu.c
>> @@ -2067,9 +2067,16 @@ static int domain_context_mapping_one(struct dmar_domain *domain,
>>        * It's a non-present to present mapping. If hardware doesn't cache
>>        * non-present entry we only need to flush the write-buffer. If the
>>        * _does_ cache non-present entries, then it does so in the special

If this does get accepted then we should fix the above grammar also -
  "If the _does_ cache ..." -> "If the hardware _does_ cache ..."

>> -      * domain #0, which we have to flush:
>> +      * domain #0, which we have to flush.
>> +      *
>> +      * For kdump cases, present entries may be cached due to the in-flight
>> +      * DMA and copied old pgtable, but there is no unmapping behaviour for
>> +      * them, so we need an explicit iotlb flush for the newly-mapped device.
>> +      * For kdump, at this point, the device is supposed to finish reset at
>> +      * the driver probe stage, no in-flight DMA will exist, thus we do not
>> +      * need to worry about that anymore hereafter.
>>        */
>> -     if (cap_caching_mode(iommu->cap)) {
>> +     if (is_kdump_kernel() || cap_caching_mode(iommu->cap)) {
>>               iommu->flush.flush_context(iommu, 0,
>>                                          (((u16)bus) << 8) | devfn,
>>                                          DMA_CCMD_MASK_NOBIT,
>
> _______________________________________________
> iommu mailing list
> iommu@...ts.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu

Powered by blists - more mailing lists