linux-kernel - Re: [PATCH v5 0/2] Reset PCIe devices to address DMA problem on kdump with iommu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <5090E865.1060503@jp.fujitsu.com>
Date:	Wed, 31 Oct 2012 17:59:17 +0900
From:	Takao Indoh <indou.takao@...fujitsu.com>
To:	linux-pci@...r.kernel.org, x86@...nel.org,
	linux-kernel@...r.kernel.org, martin.wilck@...fujitsu.com,
	andi@...stfloor.org, kexec@...ts.infradead.org, hbabu@...ibm.com,
	mingo@...hat.com, ddutile@...hat.com, vgoyal@...hat.com,
	ishii.hironobu@...fujitsu.com, hpa@...or.com, bhelgaas@...gle.com,
	tglx@...utronix.de, khalid@...ehiking.org
Subject: Re: [PATCH v5 0/2] Reset PCIe devices to address DMA problem on kdump
 with iommu

(2012/10/17 15:23), Takao Indoh wrote:
> These patches reset PCIe devices at boot time to address DMA problem on
> kdump with iommu. When "reset_devices" is specified, a hot reset is
> triggered on each PCIe root port and downstream port to reset its
> downstream endpoint.
>
> Background:
> A kdump problem about DMA has been discussed for a long time. That is,
> when a kernel is switched to the kdump kernel DMA derived from first
> kernel affects second kernel. Recently this problem surfaces when iommu
> is used for PCI passthrough on KVM guest. In the case of the machine I
> use, when intel_iommu=on is specified, DMAR error is detected in kdump
> kernel and PCI SERR is also detected. Finally kdump fails because some
> devices does not work correctly.
>
> The root cause is that ongoing DMA from first kernel causes DMAR fault
> because page table of DMAR is initialized while kdump kernel is booting
> up. Therefore to address this problem DMA needs to be stopped before
> DMAR is initialized at kdump kernel boot time. By these patches, PCIe
> devices are reset by hot reset and its DMA is stopped when reset_devices
> is specified. One problem of this solution is that the monitor blacks
> out when VGA controller is reset. So this patch does not reset the port
> whose child endpoint is VGA device.
>
> What I tried:
> - Clearing bus master bit and INTx disable bit at boot time
>      This did not solve this problem. I still got DMAR error on devices.
> - Resetting devices in fixup_final(v1 patch)
>      DMAR error disappeared, but sometimes PCI SERR was detected. This
>      is well explained here.
>      https://lkml.org/lkml/2012/9/9/245
>      This PCI SERR seems to be related to interrupt remapping.
> - Clearing bus master in setup_arch() and resetting devices in
>    fixup_final
>      Neither DMAR error nor PCI SERR occurred. But on certain machine
>      kdump kernel hung up when resetting devices. It seems to be a
>      problem specific to the platform.
> - Resetting devices in setup_arch() (v2 and later patch)
>      This solution solves all problems I found so far.
>
> v5:
> Do bus reset after all devices are scanned and its config registers are
> saved. This fixes a bug that config register is accessed without delay
> after reset.
>
> v4:
> Reduce waiting time after resetting devices. A previous patch does reset
> like this:
>    for (each device) {
>      save config registers
>      reset
>      wait for 500 ms
>      restore config registers
>    }
>
> If there are N devices to be reset, it takes N*500 ms. On the other
> hand, the v4 patch does:
>    for (each device) {
>      save config registers
>      reset
>    }
>    wait 500 ms
>    for (each device) {
>      restore config registers
>    }
> Though it needs more memory space to save config registers, the waiting
> time is always 500ms.
> https://lkml.org/lkml/2012/10/15/49
>
> v3:
> Move alloc_bootmem and free_bootmem to early_reset_pcie_devices so that
> they are called only once.
> https://lkml.org/lkml/2012/10/10/57
>
> v2:
> Reset devices in setup_arch() because reset need to be done before
> interrupt remapping is initialized.
> https://lkml.org/lkml/2012/10/2/54
>
> v1:
> Add fixup_final quirk to reset PCIe devices
> https://lkml.org/lkml/2012/8/3/160

Any other comments or ack/nack? If this is accepted I'll try multiple
domain support as next step.

Thanks,
Takao Indoh

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/