linux-kernel - Re: [PATCH 0/5] iommu/amd: fixes for suspend/resume

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5d7d591abdbfa8a5cb327a59b94ff7a6c1cfdee6.camel@redhat.com>
Date:   Fri, 10 Dec 2021 10:00:26 +0200
From:   Maxim Levitsky <mlevitsk@...hat.com>
To:     linux-kernel@...r.kernel.org
Cc:     David Woodhouse <dwmw@...zon.co.uk>,
        Suravee Suthikulpanit <suravee.suthikulpanit@....com>,
        Joerg Roedel <joro@...tes.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        "open list:AMD IOMMU (AMD-VI)" <iommu@...ts.linux-foundation.org>,
        Will Deacon <will@...nel.org>
Subject: Re: [PATCH 0/5] iommu/amd: fixes for suspend/resume

On Thu, 2021-12-02 at 01:08 +0200, Maxim Levitsky wrote:
> On Tue, 2021-11-23 at 18:10 +0200, Maxim Levitsky wrote:
> > As I sadly found out, a s3 cycle makes the AMD's iommu stop sending interrupts
> > until the system is rebooted.
> > 
> > I only noticed it now because otherwise the IOMMU works, and these interrupts
> > are only used for errors and for GA log which I tend not to use by
> > making my VMs do mwait/pause/etc in guest (cpu-pm=on).
> > 
> > There are two issues here that prevent interrupts from being generated after
> > s3 cycle:
> > 
> > 1. GA log base address was not restored after resume, and was all zeroed
> > after resume (by BIOS or such).
> > 
> > In theory if BIOS writes some junk to it, that can even cause a memory corruption.
> > Patch 2 fixes that.
> > 
> > 2. INTX (aka x2apic mode) settings were not restored after resume.
> > That mode is used regardless if the host uses/supports x2apic, but rather when
> > the IOMMU supports it, and mine does.
> > Patches 3-4 fix that.
> > 
> > Note that there is still one slight (userspace) bug remaining:
> > During suspend all but the boot CPU are offlined and then after resume
> > are onlined again.
> > 
> > The offlining moves all non-affinity managed interrupts to CPU0, and
> > later when all other CPUs are onlined, there is nothing in the kernel
> > to spread back the interrupts over the cores.
> > 
> > The userspace 'irqbalance' daemon does fix this but it seems to ignore
> > the IOMMU interrupts in INTX mode since they are not attached to any
> > PCI device, and thus they remain on CPU0 after a s3 cycle,
> > which is suboptimal when the system has multiple IOMMUs
> > (mine has 4 of them).
> > 
> > Setting the IRQ affinity manually via /proc/irq/ does work.
> > 
> > This was tested on my 3970X with both INTX and regular MSI mode (later was enabled
> > by patching out INTX detection), by running a guest with AVIC enabled and with
> > a PCI assigned device (network card), and observing interrupts from
> > IOMMU while guest is mostly idle.
> > 
> > This was also tested on my AMD laptop with 4650U (which has the same issue)
> > (I tested only INTX mode)
> > 
> > Patch 1 is a small refactoring to remove an unused struct field.
> > 
> > Best regards,
> >    Maxim Levitsky
> > 
> > Maxim Levitsky (5):
> >   iommu/amd: restore GA log/tail pointer on host resume
> >   iommu/amd: x2apic mode: re-enable after resume
> >   iommu/amd: x2apic mode: setup the INTX registers on mask/unmask
> >   iommu/amd: x2apic mode: mask/unmask interrupts on suspend/resume
> >   iommu/amd: remove useless irq affinity notifier
> > 
> >  drivers/iommu/amd/amd_iommu_types.h |   2 -
> >  drivers/iommu/amd/init.c            | 107 +++++++++++++++-------------
> >  2 files changed, 58 insertions(+), 51 deletions(-)
> > 
> > -- 
> > 2.26.3
> > 
> > 
> 
> Polite ping on these patches.

Another very polite ping on these patches :)

Best regards,
	Maxim Levitsky

> Best regards,
> 	Maxim Levitsky