Message-ID: <19901a08ab931a0200f7c079f36e4b27ed2e1616.camel@intel.com>
Date: Thu, 16 Jan 2025 20:16:22 +0000
From: "Huang, Kai" <kai.huang@...el.com>
To: "seanjc@...gle.com" <seanjc@...gle.com>
CC: "Gao, Chao" <chao.gao@...el.com>, "Edgecombe, Rick P"
	<rick.p.edgecombe@...el.com>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>, "binbin.wu@...ux.intel.com"
	<binbin.wu@...ux.intel.com>, "Li, Xiaoyao" <xiaoyao.li@...el.com>, "Chatre,
 Reinette" <reinette.chatre@...el.com>, "Zhao, Yan Y" <yan.y.zhao@...el.com>,
	"Hunter, Adrian" <adrian.hunter@...el.com>, "kvm@...r.kernel.org"
	<kvm@...r.kernel.org>, "pbonzini@...hat.com" <pbonzini@...hat.com>,
	"tony.lindgren@...ux.intel.com" <tony.lindgren@...ux.intel.com>, "Yamahata,
 Isaku" <isaku.yamahata@...el.com>
Subject: Re: [PATCH 12/16] KVM: TDX: Inhibit APICv for TDX guest

On Thu, 2025-01-16 at 06:50 -0800, Sean Christopherson wrote:
> On Thu, Jan 16, 2025, Kai Huang wrote:
> > On Mon, 2025-01-13 at 10:09 +0800, Binbin Wu wrote:
> > > Lazy check for pending APIC EOI when In-kernel IOAPIC
> > > -----------------------------------------------------
> > > In-kernel IOAPIC does not receive EOI with AMD SVM AVIC since the processor
> > > accelerates write to APIC EOI register and does not trap if the interrupt
> > > is edge-triggered. So there is a workaround by lazy check for pending APIC
> > > EOI at the time when setting new IOAPIC irq, and update IOAPIC EOI if no
> > > pending APIC EOI.
> > > KVM is also not be able to intercept EOI for TDX guests.
> > > - When APICv is enabled
> > >    The code of lazy check for pending APIC EOI doesn't work for TDX because
> > >    KVM can't get the status of real IRR and ISR, and the values are 0s in
> > >    vIRR and vISR in apic->regs[], kvm_apic_pending_eoi() will always return
> > >    false. So the RTC pending EOI will always be cleared when ioapic_set_irq()
> > >    is called for RTC. Then userspace may miss the coalesced RTC interrupts.
> > > - When When APICv is disabled
> > >    ioapic_lazy_update_eoi() will not be called，then pending EOI status for
> > >    RTC will not be cleared after setting and this will mislead userspace to
> > >    see coalesced RTC interrupts.
> > > Options:
> > > - Force irqchip split for TDX guests to eliminate the use of in-kernel IOAPIC.
> > > - Leave it as it is, but the use of RTC may not be accurate.
> > 
> > Looking at the code, it seems KVM only traps EOI for level-triggered interrupt
> > for in-kernel IOAPIC chip, but IIUC IOAPIC in userspace also needs to be told
> > upon EOI for level-triggered interrupt.  I don't know how does KVM works with
> > userspace IOAPIC w/o trapping EOI for level-triggered interrupt, but "force
> > irqchip split for TDX guest" seems not right.
> 
> Forcing a "split" IRQ chip is correct, in the sense that TDX doesn't support an
> I/O APIC and the "split" model is the way to concoct such a setup.  With a "full"
> IRQ chip, KVM is responsible for emulating the I/O APIC, which is more or less
> nonsensical on TDX because it's fully virtual world, i.e. there's no reason to
> emulate legacy devices that only know how to talk to the I/O APIC (or PIC, etc.).
> Disallowing an in-kernel I/O APIC is ideal from KVM's perspective, because
> level-triggered interrupts and thus the I/O APIC as a whole can't be faithfully
> emulated (see below).

Disabling in-kernel IOAPIC/PIC for TDX guests is fine to me, but I think that,
"conceptually", having IOAPIC/PIC in userspace doesn't mean disabling IOAPIC,
because theoretically usrespace IOAPIC still needs to be told about the EOI for
emulation.  I just haven't figured out how does userpsace IOAPIC work with KVM
in case of "split IRQCHIP" w/o trapping EOI for level-triggered interrupt. :-)

If the point is to disable in-kernel IOAPIC/PIC for TDX guests, then I think
both KVM_IRQCHIP_NONE and KVM_IRQCHIP_SPLIT should be allowed for TDX, but not
just KVM_IRQCHIP_SPLIT?

> 
> > I think the problem is level-triggered interrupt,
> 
> Yes, because the TDX Module doesn't allow the hypervisor to modify the EOI-bitmap,
> i.e. all EOIs are accelerated and never trigger exits.
> 
> > so I think another option is to reject level-triggered interrupt for TDX guest.
> 
> This is a "don't do that, it will hurt" situation.  With a sane VMM, the level-ness
> of GSIs is controlled by the guest.  For GSIs that are routed through the I/O APIC,
> the level-ness is determined by the corresponding Redirection Table entry.  For
> "GSIs" that are actually MSIs (KVM piggybacks legacy GSI routing to let userspace
> wire up MSIs), and for direct MSIs injection (KVM_SIGNAL_MSI), the level-ness is
> dictated by the MSI itself, which again is guest controlled.
> 
> If the guest induces generation of a level-triggered interrupt, the VMM is left
> with the choice of dropping the interrupt, sending it as-is, or converting it to
> an edge-triggered interrupt.  Ditto for KVM.  All of those options will make the
> guest unhappy.
> 
> So while it _might_ make debugging broken guests either, I don't think it's worth
> the complexity to try and prevent the VMM/guest from sending level-triggered
> GSI-routed interrupts.  
> 

KVM can at least have some chance to print some error message?

> It'd be a bit of a whack-a-mole and there's no architectural
> behavior KVM can provide that's better than sending the interrupt and hoping for
> the best.

Powered by blists - more mailing lists