Date:   Wed, 16 Jun 2021 20:17:19 +0000
From:   Michael Kelley <mikelley@...rosoft.com>
To:     Mark Rutland <Mark.Rutland@....com>
CC:     "will@...nel.org" <will@...nel.org>,
        "catalin.marinas@....com" <catalin.marinas@....com>,
        "lorenzo.pieralisi@....com" <lorenzo.pieralisi@....com>,
        "sudeep.holla@....com" <sudeep.holla@....com>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
        "linux-efi@...r.kernel.org" <linux-efi@...r.kernel.org>,
        "arnd@...db.de" <arnd@...db.de>,
        "wei.liu@...nel.org" <wei.liu@...nel.org>,
        "ardb@...nel.org" <ardb@...nel.org>,
        "daniel.lezcano@...aro.org" <daniel.lezcano@...aro.org>,
        KY Srinivasan <kys@...rosoft.com>
Subject: RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent
 support

From: Michael Kelley <mikelley@...rosoft.com> Sent: Sunday, June 13, 2021 7:42 PM
> 
> From: Mark Rutland <mark.rutland@....com> Sent: Thursday, June 10, 2021 9:45 AM
> >
> > Hi Michael,
> >
> > [trimming the bulk of the thrread]
> >
> > On Tue, Jun 08, 2021 at 03:36:06PM +0000, Michael Kelley wrote:
> > > I've had a couple rounds of discussions with the Hyper-V team.   For
> > > the clocksource we've agreed to table the live migration discussion, and
> > > I'll resubmit the code so that arm_arch_timer.c provides the
> > > standard arch_sys_counter clocksource.  As noted previously, this just
> > > works for a Hyper-V guest.  The live migration discussion may come
> > > back later after a deeper investigation by Hyper-V.
> >
> > Great; thanks for this!
> >
> > > For clockevents, there's not a near term fix.  It's more than just plumbing
> > > an interrupt for Hyper-V to virtualize the ARM64 arch timer in a guest VM.
> > > From their perspective there's also benefit in having a timer abstraction
> > > that's independent of the architecture, and in the Linux guest, the STIMER
> > > code is common across x86/x64 and ARM64.  It follows the standard Linux
> > > clockevents model, as it should. The code is already in use in out-of-tree
> > > builds in the Linux VMs included in Windows 10 on ARM64 as part of the
> > > so-called "Windows Subsystem for Linux".
> > >
> > > So I'm hoping we can get this core support for ARM64 guests on Hyper-V
> > > into upstream using the existing STIMER support.  At some point, Hyper-V
> > > will do the virtualization of the ARM64 arch timer, but we don't want to
> > > have to stay out-of-tree until after that happens.
> >
> > My main concern here is making sure that we can rely on architected
> > properties, and don't have to special-case architected bits for hyperv
> > (or any other hypervisor), since that inevitably causes longer-term
> > pain.
> >
> > While in abstract I'm not as worried about using the timer
> > clock_event_device specifically, that same driver provides the
> > clocksource and the event stream, and I want those to work as usual,
> > without being tied into the hyperv code. IIUC that will require some
> > work, since the driver won't register if the GTDT is missing timer
> > interrupts (or if there is no GTDT).
> >
> > I think it really depends on what that looks like.
> 
> Mark,
> 
> Here are the details:
> 
> The existing initialization and registration code in arm_arch_timer.c
> works in a Hyper-V guest with no changes.  As previously mentioned,
> the GTDT exists and is correctly populated.  Even though it isn't used,
> there's a PPI INTID specified for the virtual timer, just so
> the "arm_sys_timer" clockevent can be initialized and registered.
> The IRQ shows up in the output of "cat /proc/interrupts" with zero counts
> for all CPUs since no interrupts are ever generated.  The EL1 virtual
> timer registers (CNTV_CVAL_EL0, CNTV_TVAL_EL0, and CNTV_CTL_EL0)
> are accessible in the VM.  The "arm_sys_timer" clockevent is left in
> a shutdown state with CNTV_CTL_EL0.ENABLE set to zero when the
> Hyper-V STIMER clockevent is registered with a higher rating.
> 
> Event streams are initialized and the __delay() implementation
> for ARM64 inside the kernel works.  However, on the Ampere
> eMAG hardware I'm using for testing, the WFE instruction returns
> more quickly than it should even though the event stream fields in
> CNTKCTL_EL1 are correct.  I have a query in to the Hyper-V team
> to see if they are trapping WFE and just returning, vs. perhaps the
> eMAG processor takes the easy way out and has WFE just return
> immediately.  I'm not knowledgeable about other uses of timer
> event streams, so let me know if there are other usage scenarios
> I should check.

I confirmed that Hyper-V is not trapping the WFE instruction.  And
on a Marvell TX2 and on an Ampere Altra, the counter event stream
and WFE provide the expected delay.  Evidently WFE on the eMAG
doesn't actually delay.  Bottom line:  event streams work as expected
in a Hyper-V VM.  No changes needed to arm_arch_timer.[ch].

Michael

> 
> Finally, the "arch_sys_counter" clocksource gets initialized and
> setup correctly.  If the Hyper-V clocksource is also initialized,
> you can flip between the two clocksources at runtime as expected.
> If the Hyper-V clocksource is not setup, then Linux in the VM runs
> fine with the "arch_sys_counter" clocksource.
> 
> Michael

Powered by blists - more mailing lists