Message-ID: <MWHPR21MB15930A4EE785984292B1D72BD72D9@MWHPR21MB1593.namprd21.prod.outlook.com>
Date:   Mon, 17 May 2021 17:27:49 +0000
From:   Michael Kelley <mikelley@...rosoft.com>
To:     Mark Rutland <Mark.Rutland@....com>
CC:     "will@...nel.org" <will@...nel.org>,
        "catalin.marinas@....com" <catalin.marinas@....com>,
        "lorenzo.pieralisi@....com" <lorenzo.pieralisi@....com>,
        "sudeep.holla@....com" <sudeep.holla@....com>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
        "linux-efi@...r.kernel.org" <linux-efi@...r.kernel.org>,
        "arnd@...db.de" <arnd@...db.de>,
        "wei.liu@...nel.org" <wei.liu@...nel.org>,
        "ardb@...nel.org" <ardb@...nel.org>,
        "daniel.lezcano@...aro.org" <daniel.lezcano@...aro.org>,
        KY Srinivasan <kys@...rosoft.com>
Subject: RE: [PATCH v10 3/7] arm64: hyperv: Add Hyper-V clocksource/clockevent
 support

From: Mark Rutland <mark.rutland@....com> Sent: Monday, May 17, 2021 6:08 AM
> 
> On Fri, May 14, 2021 at 03:35:15PM +0000, Michael Kelley wrote:
> > From: Mark Rutland <mark.rutland@....com> Sent: Friday, May 14, 2021 5:37 AM
> > > On Wed, May 12, 2021 at 10:37:43AM -0700, Michael Kelley wrote:
> > > > Add architecture specific definitions and functions needed
> > > > by the architecture independent Hyper-V clocksource driver.
> > > > Update the Hyper-V clocksource driver to be initialized
> > > > on ARM64.
> > >
> > > Previously we've said that for a clocksource we must use the architected
> > > counter, since that's necessary for things like the VDSO to work
> > > correctly and efficiently.
> > >
> > > Given that, I'm a bit confused that we're registering a per-cpu
> > > clocksource that is in part based on the architected counter. Likewise,
> > > I don't entirely follow why it's necessary to PV the clock_event_device.
> > >
> > > Are the architected counter and timer reliable without this PV
> > > infrastructure? Why do we need to PV either of those?
> > >
> > > Thanks,
> > > Mark.
> >
> > For the clocksource, we have a requirement to live migrate VMs
> > between Hyper-V hosts running on hardware that may have different
> > arch counter frequencies (it's not conformant to the ARM v8.6 1 GHz
> > requirement).  The Hyper-V virtualization does scaling to handle the
> > frequency difference.  And yes, there's a tradeoff with vDSO not
> > working, though we have an out-of-tree vDSO implementation that
> > we can use when necessary.
> 
> Just to be clear, the vDSO is *one example* of something that won't
> function correctly. More generally, because this undermines core
> architectural guarantees, it requires more invasive changes (e.g. we'd
> have to weaken the sanity checks, and not use the counter in things like
> kexec paths), impacts any architectural features tied to the generic
> timer/counter (e.g. the event stream, SPE and tracing, future features),
> and means that other SW (e.g. bootloaders and other EFI applications)
> are unlikley to function correctly in this environment.
> 
> I am very much not keen on trying to PV this.
> 
> What does the guest see when it reads CNTFRQ_EL0? Does this match the
> real HW value (and can this change over time)? Or is this entirely
> synthetic?
> 
> What do the ACPI tables look like in the guest? Is there a GTDT table at
> all?
> 
> How does the counter event stream behave?
> 
> Are there other architectural features which Hyper-V does not implement
> for a guest?
> 
> Is there anything else that may change across a migration? e.g. MIDR?
> MPIDR? Any of the ID registers?

The ARMv8 architectural system counter and associated registers are visible
and functional in a VM on Hyper-V.   The "arch_sys_counter" clocksource is
instantiated by the arm_arch_timer.c driver based on the GTDT in the guest,
and a Linux guest on Hyper-V runs fine with this clocksource.  Low level code
like bootloaders and EFI applications work normally.

The Hyper-V virtualization provides another Linux clocksource that is an
overlay on the arch counter and that provides time consistency across a live
migration. Live migration of ARM64 VMs on Hyper-V is not functional today,
but the Hyper-V team believes they can make it functional.  I have not
explored with them the live migration implications of things beyond time
consistency, like event streams, CNTFRQ_EL0, MIDR/MPIDR, etc.

Would a summary of your point be that live migration across hardware
with different arch counter frequencies is likely to not be possible with
100% fidelity because of these other dependencies on the arch counter
frequency?  (hence the fixed 1 GHz frequency in ARM v8.6)

> 
> > For clockevents, the only timer interrupt that Hyper-V provides
> > in a guest VM is its virtualized "STIMER" interrupt.  There's no
> > virtualization of the ARM arch timer in the guest.
> 
> I think that is rather unfortunate, given it's a core architectural
> feature. Is it just the interrupt that's missing? i.e. does all the
> PE-local functionality behave as the architecture requires?

Right off the bat, I don't know about timer-related PE-local
functionality as it's not exercised in a Linux VM on Hyper-V that is
using STIMER-based clockevents.  I'll explore with the Hyper-V
team.  My impression is that enabling the ARM arch timer in a
guest VM is more work for Hyper-V than just wiring up an
interrupt.

Michael

> 
> Thanks,
> Mark.
> 
> >
> > > >
> > > > Signed-off-by: Michael Kelley <mikelley@...rosoft.com>
> > > > Reviewed-by: Sunil Muthuswamy <sunilmut@...rosoft.com>
> > > > ---
> > > >  arch/arm64/include/asm/mshyperv.h  | 12 ++++++++++++
> > > >  drivers/clocksource/hyperv_timer.c | 14 ++++++++++++++
> > > >  2 files changed, 26 insertions(+)
> > > >
> > > > diff --git a/arch/arm64/include/asm/mshyperv.h
> b/arch/arm64/include/asm/mshyperv.h
> > > > index c448704..b17299c 100644
> > > > --- a/arch/arm64/include/asm/mshyperv.h
> > > > +++ b/arch/arm64/include/asm/mshyperv.h
> > > > @@ -21,6 +21,7 @@
> > > >  #include <linux/types.h>
> > > >  #include <linux/arm-smccc.h>
> > > >  #include <asm/hyperv-tlfs.h>
> > > > +#include <clocksource/arm_arch_timer.h>
> > > >
> > > >  /*
> > > >   * Declare calls to get and set Hyper-V VP register values on ARM64, which
> > > > @@ -41,6 +42,17 @@ static inline u64 hv_get_register(unsigned int reg)
> > > >  	return hv_get_vpreg(reg);
> > > >  }
> > > >
> > > > +/* Define the interrupt ID used by STIMER0 Direct Mode interrupts. This
> > > > + * value can't come from ACPI tables because it is needed before the
> > > > + * Linux ACPI subsystem is initialized.
> > > > + */
> > > > +#define HYPERV_STIMER0_VECTOR	31
> > > > +
> > > > +static inline u64 hv_get_raw_timer(void)
> > > > +{
> > > > +	return arch_timer_read_counter();
> > > > +}
> > > > +
> > > >  /* SMCCC hypercall parameters */
> > > >  #define HV_SMCCC_FUNC_NUMBER	1
> > > >  #define HV_FUNC_ID	ARM_SMCCC_CALL_VAL(			\
> > > > diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> > > > index 977fd05..270ad9c 100644
> > > > --- a/drivers/clocksource/hyperv_timer.c
> > > > +++ b/drivers/clocksource/hyperv_timer.c
> > > > @@ -569,3 +569,17 @@ void __init hv_init_clocksource(void)
> > > >  	hv_setup_sched_clock(read_hv_sched_clock_msr);
> > > >  }
> > > >  EXPORT_SYMBOL_GPL(hv_init_clocksource);
> > > > +
> > > > +/* Initialize everything on ARM64 */
> > > > +static int __init hyperv_timer_init(struct acpi_table_header *table)
> > > > +{
> > > > +	if (!hv_is_hyperv_initialized())
> > > > +		return -EINVAL;
> > > > +
> > > > +	hv_init_clocksource();
> > > > +	if (hv_stimer_alloc(true))
> > > > +		return -EINVAL;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +TIMER_ACPI_DECLARE(hyperv, ACPI_SIG_GTDT, hyperv_timer_init);
> > > > --
> > > > 1.8.3.1
> > > >

Powered by blists - more mailing lists