Message-ID: <CAL715WKOPBG+G-hTDNw7S12uRMOP=3JgcYcys2xbKWrQVrXyDQ@mail.gmail.com>
Date: Thu, 25 Apr 2024 21:46:47 -0700
From: Mingwei Zhang <mizhang@...gle.com>
To: "Mi, Dapeng" <dapeng1.mi@...ux.intel.com>
Cc: Sean Christopherson <seanjc@...gle.com>, Kan Liang <kan.liang@...ux.intel.com>, 
	maobibo <maobibo@...ngson.cn>, Xiong Zhang <xiong.y.zhang@...ux.intel.com>, pbonzini@...hat.com, 
	peterz@...radead.org, kan.liang@...el.com, zhenyuw@...ux.intel.com, 
	jmattson@...gle.com, kvm@...r.kernel.org, linux-perf-users@...r.kernel.org, 
	linux-kernel@...r.kernel.org, zhiyuan.lv@...el.com, eranian@...gle.com, 
	irogers@...gle.com, samantha.alt@...el.com, like.xu.linux@...il.com, 
	chao.gao@...el.com
Subject: Re: [RFC PATCH 23/41] KVM: x86/pmu: Implement the save/restore of PMU
 state for Intel CPU

On Thu, Apr 25, 2024 at 9:03 PM Mi, Dapeng <dapeng1.mi@...ux.intel.com> wrote:
>
>
> On 4/26/2024 11:12 AM, Mingwei Zhang wrote:
> > On Thu, Apr 25, 2024 at 6:46 PM Mi, Dapeng <dapeng1.mi@...ux.intel.com> wrote:
> >>
> >> On 4/26/2024 5:46 AM, Sean Christopherson wrote:
> >>> On Thu, Apr 25, 2024, Kan Liang wrote:
> >>>> On 2024-04-25 4:16 p.m., Mingwei Zhang wrote:
> >>>>> On Thu, Apr 25, 2024 at 9:13 AM Liang, Kan <kan.liang@...uxintel.com> wrote:
> >>>>>> It should not happen. For the current implementation, perf rejects all
> >>>>>> the !exclude_guest system-wide event creation if a guest with the vPMU
> >>>>>> is running.
> >>>>>> However, it's possible to create an exclude_guest system-wide event at
> >>>>>> any time. KVM cannot use the information from the VM-entry to decide if
> >>>>>> there will be active perf events in the VM-exit.
> >>>>> Hmm, why not? If there is any exclude_guest system-wide event,
> >>>>> perf_guest_enter() can return something to tell KVM "hey, some active
> >>>>> host events are swapped out. they are originally in counter #2 and
> >>>>> #3". If so, at the time when perf_guest_enter() returns, KVM will ack
> >>>>> that and keep it in its pmu data structure.
> >>>> I think it's possible that someone creates !exclude_guest event after
> >>> I assume you mean an exclude_guest=1 event?  Because perf should be in a state
> >>> where it rejects exclude_guest=0 events.
> >> Suppose should be exclude_guest=1 event, the perf event without
> >> exclude_guest attribute would be blocked to create in the v2 patches
> >> which we are working on.
> >>
> >>
> >>>> the perf_guest_enter(). The stale information is saved in the KVM. Perf
> >>>> will schedule the event in the next perf_guest_exit(). KVM will not know it.
> >>> Ya, the creation of an event on a CPU that currently has guest PMU state loaded
> >>> is what I had in mind when I suggested a callback in my sketch:
> >>>
> >>>    :  D. Add a perf callback that is invoked from IRQ context when perf wants to
> >>>    :     configure a new PMU-based events, *before* actually programming the MSRs,
> >>>    :     and have KVM's callback put the guest PMU state
> >>
> >> when host creates a perf event with exclude_guest attribute which is
> >> used to profile KVM/VMM user space, the vCPU process could work at three
> >> places.
> >>
> >> 1. in guest state (non-root mode)
> >>
> >> 2. inside vcpu-loop
> >>
> >> 3. outside vcpu-loop
> >>
> >> Since the PMU state has already been switched to host state, we don't
> >> need to consider the case 3 and only care about cases 1 and 2.
> >>
> >> when host creates a perf event with exclude_guest attribute to profile
> >> KVM/VMM user space,  an IPI is triggered to enable the perf event
> >> eventually like the following code shows.
> >>
> >> event_function_call(event, __perf_event_enable, NULL);
> >>
> >> For case 1,  a vm-exit is triggered and KVM starts to process the
> >> vm-exit and then run IPI irq handler, exactly speaking
> >> __perf_event_enable() to enable the perf event.
> >>
> >> For case 2, the IPI irq handler would preempt the vcpu-loop and call
> >> __perf_event_enable() to enable the perf event.
> >>
> >> So IMO KVM just needs to provide a callback to switch guest/host PMU
> >> state, and __perf_event_enable() calls this callback before really
> >> touching PMU MSRs.
> > ok, in this case, do we still need KVM to query perf if there are
> > active exclude_guest events? yes? Because there is an ordering issue.
> > The above suggests that the host-level perf profiling comes when a VM
> > is already running, there is an IPI that can invoke the callback and
> > trigger preemption. In this case, KVM should switch the context from
> > guest to host. What if it is the other way around, ie., host-level
> > profiling runs first and then VM runs?
> >
> > In this case, just before entering the vcpu loop, kvm should check
> > whether there is an active host event and save that into a pmu data
> > structure. If none, do the context switch early (so that KVM saves a
> > huge amount of unnecessary PMU context switches in the future).
> > Otherwise, keep the host PMU context until vm-enter. At the time of
> > vm-exit, do the check again using the data stored in pmu structure. If
> > there is an active event do the context switch to the host PMU,
> > otherwise defer that until exiting the vcpu loop. Of course, in the
> > meantime, if there is any perf profiling started causing the IPI, the
> > irq handler calls the callback, preempting the guest PMU context. If
> > that happens, at the time of exiting the vcpu boundary, PMU context
> > switch is skipped since it is already done. Of course, note that the
> > irq could come at any time, so the PMU context switch in all 4
> > locations need to check the state flag (and skip the context switch if
> > needed).
> >
> > So this requires vcpu->pmu has two pieces of state information: 1) the
> > flag similar to TIF_NEED_FPU_LOAD; 2) host perf context info (phase #1
> > just a boolean; phase #2, bitmap of occupied counters).
>
> I still had no chance to look at the details about vFPU implementation,
> currently I have no idea what we need exactly on vPMU side, a flag or a
> callback. Anyway, that's just implementation details, we can look at it
> when starting to implement it.

I think both. The flag helps to decide whether the context switch has
already been done. The callback will always trigger the context
switch, but the context switch code should always check if the switch
has already been done.

FPU context switch is similar but slightly different. That is done at
the host-level context switch boundary and even crossing that boundary
as long as the next process/thread is not using FPU and/or not going
back to userspace. I don't think we want to defer it that far.
Instead, the PMU context switch should still happen within the range
of KVM.

>
> >
> > This is a non-trivial optimization on the PMU context switch. I am
> > thinking about splitting them into the following phases:
> >
> > 1) lazy PMU context switch, i.e., wait until the guest touches PMU MSR
> > for the 1st time.
> > 2) fast PMU context switch on KVM side, i.e., KVM checking event
> > selector value (enable/disable) and selectively switch PMU state
> > (reducing rd/wr msrs)
> > 3) dynamic PMU context boundary, ie., KVM can dynamically choose PMU
> > context switch boundary depending on existing active host-level
> > events.
> > 3.1) more accurate dynamic PMU context switch, ie., KVM checking
> > host-level counter position and further reduces the number of msr
> > accesses.
> > 4) guest PMU context preemption, i.e., any new host-level perf
> > profiling can immediately preempt the guest PMU in the vcpu loop
> > (instead of waiting for the next PMU context switch in KVM).
>
> Great! we have a whole clear picture about the optimization right now.
> BTW, the optimization 1 and 2 are already on our original to-do list. We
> plan to do it after RFC v2 is ready.
>

I am going to summarize that into a design doc. It has been 50 emails
in this thread. I am sure no one has the patience to read our garbage
unless they are involved at the very beginning :)

Any of the implementations are very welcome. 1) and 2) are low hanging
fruit and we can finish that quickly after v2. 3-4 is error prone and
needs further discussions so let's not rush for that.

On the other hand, how do we test this is the question we need to think about.

Thanks.
-Mingwei

Powered by blists - more mailing lists