Message-ID: <54cb5388-e278-41de-be29-d578cc5c37c5@daynix.com>
Date: Fri, 21 Mar 2025 15:20:49 +0900
From: Akihiko Odaki <akihiko.odaki@...nix.com>
To: Marc Zyngier <maz@...nel.org>, Oliver Upton <oliver.upton@...ux.dev>
Cc: Joey Gouly <joey.gouly@....com>, Suzuki K Poulose
 <suzuki.poulose@....com>, Zenghui Yu <yuzenghui@...wei.com>,
 Catalin Marinas <catalin.marinas@....com>, Will Deacon <will@...nel.org>,
 Kees Cook <kees@...nel.org>, "Gustavo A. R. Silva" <gustavoars@...nel.org>,
 linux-arm-kernel@...ts.infradead.org, kvmarm@...ts.linux.dev,
 linux-kernel@...r.kernel.org, linux-hardening@...r.kernel.org,
 devel@...nix.com
Subject: Re: [PATCH RFC] KVM: arm64: PMU: Use multiple host PMUs

On 2025/03/21 2:14, Marc Zyngier wrote:
> On Thu, 20 Mar 2025 09:52:59 +0000,
> Akihiko Odaki <akihiko.odaki@...nix.com> wrote:
>>
>> On 2025/03/20 18:10, Marc Zyngier wrote:
>>> On Thu, 20 Mar 2025 06:03:35 +0000,
>>> Akihiko Odaki <akihiko.odaki@...nix.com> wrote:
>>>>
>>>> On 2025/03/20 3:51, Oliver Upton wrote:
>>>>> On Wed, Mar 19, 2025 at 06:38:38PM +0000, Marc Zyngier wrote:
>>>>>> On Wed, 19 Mar 2025 11:51:21 +0000, Akihiko Odaki <akihiko.odaki@...nix.com> wrote:
>>>>>>> What about setting the flag automatically when a user fails to pin
>>>>>>> vCPUs to CPUs that are covered by one PMU? There would be no change if
>>>>>>> a user correctly pins vCPUs as it is. Otherwise, they will see a
>>>>>>> correct feature set advertised to the guest and the cycle counter
>>>>>>> working.
>>>>>>
>>>>>> How do you know that the affinity is "correct"? VCPU affinity can be
>>>>>> changed at any time. I, for one, do not want my VMs to change
>>>>>> behaviour because I let the vcpus bounce around as the scheduler sees
>>>>>> fit.
>>>>
>>>> Checking the affinity when picking the default PMU; the vCPU affinity
>>>> is the only thing that rules the choice of the default PMU even now.
>>>>
>>>> Perhaps we may model the API as follows: introduce another "composite"
>>>> PMU that works on any core but only exposes the cycle counter. Robust
>>>> VMMs will choose it or one of hardware PMUs with
>>>> KVM_ARM_VCPU_PMU_V3_SET_PMU. KVM will choose the default PMU according
>>>> to the vCPU affinity at the point of KVM_ARM_VCPU_INIT otherwise. If
>>>> the affinity is covered by one hardware PMU, that PMU will be chosen
>>>> as the default. The "composite" PMU will be the default otherwise.
>>>
>>> This makes no sense to me. A VCPU is always affine to a PMU, because
>>> we do not support configurations where only some CPUs have a PMU. This
>>> is an all-or-nothing situation.
>>
>> At least isn't it fine to have the composite PMU with a new value for
>> KVM_ARM_VCPU_PMU_V3_SET_PMU?
> 
> Not sure KVM_ARM_VCPU_PMU_V3_SET_PMU is the right hook (it takes a PMU
> 'type', which is under control of the perf subsystem). But if we can
> find a value that is guaranteed to be unique, why not.
> 
>>> More importantly, you keep suggesting the same "new default", and I
>>> keep saying NO.
>>>
>>> My position is clear: if you want a *new* behaviour, you *must* add a
>>> new flag that the VMM explicitly provides to enable this CC-only PMU.
>>> No change in default behaviour at all.
>>>
>>> I'm not going to move from that.
>>
>> Why not? It will not break anything guaranteed to work in the past.
> 
> It doesn't have to be guaranteed. It just has to *exist*. That's the
> Linux ABI for you.
> 
>> Currently KVM only guarantees that the emulated PMU correctly counts
>> only when
>> 1) the vCPU affinity is contained by one PMU and
>> 2) it will not expand
>>
>> Breaking these conditions will make the behavior of the emulated PMU
>> undefined. Now I'm proposing to remove 1).
> 
> And I'm saying no to that. I'm also getting tired of arguing the same
> point on and on.
> 
> We currently have two different behaviours:
> 
> - either you explicitly set a PMU, and the affinity of this PMU
>    constraints the affinity of the vcpus. The vcpus are not allowed to
>    run outside of this affinity. Everything counts all the time.
> 
> - or you don't explicitly set a PMU, and a default PMU will be picked
>    from the current affinity of the first vcpu. Your vcpus will be able
>    to run anywhere, but the virtual PMU will *only* count when the
>    vcpus are affine to the default PMU. When the vcpus are not affine
>    to default PMU, *nothing* counts.
> 
> These two behaviours are ABI. They are not changing. They don't get
> relaxed, they don't get tightened, they stay just as they are,
> forever.

Is the latter one is really ABI? I see it as part of behaviors that are 
undefined and not included in ABI for the following reasons:

1) It depends on the scheduler behavior, which cannot be ABI.

2) It provides a broken PMU so the proposed behavioral change is similar 
a bug fix though I call it a undefined behavior instead of a bug as it 
is explicitly told that there is no assurance that the PMU works in such 
a scenario.

3) The userspace could not have relied on it so the "no regressions" 
rule cannot be applied here; how can anyone have a userspace that relies 
on a kernel behavior that depends on scheduling?

But for 3), Oliver raised a concern for the guest compatibility so I'd 
like to hear an explanation for that concern.

> 
> You want a *third* behaviour, go ahead. Define it the way you want.
> But the behaviours described above will stay unchanged.> > I'm looking forward to your patches implementing it, but I am also
> done arguing on it.

I understand the discussion is tiring but I want to know the reasoning 
behind such a design decision before sending an RFC patch to a VMM 
(QEMU) so that I can explain them why it is necessary in turn.

Regards,
Akihiko Odaki

> 
> 	M.
> 

Powered by blists - more mailing lists