linux-kernel - Re: KVM: x86/vPMU/AMD: Can we detect PMU is off for a VM?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <971a67f9-fa21-42d5-b722-611bdfdc760e@virtuozzo.com>
Date:   Mon, 13 Nov 2023 17:33:35 +0100
From:   "Denis V. Lunev" <den@...tuozzo.com>
To:     Dongli Zhang <dongli.zhang@...cle.com>,
        Konstantin Khorenko <khorenko@...tuozzo.com>
Cc:     Sean Christopherson <seanjc@...gle.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
        Dave Hansen <dave.hansen@...ux.intel.com>, x86@...nel.org,
        "H . Peter Anvin" <hpa@...or.com>, kvm@...r.kernel.org,
        linux-kernel@...r.kernel.org,
        Alexander Ivanov <alexander.ivanov@...tuozzo.com>,
        Jim Mattson <jmattson@...gle.com>
Subject: Re: KVM: x86/vPMU/AMD: Can we detect PMU is off for a VM?

On 11/13/23 17:17, Dongli Zhang wrote:
>
> On 11/13/23 06:42, Denis V. Lunev wrote:
>> On 11/13/23 15:14, Dongli Zhang wrote:
>>> Hi Denis,
>>>
>>> On 11/13/23 01:31, Denis V. Lunev wrote:
>>>> On 11/10/23 01:01, Dongli Zhang wrote:
>>>>> On 11/9/23 3:46 PM, Denis V. Lunev wrote:
>>>>>> On 11/9/23 23:52, Jim Mattson wrote:
>>>>>>> On Thu, Nov 9, 2023 at 10:18 AM Konstantin Khorenko
>>>>>>> <khorenko@...tuozzo.com> wrote:
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> as a followup for my patch: i have noticed that
>>>>>>>> currently Intel kernel code provides an ability to detect if PMU is totally
>>>>>>>> disabled for a VM
>>>>>>>> (pmu->version == 0 in this case), but for AMD code pmu->version is never 0,
>>>>>>>> no matter if PMU is enabled or disabled for a VM (i mean <pmu state='off'/>
>>>>>>>> in the VM config which
>>>>>>>> results in "-cpu pmu=off" qemu option).
>>>>>>>>
>>>>>>>> So the question is - is it possible to enhance the code for AMD to also
>>>>>>>> honor
>>>>>>>> PMU VM setting or it is
>>>>>>>> impossible by design?
>>>>>>> The AMD architectural specification prior to AMD PMU v2 does not allow
>>>>>>> one to describe a CPU (via CPUID or MSRs) that has fewer than 4
>>>>>>> general purpose PMU counters. While AMD PMU v2 does allow one to
>>>>>>> describe such a CPU, legacy software that knows nothing of AMD PMU v2
>>>>>>> can expect four counters regardless.
>>>>>>>
>>>>>>> Having said that, KVM does provide a per-VM capability for disabling
>>>>>>> the virtual PMU: KVM_CAP_PMU_CAPABILITY(KVM_PMU_CAP_DISABLE). See
>>>>>>> section 8.35 in Documentation/virt/kvm/api.rst.
>>>>>> But this means in particular that QEMU should immediately
>>>>>> use this KVM_PMU_CAP_DISABLE if this capability is supported and PMU=off. I am
>>>>>> not seeing this code thus I believe that we have missed this. I think that
>>>>>> this
>>>>>> change worth adding. We will measure the impact :-) Den
>>>>>>
>>>>> I used to have a patch to use KVM_PMU_CAP_DISABLE in QEMU, but that did not
>>>>> draw
>>>>> many developers' attention.
>>>>>
>>>>> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/20230621013821.6874-2-dongli.zhang@oracle.com/__;!!ACWV5N9M2RV99hQ!McSH2M-kuHmzAwTuXKxrjLkrdJoPqML6cY_Ndc-8k9LRQ7D1V9bSBRQPwHqtx9XCVLK3uzdsMaxyfwve$
>>>>> It is time to first re-send that again.
>>>>>
>>>>> Dongli Zhang
>>>> We have checked that setting KVM_PMU_CAP_DISABLE really helps. Konstantin has
>>>> done this and this is good. On the other hand, looking into these patches I
>>>> disagree with them. We should not introduce new option for QEMU. If PMU is
>>>> disabled, i.e. we assume that pmu=off passed in the command line, we should set
>>>> KVM_PMU_CAP_DISABLE for that virtual machine. Den
>>> Can I assume you meant pmu=off, that is, cpu->enable_pmu in QEMU?
>>>
>>> In my opinion, cpu->enable_pmu indicates the option to control the cpu features.
>>> It may be used by any accelerators, and it is orthogonal to the KVM cap.
>>>
>>>
>>> The KVM_PMU_CAP_DISABLE is only specific to the KVM accelerator.
>>>
>>>
>>> That's why I had introduced a new option, to allow to configure the VM in my
>>> dimensions.
>>>
>>> It means one dimension to AMD, but two for Intel: to disable PMU via cpuid, or
>>> KVM cap.
>>>
>>> Anyway, this is KVM mailing list, and I may initiate the discussion in QEMU list.
>>>
>>> Thank you very much!
>>>
>>> Dongli Zhang
>> with the option pmu='off' it is expected that PMU should be
>> off for the guest. At the moment (without this KVM capability)
>> we can disable PMU for Intel only and thus have performance
>> degradation on AMD.
>>
>> This option disables PMU and thus normally when we are
>> running KVM guest and wanting PMU to be off it would
>> be required to
>> * disable CPUID leaf for Intel
>> * set KVM_PMU_CAP_DISABLE for both processors This would be quite natural and
>> transparent for the libvirt. Alexander will prepare the patch today or tomorrow
>> for the discussion. Den
> That is what I had implemented in the v1 of patch.
>
> https://lore.kernel.org/all/20221119122901.2469-3-dongli.zhang@oracle.com/
>
> However, I changed that after people suggested introduce a new property.
>
> Dongli Zhang
That would save a bit of our work :)

For me this patch looks absolutely awesome and is doing exactly
what I want to do in our downstream. This would get us required
15+% benefit for each VMexit.

Den