linux-kernel - RE: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <d13c4cb44a2b4b42a8b534c38c402a1d@huawei.com>
Date:   Tue, 25 Jul 2023 00:56:21 +0000
From:   Salil Mehta <salil.mehta@...wei.com>
To:     bibo mao <maobibo@...ngson.cn>
CC:     Catalin Marinas <catalin.marinas@....com>,
        Jonathan Cameron <jonathan.cameron@...wei.com>,
        Marc Zyngier <maz@...nel.org>, Will Deacon <will@...nel.org>,
        "christoffer.dall@....com" <christoffer.dall@....com>,
        "oliver.upton@...ux.dev" <oliver.upton@...ux.dev>,
        "mark.rutland@....com" <mark.rutland@....com>,
        "pbonzini@...hat.com" <pbonzini@...hat.com>,
        "Salil Mehta" <salil.mehta@...src.net>,
        "andrew.jones@...ux.dev" <andrew.jones@...ux.dev>,
        yuzenghui <yuzenghui@...wei.com>,
        "kvmarm@...ts.cs.columbia.edu" <kvmarm@...ts.cs.columbia.edu>,
        "linux-arm-kernel@...ts.infradead.org" 
        <linux-arm-kernel@...ts.infradead.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "kvm@...r.kernel.org" <kvm@...r.kernel.org>,
        "qemu-devel@...gnu.org" <qemu-devel@...gnu.org>,
        "james.morse@....com" <james.morse@....com>,
        "steven.price@....com" <steven.price@....com>,
        "Suzuki K Poulose" <suzuki.poulose@....com>,
        Jean-Philippe Brucker <jean-philippe@...aro.org>,
        "kvmarm@...ts.linux.dev" <kvmarm@...ts.linux.dev>,
        "linux-coco@...ts.linux.dev" <linux-coco@...ts.linux.dev>
Subject: RE: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world
 might require ARM spec change?

Hi Bibo,

> From: bibo mao <maobibo@...ngson.cn>
> Sent: Tuesday, July 25, 2023 1:29 AM
> To: Salil Mehta <salil.mehta@...wei.com>
> Cc: Catalin Marinas <catalin.marinas@....com>; Jonathan Cameron
> <jonathan.cameron@...wei.com>; Marc Zyngier <maz@...nel.org>; Will Deacon
> <will@...nel.org>; christoffer.dall@....com; oliver.upton@...ux.dev;
> mark.rutland@....com; pbonzini@...hat.com; Salil Mehta
> <salil.mehta@...src.net>; andrew.jones@...ux.dev; yuzenghui
> <yuzenghui@...wei.com>; kvmarm@...ts.cs.columbia.edu; linux-arm-
> kernel@...ts.infradead.org; linux-kernel@...r.kernel.org;
> kvm@...r.kernel.org; qemu-devel@...gnu.org; james.morse@....com;
> steven.price@....com; Suzuki K Poulose <suzuki.poulose@....com>; Jean-
> Philippe Brucker <jean-philippe@...aro.org>; kvmarm@...ts.linux.dev; linux-
> coco@...ts.linux.dev
> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world
> might require ARM spec change?
> 
> Is vcpu hotplug supported in arm virt-machine now?

Not yet. We are working on it. Please check the RFCs being tested.


[1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch  (WIP)
    https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
[2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug  
    https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
    
    
> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> I do not know why it is not merged.


I think you are referring to patches [3], [4]? Please follow the discussion
for details. 


[3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
    https://lore.kernel.org/all/20200613213629.21984-1-salil.mehta@huawei.com/
[4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
    https://lore.kernel.org/all/20200625133757.22332-1-salil.mehta@huawei.com/#r


In summary, there were some ARM64 Architecture constraints which were being
violated in the earlier patches of the kernel [4] so we had to re-think of the
kernel changes. The Qemu part mostly remains same with some new introductions
of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
in VMM/Qemu. 


You can follow the KVMForum conference presentations [5], [6] delivered in the
year 2020 and 2023 to get hold of more details related to this.


[5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
    https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
[6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
    https://kvmforum2020.sched.com/event/eE4m



> I ask this question because we propose
> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> modified for vcpu hotplug.


Could you please share the link of your patches so that we can have a look and
draw a comparison?


Thanks
Salil.

> 
> Regards
> Bibo, mao
> 
> 在 2023/7/19 10:35, Salil Mehta 写道:
> > [Reposting it here from Linaro Open Discussion List for more eyes to look
> at]
> >
> > Hello,
> > I have recently started to dabble with ARM CCA stuff and check if our
> > recent changes to support vCPU Hotplug in ARM64 can work in the realm
> > world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> > command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> > from the host. This might be required to support vCPU Hotplug feature
> > in the realm world in future. vCPU Hotplug is an important feature to
> > support kata-containers in realm world as it reduces the VM boot time
> > and facilitates dynamic adjustment of vCPUs (which I think should be
> > true even with Realm world as current implementation only makes use
> > of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >
> >
> > As per our recent changes [2], [3] related to support vCPU Hotplug on
> > ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> > user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> > PSCI_CPU_ON should undergo similar policy checks and I think,
> >
> > 1. Host should *deny* to online the target vCPUs which are NOT plugged
> > 2. This means target REC should be denied by host. Can host call
> >    RMI_PSCI_COMPETE in such s case?
> > 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> > 4. Failure condition (B5.3.3.2) should be amended with
> >    runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >             post: result == PSCI_DENIED (?)
> > 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >    PSCI_CPU_ON flow (D1.4.1)
> >
> >
> > I do understand that ARM CCA support is in its infancy stage and
> > discussing about vCPU Hotplug in realm world seem to be a far-fetched
> > idea right now. But specification changes require lot of time and if
> > this change is really required then it should be further discussed
> > within ARM.
> >
> > Many thanks!
> >
> >
> > Bes regards
> > Salil
> >
> >
> > References:
> >
> > [1] https://developer.arm.com/documentation/den0137/latest/
> > [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> > [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2