Message-ID: <c6b49c81-58ba-40fc-9e17-892fdd79431d@arm.com>
Date: Wed, 26 Jun 2024 14:39:27 +0100
From: Steven Price <steven.price@....com>
To: Itaru Kitayama <itaru.kitayama@...ux.dev>
Cc: kvm@...r.kernel.org, kvmarm@...ts.linux.dev,
 Catalin Marinas <catalin.marinas@....com>, Marc Zyngier <maz@...nel.org>,
 Will Deacon <will@...nel.org>, James Morse <james.morse@....com>,
 Oliver Upton <oliver.upton@...ux.dev>,
 Suzuki K Poulose <suzuki.poulose@....com>, Zenghui Yu
 <yuzenghui@...wei.com>, linux-arm-kernel@...ts.infradead.org,
 linux-kernel@...r.kernel.org, Joey Gouly <joey.gouly@....com>,
 Alexandru Elisei <alexandru.elisei@....com>,
 Christoffer Dall <christoffer.dall@....com>, Fuad Tabba <tabba@...gle.com>,
 linux-coco@...ts.linux.dev,
 Ganapatrao Kulkarni <gankulkarni@...amperecomputing.com>
Subject: Re: [v2] Support for Arm CCA VMs on Linux

On 24/06/2024 07:13, Itaru Kitayama wrote:
> Hi Steven,
> On Fri, Apr 12, 2024 at 09:40:56AM +0100, Steven Price wrote:
>> We are happy to announce the second version of the Arm Confidential
>> Compute Architecture (CCA) support for the Linux stack. The intention is
>> to seek early feedback in the following areas:
>>  * KVM integration of the Arm CCA;
>>  * KVM UABI for managing the Realms, seeking to generalise the
>>    operations where possible with other Confidential Compute solutions;
>>  * Linux Guest support for Realms.
>>
>> See the previous RFC[1] for a more detailed overview of Arm's CCA
>> solution, or visible the Arm CCA Landing page[2].
>>
>> This series is based on the final RMM v1.0 (EAC5) specification[3].
>>
>> Quick-start guide
>> =================
>>
>> The easiest way of getting started with the stack is by using
>> Shrinkwrap[4]. Currently Shrinkwrap has a configuration for the initial
>> v1.0-EAC5 release[5], so the following overlay needs to be applied to
>> the standard 'cca-3world.yaml' file. Note that the 'rmm' component needs
>> updating to 'main' because there are fixes that are needed and are not
>> yet in a tagged release. The following will create an overlay file and
>> build a working environment:
>>
>> cat<<EOT >cca-v2.yaml
>> build:
>>   linux:
>>     repo:
>>       revision: cca-full/v2
>>   kvmtool:
>>     repo:
>>       kvmtool:
>>         revision: cca/v2
>>   rmm:
>>     repo:
>>       revision: main
>>   kvm-unit-tests:
>>     repo:
>>       revision: cca/v2
>> EOT
>>
>> shrinkwrap build cca-3world.yaml --overlay buildroot.yaml --btvar GUEST_ROOTFS='${artifact:BUILDROOT}' --overlay cca-v2.yaml
>>
>> You will then want to modify the 'guest-disk.img' to include the files
>> necessary for the realm guest (see the documentation in cca-3world.yaml
>> for details of other options):
>>
>>   cd ~/.shrinkwrap/package/cca-3world
>>   /sbin/e2fsck -fp rootfs.ext2 
>>   /sbin/resize2fs rootfs.ext2 256M
>>   mkdir mnt
>>   sudo mount rootfs.ext2 mnt/
>>   sudo mkdir mnt/cca
>>   sudo cp guest-disk.img KVMTOOL_EFI.fd lkvm Image mnt/cca/
>>   sudo umount mnt 
>>   rmdir mnt/
>>
>> Finally you can run the FVP with the host:
>>
>>   shrinkwrap run cca-3world.yaml --rtvar ROOTFS=$HOME/.shrinkwrap/package/cca-3world/rootfs.ext2
>>
>> And once the host kernel has booted, login (user name 'root') and start
>> a realm guest:
>>
>>   cd /cca
>>   ./lkvm run --realm --restricted_mem -c 2 -m 256 -k Image -p earlycon
>>
>> Be patient and you should end up in a realm guest with the host's
>> filesystem mounted via p9.
>>
>> It's also possible to use EFI within the realm guest, again see
>> cca-3world.yaml within Shrinkwrap for more details.
> 
> I am trying to see if libvirt can work with the CCA-aware KVM with minimal Ubuntu22.10 filesystem, however virt-install triggers a system failure:
> 
> $ sudo virt-install -v --name f39 --ram 4096        --disk path=fedora40.img,cache=none --nographics --os-variant fedora38         --import --arch aarch64 --vcpus 4
> [sudo] password for realm:
> [ 3694.176579] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000e00
> [ 3694.176687] Mem abort info:
> [ 3694.176745]   ESR = 0x0000000096000004
> [ 3694.176817]   EC = 0x25: DABT (current EL), IL = 32 bits
> [ 3694.176907]   SET = 0, FnV = 0
> [ 3694.176978]   EA = 0, S1PTW = 0
> [ 3694.177049]   FSC = 0x04: level 0 translation fault
> [ 3694.177132] Data abort info:
> [ 3694.177189]   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
> [ 3694.177276]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
> [ 3694.177370]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> [ 3694.177544] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000880f6e000
> [ 3694.177649] [0000000000000e00] pgd=0000000000000000, p4d=0000000000000000
> [ 3694.177788] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
> [ 3694.177887] Modules linked in:
> [ 3694.177966] CPU: 2 PID: 540 Comm: qemu-system-aar Not tainted 6.10.0-rc1-00058-gd901c27a1783 #149
> [ 3694.178105] Hardware name: FVP Base RevC (DT)
> [ 3694.178180] pstate: 61400009 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> [ 3694.178315] pc : kvm_vm_ioctl_check_extension+0x1fc/0x3c4
> [ 3694.178447] lr : kvm_vm_ioctl_check_extension_generic+0x34/0x12c
> [ 3694.178587] sp : ffff800081523cb0
> [ 3694.178657] x29: ffff800081523cb0 x28: 0000000000000051 x27: 0000000000000000
> [ 3694.178840] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
> [ 3694.179019] x23: 000000000000000a x22: 0000000000000051 x21: ffff000801075f00
> [ 3694.179200] x20: ffff000801075f01 x19: 000000000000ae03 x18: 0000000000000000
> [ 3694.179383] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> [ 3694.179565] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> [ 3694.179745] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
> [ 3694.179923] x8 : 0000000000000000 x7 : ffff000801075f18 x6 : 00000000401c5820
> [ 3694.180106] x5 : 000000000000000a x4 : 0000000000000800 x3 : 0000000000000000
> [ 3694.180285] x2 : 000000000000000b x1 : 0000000100061001 x0 : 0000000000000001
> [ 3694.180465] Call trace:
> [ 3694.180523]  kvm_vm_ioctl_check_extension+0x1fc/0x3c4
> [ 3694.180656]  kvm_vm_ioctl_check_extension_generic+0x34/0x12c
> [ 3694.180794]  kvm_dev_ioctl+0x3c8/0x8b8
> [ 3694.180938]  __arm64_sys_ioctl+0xac/0xf0
> [ 3694.181079]  invoke_syscall+0x48/0x114
> [ 3694.181220]  el0_svc_common.constprop.0+0x40/0xe0
> [ 3694.181367]  do_el0_svc+0x1c/0x28
> [ 3694.181507]  el0_svc+0x34/0xd8
> [ 3694.181608]  el0t_64_sync_handler+0x120/0x12c
> [ 3694.181723]  el0t_64_sync+0x190/0x194
> [ 3694.181865] Code: 17ffffbd 97fffc9d 12001c00 17ffff91 (39780060)
> [ 3694.181955] ---[ end trace 0000000000000000 ]---
> 
> I'd appreciate it if you could take a look at it.

Thanks for the bug report. I believe this is because 
kvm_vm_ioctl_check_extension() is being called with kvm==NULL and I've 
missed some checks. I believe the following should get things working - 
and it's probably better than attempting to remember to check with the 
NULL kvm at each call site.

---8<----
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index 27c58bbdf50b..c85e5f566506 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -602,7 +602,7 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu)
 
 static inline bool kvm_is_realm(struct kvm *kvm)
 {
-       if (static_branch_unlikely(&kvm_rme_is_available))
+       if (static_branch_unlikely(&kvm_rme_is_available) && kvm)
                return kvm->arch.is_realm;
        return false;
 }
---8<----

Thanks,
Steve

Powered by blists - more mailing lists