Message-ID: <Zzd1giMm-bzUiZgA@arm.com>
Date: Fri, 15 Nov 2024 16:23:30 +0000
From: Catalin Marinas <catalin.marinas@....com>
To: Marc Zyngier <maz@...nel.org>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@...nel.org>,
	linux-kernel@...r.kernel.org, linux-arm-kernel@...ts.infradead.org,
	kvmarm@...ts.linux.dev, Suzuki K Poulose <Suzuki.Poulose@....com>,
	Steven Price <steven.price@....com>, Will Deacon <will@...nel.org>,
	Mark Rutland <mark.rutland@....com>,
	Oliver Upton <oliver.upton@...ux.dev>,
	Joey Gouly <joey.gouly@....com>, Zenghui Yu <yuzenghui@...wei.com>
Subject: Re: [PATCH 4/4] arm64: mte: Use stage-2 NoTagAccess memory attribute
 if supported

(joining the thread as well, though not sure I'm bringing anything new)

On Tue, Nov 12, 2024 at 11:51:45AM +0000, Marc Zyngier wrote:
> On Fri, 08 Nov 2024 07:59:31 +0000, Aneesh Kumar K.V <aneesh.kumar@...nel.org> wrote:
> > Marc Zyngier <maz@...nel.org> writes:
> > > On Mon, 28 Oct 2024 13:28:42 +0000, Aneesh Kumar K.V <aneesh.kumar@...nel.org> wrote:
> > >> Marc Zyngier <maz@...nel.org> writes:
> > >> > On Mon, 28 Oct 2024 09:40:14 +0000, "Aneesh Kumar K.V (Arm)" <aneesh.kumar@...nel.org> wrote:
> > >> >> Currently, the kernel won't start a guest if the MTE feature is enabled
> > >> >> and the guest RAM is backed by memory which doesn't support access tags.
> > >> >> Update this such that the kernel uses the NoTagAccess memory attribute
> > >> >> while mapping pages from VMAs for which MTE is not allowed. The fault
> > >> >> from accessing the access tags with such pages is forwarded to VMM so
> > >> >> that VMM can decide to kill the guest or remap the pages so that
> > >> >> access tag storage is allowed.
> > >> >
> > >> > I only have questions here:
> > >> >
> > >> > - what is the benefit of such approach? why shouldn't that be the
> > >> >   kernel's job to fix it?
> > >>
> > >> IMHO leaving that policy decision to VMM makes the kernel changes
> > >> simpler. In most cases, VMM will kill the guest, because these
> > >> restrictions of MTE_ALLOWED are applied at the memslot/vma.
> > >
> > > Where is that captured? The whole idea behind FEAT_MTE_PERM was that
> > > it would be the hypervisor's task to lazily allocate MTE-capable
> > > memory as tagged-access would occur.
> > 
> > Lazily allocating MTE-capable memory requires changes to different
> > kernel subsystems and previous attempts got dropped [1] because it
> > was not clear whether the benefit of saving 3% memory overhead was worth
> > the complexity we add to the kernel.

I'd say the most complex part in Alex's approach was the need to reuse
the tag storage for classic data and kick the pages around when some
other page needs to store tags in there. That approach is pretty much
dead.

In theory, having MTE and non-MTE memory (heterogeneous) without a
carveout reuse would be a bit more manageable - not that far from the
NUMA migration and at least you only migrate the page being accessed
rather than unrelated ones where the tags need to go. But this was not
Alex's goal for Android since people were asking for the reuse of the 3%
carveout rather than a smaller carveout.

Other future deployments, CXL-attached memory etc. may benefit from a
new scheme but I wouldn't rush in implementing anything in the kernel
for now.

The VMM may be in a better position to manage such heterogeneous memory
for the guest if it knows the capabilities of the slots (e.g. some DAX
mmap() vs anonymous mmap()). This would require the VMM replacing a page
within a slot from one memory type to another (while preserving the
data). I don't think we have a concrete use-case yet to be worth the
hassle.

> That's not the point. Tagged memory doesn't have to cover the whole of
> physical memory, and it can be statically allocated. The architecture
> doesn't mandate that all of the memory is MTE-capable.

There's some vague wording that general purpose memory should be MTE
capable if FEAT_MTE2 or later is advertised. But that's not well defined
and one can have other types of memory in the physical space (e.g. CXL)
that don't support tags. Last time I looked we still haven't got a way
to describe memory capabilities in firmware.

For the time being, I think a real use-case for FEAT_MTE_PERM is in the
context of cacheable MMIO (there are some patches around from Nvidia to
do this with VFIO). That memory, if exposed to guest as WB and the guest
enables MTE, may trigger some SErrors. With FEAT_MTE_PERM KVM could trap
and inject a fault back into the guest - maybe SEA. Is it easier to do
this from KVM itself or we would rather exit to the VMM and let it
handle? The latter allows room for other fancier things in the future
but the former may be quicker, in the absence of other strong use-cases.

> > This patchset is not looking at that feature. Instead, it can be used to
> > enable MTE in configurations that currently won't allow MTE. One such
> > example is libkrun which includes linux kernel as firmware in a
> > dynamically linked library (libkrunfw). libkrun can insert the kernel
> > region which got mmaped as part of the library load, directly into the
> > guest memory map instead of copying the kernel. Such a guest config
> > can't enable MTE currently even though we will never use the newly
> > inserted memory regions as tag access memory.

I've never played with libkrunfw. Does it handle inserting a Linux
kernel? Such approach may not work well with MTE. The kernel frees the
init text/data sections back into the page allocator. With MTE
advertised as present, the guest will try to reuse that memory,
potentially as tagged. However, since the VMM mmap'ed the guest kernel
from a file, VM_MTE is not supported in the VMM address space. Even if
the mapping is MAP_PRIVATE and the page copied on write, the vma remains
the original one associated with the file, so VM_MTE_ALLOWED not set.

We could revisit this and allow mprotect(PROT_MTE) to force the CoW on
private file mappings but we'd need some strong requirement for this
(MTE+libkrunfw could be such thing if people need this combination).

> > Similarly, virtiofs dax support can use a page cache region as
> > virtio-shm region. We can use MTE_PERM to enable MTE in this config.
> 
> And this use case doesn't contradict what I am stating above. But it
> definitely contradicts what you wrote: "In most cases, VMM will kill
> the guest".

To simplify things, I think whatever is presented to the VM as standard
RAM (typically present at the VM boot) should either support MTE or MTE
will be disabled for the guest. For other types of memory, whether WB
MMIO or RAM presented as virtio-(pmem, shm etc.) backed by files in the
VMM, the VM should be aware it is not standard RAM and should not
attempt to enable MTE on it. If it does (either by mistake or malice),
FEAT_MTE_PERM should trap and inject a fault back into the guest (or
kill it altogether but for debugging, I'd rather inject a fault if
possible).

-- 
Catalin

Powered by blists - more mailing lists