Message-ID:
 <PAWPR08MB89092CC3B8587E9938CCAA0C9FD22@PAWPR08MB8909.eurprd08.prod.outlook.com>
Date: Fri, 14 Mar 2025 01:49:18 +0000
From: Wathsala Wathawana Vithanage <wathsala.vithanage@....com>
To: "Tian, Kevin" <kevin.tian@...el.com>, Alex Williamson
	<alex.williamson@...hat.com>
CC: Jason Gunthorpe <jgg@...pe.ca>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>, nd <nd@....com>, Philipp Stanner
	<pstanner@...hat.com>, Yunxiang Li <Yunxiang.Li@....com>, "Dr. David Alan
 Gilbert" <linux@...blig.org>, Ankit Agrawal <ankita@...dia.com>, "open
 list:VFIO DRIVER" <kvm@...r.kernel.org>, Dhruv Tripathi
	<Dhruv.Tripathi@....com>, Honnappa Nagarahalli
	<Honnappa.Nagarahalli@....com>, Jeremy Linton <Jeremy.Linton@....com>
Subject: RE: [RFC PATCH] vfio/pci: add PCIe TPH to device feature ioctl



> -----Original Message-----
> From: Tian, Kevin <kevin.tian@...el.com>
> Sent: Wednesday, March 12, 2025 2:53 AM
> To: Wathsala Wathawana Vithanage <wathsala.vithanage@....com>; Alex
> Williamson <alex.williamson@...hat.com>
> Cc: Jason Gunthorpe <jgg@...pe.ca>; linux-kernel@...r.kernel.org; nd
> <nd@....com>; Philipp Stanner <pstanner@...hat.com>; Yunxiang Li
> <Yunxiang.Li@....com>; Dr. David Alan Gilbert <linux@...blig.org>; Ankit
> Agrawal <ankita@...dia.com>; open list:VFIO DRIVER <kvm@...r.kernel.org>;
> Dhruv Tripathi <Dhruv.Tripathi@....com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@....com>; Jeremy Linton <Jeremy.Linton@....com>
> Subject: RE: [RFC PATCH] vfio/pci: add PCIe TPH to device feature ioctl
> 
> > From: Wathsala Wathawana Vithanage <wathsala.vithanage@....com>
> > Sent: Wednesday, March 5, 2025 2:11 PM
> >
> > > -----Original Message-----
> > > From: Alex Williamson <alex.williamson@...hat.com>
> > > Sent: Tuesday, March 4, 2025 7:24 PM
> > > To: Wathsala Wathawana Vithanage <wathsala.vithanage@....com>
> > > Cc: Jason Gunthorpe <jgg@...pe.ca>; linux-kernel@...r.kernel.org; nd
> > > <nd@....com>; Kevin Tian <kevin.tian@...el.com>; Philipp Stanner
> > > <pstanner@...hat.com>; Yunxiang Li <Yunxiang.Li@....com>; Dr. David
> > Alan
> > > Gilbert <linux@...blig.org>; Ankit Agrawal <ankita@...dia.com>; open
> > list:VFIO
> > > DRIVER <kvm@...r.kernel.org>
> > > Subject: Re: [RFC PATCH] vfio/pci: add PCIe TPH to device feature ioctl
> > >
> > > On Tue, 4 Mar 2025 22:38:16 +0000
> > > Wathsala Wathawana Vithanage <wathsala.vithanage@....com> wrote:
> > >
> > > > > > Linux v6.13 introduced the PCIe TLP Processing Hints (TPH) feature for
> > > > > > direct cache injection. As described in the relevant patch set [1],
> > > > > > direct cache injection in supported hardware allows optimal platform
> > > > > > resource utilization for specific requests on the PCIe bus. This feature
> > > > > > is currently available only for kernel device drivers. However,
> > > > > > user space applications, especially those whose performance is
> > sensitive
> > > > > > to the latency of inbound writes as seen by a CPU core, may benefit
> > from
> > > > > > using this information (E.g., DPDK cache stashing RFC [2] or an HPC
> > > > > > application running in a VM).
> > > > > >
> > > > > > This patch enables configuring of TPH from the user space via
> > > > > > VFIO_DEVICE_FEATURE IOCLT. It provides an interface to user space
> > > > > > drivers and VMMs to enable/disable the TPH feature on PCIe devices
> > and
> > > > > > set steering tags in MSI-X or steering-tag table entries using
> > > > > > VFIO_DEVICE_FEATURE_SET flag or read steering tags from the kernel
> > using
> > > > > > VFIO_DEVICE_FEATURE_GET to operate in device-specific mode.
> > > > >
> > > > > What level of protection do we expect to have here? Is it OK for
> > > > > userspace to make up any old tag value or is there some security
> > > > > concern with that?
> > > > >
> > > > Shouldn't be allowed from within a container.
> > > > A hypervisor should have its own STs and map them to platform STs for
> > > > the cores the VM is pinned to and verify any old ST is not written to the
> > > > device MSI-X, ST table or device specific locations.
> > >
> > > And how exactly are we mediating device specific steering tags when we
> > > don't know where/how they're written to the device.  An API that
> > > returns a valid ST to userspace doesn't provide any guarantees relative
> > > to what userspace later writes.  MSI-X tables are also writable by
> >
> > By not enabling TPH in device-specific mode, hypervisors can ensure that
> > setting an ST in a device-specific location (like queue contexts) will have no
> > effect. VMs should also not be allowed to enable TPH. I believe this could
> > be enforced by trapping (causing VM exits) on MSI-X/ST table writes.
> 
> Probably we should not allow device-specific mode unless the user is
> capable of CAP_SYS_RAWIO? It allows an user to pollute caches on

Sounds plausible. 

> CPUs which its processes are not affined to, hence could easily break
> SLAs which CSPs try to achieve...
>
> Interrupt vector mode sounds safer as it only needs to provide an
> enable/disable cmd to the user and it's the kernel VFIO driver
> managing the steering table, e.g. also in irq affinity handler.
> 
> >
> > Having said that, regardless of this proposal or the availability of kernel
> > TPH support, a VFIO driver could enable TPH and set an arbitrary ST on the
> > MSI-X/ST table or a device-specific location on supported platforms. If the
> > driver doesn't have a list of valid STs, it can enumerate 8- or 16-bit STs and
> > measure access latencies to determine valid ones.
> >
> 
> PCI capabilities are managed by the kernel VFIO driver. So w/o this
> patch no userspace driver can enable TPH to try that trick?

Yes, it's possible. It's just a matter of setting the right bits in the PCI config
space to enable TPH on the device.

Thanks 
--wathsala

Powered by blists - more mailing lists