Message-ID: <606f65e1-ccfc-4492-a32f-90343be654e7@gmail.com>
Date: Sun, 27 Jul 2025 20:48:26 +0800
From: Ethan Zhao <etzhao1900@...il.com>
To: Nicolin Chen <nicolinc@...dia.com>
Cc: jgg@...dia.com, joro@...tes.org, will@...nel.org, robin.murphy@....com,
 rafael@...nel.org, lenb@...nel.org, bhelgaas@...gle.com,
 iommu@...ts.linux.dev, linux-kernel@...r.kernel.org,
 linux-acpi@...r.kernel.org, linux-pci@...r.kernel.org,
 patches@...ts.linux.dev, pjaroszynski@...dia.com, vsethi@...dia.com,
 helgaas@...nel.org, baolu.lu@...ux.intel.com
Subject: Re: [PATCH RFC v2 0/4] Disable ATS via iommu during PCI resets



On 7/26/2025 12:41 AM, Nicolin Chen wrote:
> On Thu, Jul 24, 2025 at 02:50:53PM +0800, Ethan Zhao wrote:
>> On 6/28/2025 3:42 PM, Nicolin Chen wrote:
>>> PCIe permits a device to ignore ATS invalidation TLPs, while processing a
>>> reset. This creates a problem visible to the OS where an ATS invalidation
>>> command will time out: e.g. an SVA domain will have no coordination with a
>>> reset event and can racily issue ATS invalidations to a resetting device.
>>>
>>> The OS should do something to mitigate this as we do not want production
>>> systems to be reporting critical ATS failures, especially in a hypervisor
>>> environment. Broadly, OS could arrange to ignore the timeouts, block page
>>> table mutations to prevent invalidations, or disable and block ATS.
>>>
>>> The PCIe spec in sec 10.3.1 IMPLEMENTATION NOTE recommends to disable and
>>> block ATS before initiating a Function Level Reset. It also mentions that
>>> other reset methods could have the same vulnerability as well.
>>>
>>> Provide a callback from the PCI subsystem that will enclose the reset and
>>> have the iommu core temporarily change all the attached domain to BLOCKED.
>>> After attaching a BLOCKED domain, IOMMU drivers should fence any incoming
>>> ATS queries, synchronously stop issuing new ATS invalidations, and wait
>>> for all ATS invalidations to complete. This can avoid any ATS invaliation
>>> timeouts.
>>
>> This approach seems effective for reset operations initiated through
>> software interface functions, but how would we handle those triggered by
>> hardware mechanisms? For example, resets caused by PCIe DPC mechanisms,
>> device firmware, or manual hot-plug operations?
> 
> That's a good point. But I am not sure what SW can do about those.
> 
> IIUIC, DPC resets PCI at the HW level, SW only gets a notification
> after the HW reset finishes. So, during this HW reset, iommu might
> issue ATC invalidations (resulting in invalidation timeout noises)
> since at the SW level the device is still actively attached to an
> IOMMU instance. Right? 

Yup, the situation is this: When the system receives notification of a 
DPC event, the reset action triggered by the DPC has already occurred. 
At the very least, the software has an opportunity to be notified that a 
reset happened – though this notification inevitably lags behind the 
actual reset behavior, creating a time window between the reset action 
and its notification.​​

​For DPC specifically, there is no notification mechanism before the 
reset behavior takes place. Surprise Hot-plug events likely operate 
under a similar constraint.​​ (while we do have good opportunity to know
a hot-plug action is about to happen after attention button was pressed
for standard hot-plug hardware, adding code there is okay for now).

​This becomes particularly thorny if an Address Translation Cache (ATC) 
Invalidation request occurs within this time window. Asynchronously 
cancelling such requests later would likely be problematic. Is this an 
accurate assessment ?

At least, we can do some attempt in DPC and Hot-plug driver, and then
push the hardware specification update to provide pre-reset notification 
for DPC & hotplug. does it make sense ?

Thanks,
Ethan

> 
> Nicolin

Powered by blists - more mailing lists