linux-kernel - Re: [REGRESSION][BISECTED] Performance Regression in IOMMU/VT-d Since Kernel 6.10

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7d2214f3-3b54-4b74-a18b-aca1fdf4fdb4@linux.intel.com>
Date: Wed, 2 Jul 2025 13:14:12 +0800
From: Baolu Lu <baolu.lu@...ux.intel.com>
To: Ioanna Alifieraki <ioanna-maria.alifieraki@...onical.com>,
 kevin.tian@...el.com, jroedel@...e.de, robin.murphy@....com,
 will@...nel.org, joro@...tes.org, dwmw2@...radead.org,
 iommu@...ts.linux.dev, linux-kernel@...r.kernel.org,
 regressions@...ts.linux.dev, stable@...r.kernel.org
Subject: Re: [REGRESSION][BISECTED] Performance Regression in IOMMU/VT-d Since
 Kernel 6.10

On 7/2/25 01:11, Ioanna Alifieraki wrote:
> #regzbot introduced: 129dab6e1286
> 
> Hello everyone,
> 
> We've identified a performance regression that starts with linux
> kernel 6.10 and persists through 6.16(tested at commit e540341508ce).
> Bisection pointed to commit:
> 129dab6e1286 ("iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map").
> 
> The issue occurs when running fio against two NVMe devices located
> under the same PCIe bridge (dual-port NVMe configuration). Performance
> drops compared to configurations where the devices are on different
> bridges.
> 
> Observed Performance:
> - Before the commit: ~6150 MiB/s, regardless of NVMe device placement.
> - After the commit:
>    -- Same PCIe bridge: ~4985 MiB/s
>    -- Different PCIe bridges: ~6150 MiB/s
> 
> 
> Currently we can only reproduce the issue on a Z3 metal instance on
> gcp. I suspect the issue can be reproducible if you have a dual port
> nvme on any machine.
> At [1] there's a more detailed description of the issue and details
> on the reproducer.

This test was running on bare metal hardware instead of any
virtualization guest, right? If that's the case,
cache_tag_flush_range_np() is almost a no-op.

Can you please show me the capability register of the IOMMU by:

#cat /sys/bus/pci/devices/[pci_dev_name]/iommu/intel-iommu/cap

> 
> Could you please advise on the appropriate path forward to mitigate or
> address this regression?
> 
> Thanks,
> Jo
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2115738

Thanks,
baolu