linux-kernel - Re: [PATCH net] net/mlx5e: Skip NAPI polling when PCI channel is offline

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aYyHNGBPu0dEIEzS@gmail.com>
Date: Wed, 11 Feb 2026 05:44:48 -0800
From: Breno Leitao <leitao@...ian.org>
To: Tariq Toukan <ttoukan.linux@...il.com>
Cc: Saeed Mahameed <saeedm@...dia.com>, Tariq Toukan <tariqt@...dia.com>, 
	Mark Bloch <mbloch@...dia.com>, Leon Romanovsky <leon@...nel.org>, 
	Andrew Lunn <andrew+netdev@...n.ch>, "David S. Miller" <davem@...emloft.net>, 
	Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>, 
	Paolo Abeni <pabeni@...hat.com>, Amir Vadai <amirv@...lanox.com>, netdev@...r.kernel.org, 
	linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org, dcostantino@...a.com, 
	rneu@...a.com, kernel-team@...a.com
Subject: Re: [PATCH net] net/mlx5e: Skip NAPI polling when PCI channel is
 offline

Hello Tariq,

On Wed, Feb 11, 2026 at 01:26:35PM +0200, Tariq Toukan wrote:
> On 09/02/2026 20:01, Breno Leitao wrote:
> > When a PCI error (e.g. AER error or DPC containment) marks the PCI
> > channel as frozen or permanently failed, the IOMMU mappings for the
> > device may already be torn down. If mlx5e_napi_poll() continues
> > processing CQEs in this state, every call to dma_unmap_page() triggers
> > a WARN_ON in iommu_dma_unmap_phys().
> > 
> > In a real-world crash scenario on an NVIDIA Grace (ARM64) platform,
> > a DPC event froze the PCI channel and the mlx5 NAPI poll continued
> > processing error CQEs, calling dma_unmap for each pending WQE. Here is
> > an example:
> > 
> > The DPC event on port 0007:00:00.0 fires and eth1 (on 0017:01:00.0) starts
> > seeing error CQEs almost immediately:
> > 
> >      pcieport 0007:00:00.0: DPC: containment event, status:0x2009
> >      mlx5_core 0017:01:00.0 eth1: Error cqe on cqn 0x54e, ci 0xb06, ...
> > 
> > The WARN_ON storm begins ~0.4s later and repeats for every pending WQE:
> > 
> >      WARNING: CPU: 32 PID: 0 at drivers/iommu/dma-iommu.c:1237 iommu_dma_unmap_phys
> >      Call trace:
> >       iommu_dma_unmap_phys+0xd4/0xe0
> >       mlx5e_tx_wi_dma_unmap+0xb4/0xf0
> >       mlx5e_poll_tx_cq+0x14c/0x438
> >       mlx5e_napi_poll+0x6c/0x5e0
> >       net_rx_action+0x160/0x5c0
> >       handle_softirqs+0xe8/0x320
> >       run_ksoftirqd+0x30/0x58
> > 
> > After 23 seconds of WARN_ON() storm, the watchdog fires:
> > 
> >      watchdog: BUG: soft lockup - CPU#32 stuck for 23s! [ksoftirqd/32:179]
> >      Kernel panic - not syncing: softlockup: hung tasks
> > 
> > Each unmap hit the WARN_ON in the IOMMU layer, printing a full stack
> > trace. With dozens of pending WQEs, this created a storm of WARN_ON
> > dumps in softirq context that monopolized the CPU for over 23 seconds,
> > triggering a soft lockup panic.
...
> You're introducing an interesting problem, but I am not convinced by this
> solution approach.
> 
> Why would the driver perform this check if it doesn't guarantee prevention
> of invalid access? It only "allows one napi cycle", which happen to be good
> enough to prevent the soft lockup in your case.
> 
> What if a napi cycle is configured with larger budget?

Very good point. In this case, we will still see some WARN_ON() in DMA, and the
patch might eventually not help much if the AER hits mid-NAPI and there is
still a long budget remaining.

> If the problem is that the WARN_ON is being called at a high rate, then it
> should be rate-limited.

That would be a solution as well, and I am happy to pursue it, if that one is
more appropriate

Thanks for reviewing it,
--breno