netdev - Re: [BUG] mlx5_core memory management issue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aJTYNG1AroAnvV31@861G6M3>
Date: Thu, 7 Aug 2025 11:45:40 -0500
From: Chris Arges <carges@...udflare.com>
To: Dragos Tatulea <dtatulea@...dia.com>
Cc: netdev@...r.kernel.org, bpf@...r.kernel.org,
	kernel-team <kernel-team@...udflare.com>,
	Jesper Dangaard Brouer <hawk@...nel.org>, tariqt@...dia.com,
	saeedm@...dia.com, Leon Romanovsky <leon@...nel.org>,
	Andrew Lunn <andrew+netdev@...n.ch>,
	"David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
	Alexei Starovoitov <ast@...nel.org>,
	Daniel Borkmann <daniel@...earbox.net>,
	John Fastabend <john.fastabend@...il.com>,
	Simon Horman <horms@...nel.org>,
	Andrew Rzeznik <arzeznik@...udflare.com>,
	Yan Zhai <yan@...udflare.com>
Subject: Re: [BUG] mlx5_core memory management issue

On 2025-07-24 17:01:16, Dragos Tatulea wrote:
> On Wed, Jul 23, 2025 at 01:48:07PM -0500, Chris Arges wrote:
> > 
> > Ok, we can reproduce this problem!
> > 
> > I tried to simplify this reproducer, but it seems like what's needed is:
> > - xdp program attached to mlx5 NIC
> > - cpumap redirect
> > - device redirect (map or just bpf_redirect)
> > - frame gets turned into an skb
> > Then from another machine send many flows of UDP traffic to trigger the problem.
> > 
> > I've put together a program that reproduces the issue here:
> > - https://github.com/arges/xdp-redirector
> >
> Much appreciated! I fumbled around initially, not managing to get
> traffic to the xdp_devmap stage. But further debugging revealed that GRO
> needs to be enabled on the veth devices for XDP redir to work to the
> xdp_devmap. After that I managed to reproduce your issue.
> 
> Now I can start looking into it.
> 

Dragos,

There was a similar reference counting issue identified in:
https://lore.kernel.org/all/20250801170754.2439577-1-kuba@kernel.org/

Part of the commit message mentioned:
> Unfortunately for fbnic since commit f7dc3248dcfb ("skbuff: Optimization
> of SKB coalescing for page pool") core _may_ actually take two extra
> pp refcounts, if one of them is returned before driver gives up the bias
> the ret < 0 check in page_pool_unref_netmem() will trigger.

In order to help debug the mlx5 issue caused by xdp redirection, I built a
kernel with commit f7dc3248dcfb reverted, but unfortunately I was still able
to reproduce the issue.

I am happy to try some other experiments, or if there are other ideas you have.

Thanks,
--chris