| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aIEuZy6fUj_4wtQ6@861G6M3>
Date: Wed, 23 Jul 2025 13:48:07 -0500
From: Chris Arges <carges@...udflare.com>
To: Dragos Tatulea <dtatulea@...dia.com>
Cc: netdev@...r.kernel.org, bpf@...r.kernel.org,
kernel-team <kernel-team@...udflare.com>,
Jesper Dangaard Brouer <hawk@...nel.org>, tariqt@...dia.com,
saeedm@...dia.com, Leon Romanovsky <leon@...nel.org>,
Andrew Lunn <andrew+netdev@...n.ch>,
"David S. Miller" <davem@...emloft.net>,
Eric Dumazet <edumazet@...gle.com>,
Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
Alexei Starovoitov <ast@...nel.org>,
Daniel Borkmann <daniel@...earbox.net>,
John Fastabend <john.fastabend@...il.com>,
Simon Horman <horms@...nel.org>,
Andrew Rzeznik <arzeznik@...udflare.com>,
Yan Zhai <yan@...udflare.com>
Subject: Re: [BUG] mlx5_core memory management issue
On 2025-07-04 12:37:36, Dragos Tatulea wrote:
> On Thu, Jul 03, 2025 at 10:49:20AM -0500, Chris Arges wrote:
> > When running iperf through a set of XDP programs we were able to crash
> > machines with NICs using the mlx5_core driver. We were able to confirm
> > that other NICs/drivers did not exhibit the same problem, and suspect
> > this could be a memory management issue in the driver code.
> > Specifically we found a WARNING at include/net/page_pool/helpers.h:277
> > mlx5e_page_release_fragmented.isra. We are able to demonstrate this
> > issue in production using hardware, but cannot easily bisect because
> > we don’t have a simple reproducer.
> >
> Thanks for the report! We will investigate.
>
> > I wanted to share stack traces in
> > order to help us further debug and understand if anyone else has run
> > into this issue. We are currently working on getting more crashdumps
> > and doing further analysis.
> >
> >
> > The test setup looks like the following:
> > ┌─────┐
> > │mlx5 │
> > │NIC │
> > └──┬──┘
> > │xdp ebpf program (does encap and XDP_TX)
> > │
> > ▼
> > ┌──────────────────────┐
> > │xdp.frags │
> > │ │
> > └──┬───────────────────┘
> > │tailcall
> > │BPF_REDIRECT_MAP (using CPUMAP bpf type)
> > ▼
> > ┌──────────────────────┐
> > │xdp.frags/cpumap │
> > │ │
> > └──┬───────────────────┘
> > │BPF_REDIRECT to veth (*potential trigger for issue)
> > │
> > ▼
> > ┌──────┐
> > │veth │
> > │ │
> > └──┬───┘
> > │
> > │
> > ▼
> >
> > Here an mlx5 NIC has an xdp.frags program attached which tailcalls via
> > BPF_REDIRECT_MAP into an xdp.frags/cpumap. For our reproducer we can
> > choose a random valid CPU to reproduce the issue. Once that packet
> > reaches the xdp.frags/cpumap program we then do another BPF_REDIRECT
> > to a veth device which has an XDP program which redirects to an
> > XSKMAP. It wasn’t until we added the additional BPF_REDIRECT to the
> > veth device that we noticed this issue.
> >
> Would it be possible to try to use a single program that redirects to
> the XSKMAP and check that the issue reproduces?
>
> > When running with 6.12.30 to 6.12.32 kernels we are able to see the
> > following KASAN use-after-free WARNINGs followed by a page fault which
> > crashes the machine. We have not been able to test earlier or later
> > kernels. I’ve tried to map symbols to lines of code for clarity.
> >
> Thanks for the KASAN reports, they are very useful. Keep us posted
> if you have other updates. A first quick look didn't reveal anything
> obvious from our side but we will keep looking.
>
> Thanks,
> Dragos
Ok, we can reproduce this problem!
I tried to simplify this reproducer, but it seems like what's needed is:
- xdp program attached to mlx5 NIC
- cpumap redirect
- device redirect (map or just bpf_redirect)
- frame gets turned into an skb
Then from another machine send many flows of UDP traffic to trigger the problem.
I've put together a program that reproduces the issue here:
- https://github.com/arges/xdp-redirector
In general the failure manifests with many different WARNs such as:
include/net/page_pool/helpers.h:277 mlx5e_page_release_fragmented.isra.0+0xf7/0x150 [mlx5_core]
Then the machine crashes.
I was able to get a crashdump which shows:
```
PID: 0 TASK: ffff8c0910134380 CPU: 76 COMMAND: "swapper/76"
#0 [fffffe10906d3ea8] crash_nmi_callback at ffffffffadc5c4fd
#1 [fffffe10906d3eb0] default_do_nmi at ffffffffae9524f0
#2 [fffffe10906d3ed0] exc_nmi at ffffffffae952733
#3 [fffffe10906d3ef0] end_repeat_nmi at ffffffffaea01bfd
[exception RIP: io_serial_in+25]
RIP: ffffffffae4cd489 RSP: ffffb3c60d6049e8 RFLAGS: 00000002
RAX: ffffffffae4cd400 RBX: 00000000000025d8 RCX: 0000000000000000
RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffffb10a9cb0
RBP: 0000000000000000 R8: 2d2d2d2d2d2d2d2d R9: 656820747563205b
R10: 000000002d2d2d2d R11: 000000002d2d2d2d R12: ffffffffb0fa5610
R13: 0000000000000000 R14: 0000000000000000 R15: ffffffffb10a9cb0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#4 [ffffb3c60d6049e8] io_serial_in at ffffffffae4cd489
#5 [ffffb3c60d6049e8] serial8250_console_write at ffffffffae4d2fcf
#6 [ffffb3c60d604a80] console_flush_all at ffffffffadd1cf26
#7 [ffffb3c60d604b00] console_unlock at ffffffffadd1d1df
#8 [ffffb3c60d604b48] vprintk_emit at ffffffffadd1dda1
#9 [ffffb3c60d604b98] _printk at ffffffffae90250c
#10 [ffffb3c60d604bf8] report_bug.cold at ffffffffae95001d
#11 [ffffb3c60d604c38] handle_bug at ffffffffae950e91
#12 [ffffb3c60d604c58] exc_invalid_op at ffffffffae9512b7
#13 [ffffb3c60d604c70] asm_exc_invalid_op at ffffffffaea0123a
[exception RIP: mlx5e_page_release_fragmented+85]
RIP: ffffffffc25f75c5 RSP: ffffb3c60d604d20 RFLAGS: 00010293
RAX: 000000000000003f RBX: ffff8bfa8f059fd0 RCX: ffffe3bf1992a180
RDX: 000000000000003d RSI: ffffe3bf1992a180 RDI: ffff8bf9b0784000
RBP: 0000000000000040 R8: 00000000000001d2 R9: 0000000000000006
R10: ffff8c06de22f380 R11: ffff8bfcfe6cd680 R12: 00000000000001d2
R13: 000000000000002b R14: ffff8bf9b0784000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#14 [ffffb3c60d604d20] mlx5e_free_rx_wqes at ffffffffc25f7e2f [mlx5_core]
#15 [ffffb3c60d604d58] mlx5e_post_rx_wqes at ffffffffc25f877c [mlx5_core]
#16 [ffffb3c60d604dc0] mlx5e_napi_poll at ffffffffc25fdd27 [mlx5_core]
#17 [ffffb3c60d604e20] __napi_poll at ffffffffae6a8ddb
#18 [ffffb3c60d604e90] __napi_poll at ffffffffae6a8db5
#19 [ffffb3c60d604e98] net_rx_action at ffffffffae6a95f1
#20 [ffffb3c60d604f98] handle_softirqs at ffffffffadc9d4bf
#21 [ffffb3c60d604fe8] irq_exit_rcu at ffffffffadc9e057
#22 [ffffb3c60d604ff0] common_interrupt at ffffffffae952015
--- <IRQ stack> ---
#23 [ffffb3c60c837de8] asm_common_interrupt at ffffffffaea01466
[exception RIP: cpuidle_enter_state+184]
RIP: ffffffffae955c38 RSP: ffffb3c60c837e98 RFLAGS: 00000202
RAX: ffff8c0cffc00000 RBX: ffff8c0911002400 RCX: 0000000000000000
RDX: 00003c630b2d073a RSI: ffffffe519600d10 RDI: 0000000000000000
RBP: 0000000000000001 R8: 0000000000000002 R9: 0000000000000001
R10: ffff8c0cffc330c4 R11: 071c71c71c71c71c R12: ffffffffb05ff820
R13: 00003c630b2d073a R14: 0000000000000001 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#24 [ffffb3c60c837ed0] cpuidle_enter at ffffffffae64b4ad
#25 [ffffb3c60c837ef0] do_idle at ffffffffadcfa7c6
#26 [ffffb3c60c837f30] cpu_startup_entry at ffffffffadcfaa09
#27 [ffffb3c60c837f40] start_secondary at ffffffffadc5ec77
#28 [ffffb3c60c837f50] common_startup_64 at ffffffffadc24d5d
```
Assuming (this is x86_64):
RDI=ffff8bf9b0784000 (rq)
RSI=ffffe3bf1992a180 (frag_page)
```
static void mlx5e_page_release_fragmented(struct mlx5e_rq *rq,
struct mlx5e_frag_page *frag_page)
{
u16 drain_count = MLX5E_PAGECNT_BIAS_MAX - frag_page->frags;
struct page *page = frag_page->page;
if (page_pool_unref_page(page, drain_count) == 0)
page_pool_put_unrefed_page(rq->page_pool, page, -1, true);
}
```
crash> struct mlx5e_frag_page ffffe3bf1992a180
struct mlx5e_frag_page {
page = 0x26ffff800000000,
frags = 49856
}
This means that drain_count could be an unexpected number (assuming that we
expect it to be less than MLX5E_PAGECNT_BIAS_MAX).
Let me know what additional experiments would be useful here.
--chris
Powered by blists - more mailing lists