linux-kernel - Re: [PATCH net] ipv4: route: Prevent rt_bind

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <aRC3zxmpOXXDqSX2@shredder>
Date: Sun, 9 Nov 2025 17:48:31 +0200
From: Ido Schimmel <idosch@...sch.org>
To: chuang <nashuiliang@...il.com>
Cc: Eric Dumazet <edumazet@...gle.com>, stable@...r.kernel.org,
	"David S. Miller" <davem@...emloft.net>,
	David Ahern <dsahern@...nel.org>, Jakub Kicinski <kuba@...nel.org>,
	Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>,
	Networking <netdev@...r.kernel.org>,
	open list <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH net] ipv4: route: Prevent rt_bind_exception() from
 rebinding stale fnhe

On Fri, Nov 07, 2025 at 05:53:51PM +0800, chuang wrote:
> Thanks for reviewing the patch. I'm providing the detailed analysis
> and debugging traces below to confirm the root cause and exact
> location of the reference count leak.
> 
> 1. Environment and Symptom
> 
> The issue was consistently reproduced when routing TCP traffic through
> a Software IP Tunnel interface (sit0). The traffic flow  is:
> 
>   APP -> sit0 (IP tunnel) -> outside
> 
> This leads to a reference count leak that prevents the device from
> being freed during unregistration, resulting in the kernel log
> warning:
> 
>   unregister_netdevice: waiting for sit0 to become free. Usage count = N
> 
> 2. Enable refcnt_tracer
> 
> Live-crash analysis identified a stale dst entry retaining a reference
> to sit0. With CONFIG_NET_DEV_REFCNT_TRACKER enabled, the allocation
> stack for the leaked reference was identified:
> 
> [1279559.416854] leaked reference.
> [1279559.416955]  dst_init+0x48/0x100
> [1279559.416965]  dst_alloc+0x66/0xd0
> [1279559.416966]  rt_dst_alloc+0x3c/0xd0
> [1279559.416974]  ip_route_output_key_hash_rcu+0x1d7/0x940
> [1279559.416978]  ip_route_output_key_hash+0x6d/0xa0
> [1279559.416979]  ip_route_output_flow+0x1f/0x70
> [1279559.416980]  __ip_queue_xmit+0x415/0x480
> [1279559.416984]  ip_queue_xmit+0x15/0x20
> [1279559.416986]  __tcp_transmit_skb+0xad4/0xc50
> 
> 3. Pinpointing the Unmatched dst_hold()
> 
> To pinpoint the specific reference not released, we added tracepoints
> to all dst_hold/put functions and used eBPF to record the full
> lifecycle. The tracing identified a hold operation with the following
> call stack:
> 
> do_trace_dst_entry_inc+0x45
> rt_set_nexthop.constprop.0+0x376      /* <<<<<<<<<<<<<<<<<<<< HERE */
> __mkroute_output+0x2B7
> ip_route_output_key_hash_rcu+0xBD
> ip_route_output_key_hash+0x6D
> ip_route_output_flow+0x1F
> inet_sk_rebuild_header+0x19C
> __tcp_retransmit_skb+0x7E
> tcp_retransmit_skb+0x19
> tcp_retransmit_timer+0x3DF
> 
> The address rt_set_nexthop.constprop.0+0x376 corresponds to the
> dst_hold() call inside rt_bind_exception().
> 
> 4. Root Cause Analysis
> 
> The sit driver's packet transmission path calls: sit_tunnel_xmit() ->
> ... -> update_or_create_fnhe(), which lead to fnhe_remove_oldest()
> being called to delete entries exceeding the
> FNHE_RECLAIM_DEPTH+random.
> 
> The race window is between fnhe_remove_oldest() selecting fnheX for
> deletion and the subsequent kfree_rcu(). During this time, the
> concurrent path's __mkroute_output() -> find_exception() can fetch the
> soon-to-be-deleted fnheX, and rt_bind_exception() then binds it with a
> new dst using a dst_hold(). When the original fnheX is freed via RCU,
> the dst reference remains permanently leaked.
> 
> 5. Fix Validation with eBPF
> 
> The patch mitigates this by zeroing fnhe_daddr before the
> RCU-protected deletion steps. This prevents rt_bind_exception() from
> attempting to reuse the entry.
> The fix was validated by probing the rt_bind_exception path (which in
> my environment is optimized to rt_set_nexthop.constprop.0) to catch
> any zeroed but active FNHEs being processed:
> 
> bpftrace -e 'kprobe:rt_set_nexthop.constprop.0
> {
>     $rt = (struct rtable *)arg0;
>     $fnhe = (struct fib_nh_exception *)arg3;
>     $fi = (struct flowi *)arg4;
> 
>     /* Check for an FNHE that is marked for deletion (daddr == 0)
>      * but is still visible/valid (fnhe_expires != 0 and not expired).
>      */
>     if ($fi != 0 && $fnhe != 0 && $fnhe->fnhe_daddr == 0 &&
> $fnhe->fnhe_expires != 0 && $fnhe->fnhe_expires >= jiffies) {
>         printf("rt: %llx, dev: %s, will leak before this patch\n",
> $rt, $rt->dst.dev->name);
>     }
> }'

Thanks for the details. I was able to reproduce the issue with [1] and I
can confirm that it does not reproduce with your fix.

Are you going to submit v2?

[1]
#!/bin/bash

ip netns add ns1
ip -n ns1 link set dev lo up
ip -n ns1 address add 192.0.2.1/32 dev lo
ip -n ns1 link add name dummy1 up type dummy
ip -n ns1 route add 192.0.2.2/32 dev dummy1
ip -n ns1 link add name gretap1 up arp off type gretap local 192.0.2.1 remote 192.0.2.2
ip -n ns1 route add 198.51.0.0/16 dev gretap1
taskset -c 0 ip netns exec ns1 mausezahn gretap1 -A 198.51.100.1 -B 198.51.0.0/16 -t udp -p 1000 -c 0 -q &
taskset -c 2 ip netns exec ns1 mausezahn gretap1 -A 198.51.100.1 -B 198.51.0.0/16 -t udp -p 1000 -c 0 -q &
sleep 10
ip netns pids ns1 | xargs kill
ip netns del ns1