netdev - Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <60999fc9-e1d6-4bb3-aa30-f959216a1b29@redhat.com>
Date: Wed, 6 Nov 2024 10:11:13 +0100
From: Paolo Abeni <pabeni@...hat.com>
To: David Ahern <dsahern@...nel.org>, netdev@...r.kernel.org
Cc: "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Simon Horman <horms@...nel.org>, Shuah Khan <shuah@...nel.org>,
 linux-kselftest@...r.kernel.org
Subject: Re: [PATCH v2 net-next 1/2] ipv6: release nexthop on device removal

On 11/5/24 22:40, David Ahern wrote:
> On 11/5/24 11:23 AM, Paolo Abeni wrote:
>> The CI is hitting some aperiodic hangup at device removal time in the
>> pmtu.sh self-test:
>>
>> unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6
>> ref_tracker: veth_A-R1@...f888013df15d8 has 1/5 users at
>> 	dst_init+0x84/0x4a0
>> 	dst_alloc+0x97/0x150
>> 	ip6_dst_alloc+0x23/0x90
>> 	ip6_rt_pcpu_alloc+0x1e6/0x520
>> 	ip6_pol_route+0x56f/0x840
>> 	fib6_rule_lookup+0x334/0x630
>> 	ip6_route_output_flags+0x259/0x480
>> 	ip6_dst_lookup_tail.constprop.0+0x5c2/0x940
>> 	ip6_dst_lookup_flow+0x88/0x190
>> 	udp_tunnel6_dst_lookup+0x2a7/0x4c0
>> 	vxlan_xmit_one+0xbde/0x4a50 [vxlan]
>> 	vxlan_xmit+0x9ad/0xf20 [vxlan]
>> 	dev_hard_start_xmit+0x10e/0x360
>> 	__dev_queue_xmit+0xf95/0x18c0
>> 	arp_solicit+0x4a2/0xe00
>> 	neigh_probe+0xaa/0xf0
>>
>> While the first suspect is the dst_cache, explicitly tracking the dst
>> owing the last device reference via probes proved such dst is held by
>> the nexthop in the originating fib6_info.
>>
>> Similar to commit f5b51fe804ec ("ipv6: route: purge exception on
>> removal"), we need to explicitly release the originating fib info when
>> disconnecting a to-be-removed device from a live ipv6 dst: move the
>> fib6_info cleanup into ip6_dst_ifdown().
>>
>> Tested running:
>>
>> ./pmtu.sh cleanup_ipv6_exception
>>
>> in a tight loop for more than 400 iterations with no spat, running an
>> unpatched kernel  I observed a splat every ~10 iterations.
>>
>> Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info")
> 
> are you sure that is the correct Fixes? That commit is June 2019 and
> there have been stable periods since then without netdev release problems.

"Sure" is a big word ;) AFAICS the mentioned commit let fib6_info store
indirectly the extra dev reference via nexthop and does not clean it at
device removal time.

Note that the issue is not deterministic - I needed ~30 mptu.sh
iterations in a row to see it, so it could go unnoticed for a long time.

>> Signed-off-by: Paolo Abeni <pabeni@...hat.com>
>> ---
>> v1 -> v2:
>>  - dropped unintended whitespace change
>> ---
>>  net/ipv6/route.c | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
> 
> Reviewed-by: David Ahern <dsahern@...nel.org>

Thanks!

Paolo