netdev - Re: new splat

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d24ea1cc-4d32-44f9-9051-0c874f73f1c5@uliege.be>
Date: Tue, 1 Apr 2025 10:44:23 +0200
From: Justin Iurman <justin.iurman@...ege.be>
To: Paolo Abeni <pabeni@...hat.com>,
 Alexei Starovoitov <alexei.starovoitov@...il.com>, bpf
 <bpf@...r.kernel.org>, Network Development <netdev@...r.kernel.org>,
 Jakub Kicinski <kuba@...nel.org>
Subject: Re: new splat

On 3/31/25 10:07, Paolo Abeni wrote:
> Adding Justin.
> 
> On 3/31/25 1:28 AM, Alexei Starovoitov wrote:
>> After bpf fast forward we see this new failure:
>>
>> [  138.359852] BUG: using __this_cpu_read() in preemptible [00000000]
>> code: test_progs/9368
>> [  138.362686] caller is lwtunnel_xmit+0x1c/0x2e0
>> [  138.364363] CPU: 9 UID: 0 PID: 9368 Comm: test_progs Tainted: G
>>        O        6.14.0-10767-g8be3a12f9f26 #1092 PREEMPT
>> [  138.364366] Tainted: [O]=OOT_MODULE
>> [  138.364366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
>> BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
>> [  138.364368] Call Trace:
>> [  138.364370]  <TASK>
>> [  138.364375]  dump_stack_lvl+0x80/0x90
>> [  138.364381]  check_preemption_disabled+0xc6/0xe0
>> [  138.364385]  lwtunnel_xmit+0x1c/0x2e0
>> [  138.364387]  ip_finish_output2+0x2f9/0x850
>> [  138.364391]  ? __ip_finish_output+0xa0/0x320
>> [  138.364394]  ip_send_skb+0x3f/0x90
>> [  138.364397]  udp_send_skb+0x1a6/0x3d0
>> [  138.364402]  udp_sendmsg+0x87b/0x1000
>> [  138.364404]  ? ip_frag_init+0x60/0x60
>> [  138.364406]  ? reacquire_held_locks+0xcd/0x1f0
>> [  138.364414]  ? copy_process+0x2ae0/0x2fa0
>> [  138.364418]  ? inet_autobind+0x41/0x60
>> [  138.364420]  ? __local_bh_enable_ip+0x79/0xe0
>> [  138.364422]  ? inet_autobind+0x41/0x60
>> [  138.364424]  ? inet_send_prepare+0xe7/0x1e0
>> [  138.364428]  __sock_sendmsg+0x38/0x70
>> [  138.364432]  ____sys_sendmsg+0x1c9/0x200
>> [  138.364437]  ___sys_sendmsg+0x73/0xa0
>> [  138.364444]  ? __fget_files+0xb9/0x180
>> [  138.364447]  ? lock_release+0x131/0x280
>> [  138.364450]  ? __fget_files+0xc3/0x180
>> [  138.364453]  __sys_sendmsg+0x5a/0xa0
> 
> Possibly a decoded stack trace could help.
> 
> I think a possible suspect is:
> 
> commit 986ffb3a57c5650fb8bf6d59a8f0f07046abfeb6
> Author: Justin Iurman <justin.iurman@...ege.be>
> Date:   Fri Mar 14 13:00:46 2025 +0100
> 
>      net: lwtunnel: fix recursion loops
> 
> with dev_xmit_recursion() in lwtunnel_xmit() being called in preemptible
> scope.

Correct, I came to the same conclusion based on that trace. However, I 
can't reproduce it with a PREEMPT kernel. It goes through without 
problem and the output is (as expected), i.e., "lwtunnel_xmit(): 
recursion limit reached on datapath".

> @Justin, could you please have a look?

I guess that using preempt_{disable|enable}() would not be enough here, 
so we may s/rcu_read_{lock|unlock}()/rcu_read_{lock|unlock}_bh()/g and 
move the call to rcu_read_lock_bh() before dev_xmit_recursion(). Thoughts?

> Thanks,
> 
> Paolo
>