linux-kernel - Re: [QUESTION] problems report: rcu_read_unlock_special() called in irq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <c448118b-9f7e-4c29-d6b3-a66e70f7163f@huawei.com>
Date: Wed, 4 Jun 2025 11:20:34 +0800
From: Xiongfeng Wang <wangxiongfeng2@...wei.com>
To: Joel Fernandes <joelagnelf@...dia.com>
CC: Joel Fernandes <joel@...lfernandes.org>, <ankur.a.arora@...cle.com>,
	Frederic Weisbecker <frederic@...nel.org>, "Paul E . McKenney"
	<paulmck@...nel.org>, Boqun Feng <boqun.feng@...il.com>,
	<neeraj.upadhyay@...nel.org>, <urezki@...il.com>, <rcu@...r.kernel.org>,
	<linux-kernel@...r.kernel.org>, <xiqi2@...wei.com>, "Wangshaobo (bobo)"
	<bobo.shaobowang@...wei.com>, Xie XiuQi <xiexiuqi@...wei.com>
Subject: Re: [QUESTION] problems report: rcu_read_unlock_special() called in
 irq_exit() causes dead loop

Hi Joel,

On 2025/6/4 3:22, Joel Fernandes wrote:
> 
> 
> On 6/3/2025 3:03 PM, Joel Fernandes wrote:
>>
>>
>> On 6/3/2025 2:59 PM, Joel Fernandes wrote:
>>> On Fri, May 30, 2025 at 09:55:45AM +0800, Xiongfeng Wang wrote:
>>>> Hi Joel,
>>>>
>>>> On 2025/5/29 0:30, Joel Fernandes wrote:
>>>>> On Wed, May 21, 2025 at 5:43 AM Xiongfeng Wang
>>>>> <wangxiongfeng2@...wei.com> wrote:
>>>>>>
>>>>>> Hi RCU experts,
>>>>>>
>>>>>> When I ran syskaller in Linux 6.6 with CONFIG_PREEMPT_RCU enabled, I got
>>>>>> the following soft lockup. The Calltrace is too long. I put it in the end.
>>>>>> The issue can also be reproduced in the latest kernel.
>>>>>>
>>>>>> The issue is as follows. CPU3 is waiting for a spin_lock, which is got by CPU1.
>>>>>> But CPU1 stuck in the following dead loop.
>>>>>>
>>>>>> irq_exit()
>>>>>>   __irq_exit_rcu()
>>>>>>     /* in_hardirq() returns false after this */
>>>>>>     preempt_count_sub(HARDIRQ_OFFSET)
>>>>>>     tick_irq_exit()
>>>>>>       tick_nohz_irq_exit()
>>>>>>             tick_nohz_stop_sched_tick()
>>>>>>               trace_tick_stop()  /* a bpf prog is hooked on this trace point */
>>>>>>                    __bpf_trace_tick_stop()
>>>>>>                       bpf_trace_run2()
>>>>>>                             rcu_read_unlock_special()
>>>>>>                               /* will send a IPI to itself */
>>>>>>                               irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
>>>>>>
>>>>>> /* after interrupt is enabled again, the irq_work is called */
>>>>>> asm_sysvec_irq_work()
>>>>>>   sysvec_irq_work()
>>>>>> irq_exit() /* after handled the irq_work, we again enter into irq_exit() */
>>>>>>   __irq_exit_rcu()
>>>>>>     ...skip...
>>>>>>            /* we queue a irq_work again, and enter a dead loop */
>>>>>>            irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
>>>>>
>>>>> This seems legitimate, Boqun and I were just talking about it. He may
>>>>> share more thoughts but here are a few:
>>>>>
>>>>> Maybe we can delay subsequent clearing of the flag in
>>>>> rcu_preempt_deferred_qs_handler() using a timer and an exponential
>>>>> back-off? That way we are not sending too many self-IPIs.
>>>>>
>>>>> And reset the process at the end of a grace period.
>>>>>
>>>>> Or just don't send subsequent self-IPIs if we just sent one for the
>>>>> rdp. Chances are, if we did not get the scheduler's attention during
>>>>> the first one, we may not in subsequent ones I think. Plus we do send
>>>>> other IPIs already if the grace period was over extended (from the FQS
>>>>> loop), maybe we can tweak that?
>>>>
>>>> Thanks a lot for your reply. I think it's hard for me to fix this issue as
>>>> above without introducing new bugs. I barely understand the RCU code. But I'm
>>>> very glad to help test if you have any code modifiction need to. I have
>>>> the VM and the syskaller benchmark which can reproduce the problem.
>>>
>>> Sure, I understand. This is already incredibly valuable so thank you again.
>>> Will request for your testing help soon. I also have a test module now which
>>> can sort-off reproduce this. Keep you posted!
>>
>> Oh sorry I meant to ask - could you provide the full kernel log and also is
>> there a standalone reproducer syzcaller binary one can run to reproduce it in a VM?

Sorry, I communicate with the teams who maintain the syzkaller tools. He said
I can't send the syskaller binary out of the company. Sorry, but I can help to
reproduce. It's not complicate and not time consuming.

I found the origin log which use kernel v6.6. But it's not complete.
Then I reprouce the problem using the latest kernel.
Both logs are attached as attachments.

>>
> 
> Sorry for the noise, but please provide the full .config as well. I am curious
> if you have CONFIG_RCU_STRICT_GRACE_PERIOD. Since that has an effect on
> rcu_read_unlock_special().

.config is also attached. CONFIG_RCU_STRICT_GRACE_PERIOD is not set.

Thanks,
Xiongfeng

> 
> Thanks!
> 
>  - Joel
> 
> .
> 

View attachment "kernel-v6.15-log.txt" of type "text/plain" (137703 bytes)

View attachment "kernel-v6.6-log.txt" of type "text/plain" (63287 bytes)

View attachment "mainline_config" of type "text/plain" (229335 bytes)