linux-kernel - INFO: rcu detected stall in net_rx

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 10 Oct 2022 23:37:55 +0800
From:   Wei Chen <harperchen1110@...il.com>
To:     davem@...emloft.net, Eric Dumazet <edumazet@...gle.com>,
        kuba@...nel.org, pabeni@...hat.com, ast@...nel.org,
        daniel@...earbox.net, hawk@...nel.org, john.fastabend@...il.com,
        andrii@...nel.org, kafai@...com, songliubraving@...com, yhs@...com,
        kpsingh@...nel.org, bigeasy@...utronix.de, imagedong@...cent.com,
        petrm@...dia.com
Cc:     netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
        bpf@...r.kernel.org
Subject: INFO: rcu detected stall in net_rx_action

Dear Linux Developer,

Recently when using our tool to fuzz kernel, the following crash was triggered:

HEAD commit: 64570fbc14f8 Linux 5.15-rc5
git tree: upstream
compiler: clang 12.0.0
console output:
https://drive.google.com/file/d/1BOhVEmi3RPIxx-F0LMLsgflaj0r0MyKv/view?usp=sharing
kernel config: https://drive.google.com/file/d/1lNwvovjLNrcuyFGrg05IoSmgO5jaKBBJ/view?usp=sharing

Unfortunately, I don't have any reproducer for this crash yet.

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 0-...!: (88 ticks this GP) idle=4c5/1/0x4000000000000000
softirq=42739/42739 fqs=1
(t=15633 jiffies g=62957 q=125)
rcu: rcu_preempt kthread starved for 15193 jiffies! g62957 f0x0
RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now
expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task     stack:27696 pid:   14
ppid:     2 flags:0x00004000
Call Trace:
 __schedule+0xc1a/0x11e0
 schedule+0x14b/0x210
 schedule_timeout+0x1b4/0x310
 rcu_gp_fqs_loop+0x1fd/0x770
 rcu_gp_kthread+0xa5/0x340
 kthread+0x419/0x510
 ret_from_fork+0x1f/0x30
rcu: Stack dump where RCU GP kthread last ran:
NMI backtrace for cpu 0
CPU: 0 PID: 13 Comm: ksoftirqd/0 Not tainted 5.15.0-rc5+ #14
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
 <IRQ>
 dump_stack_lvl+0x1d8/0x2c4
 nmi_cpu_backtrace+0x452/0x480
 nmi_trigger_cpumask_backtrace+0x1a3/0x330
 rcu_check_gp_kthread_starvation+0x1f9/0x270
 rcu_sched_clock_irq+0x1de4/0x2bc0
 update_process_times+0x1ab/0x220
 tick_sched_timer+0x2a0/0x440
 __hrtimer_run_queues+0x51a/0xae0
 hrtimer_interrupt+0x3c9/0x1130
 __sysvec_apic_timer_interrupt+0xf9/0x280
 sysvec_apic_timer_interrupt+0x8c/0xb0
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:e1000_clean+0x15ad/0x40b0
Code: c5 c8 04 00 00 4c 89 eb 48 c1 eb 03 42 80 3c 23 00 74 08 4c 89
ef e8 a2 2c 65 fc 49 8b 45 00 b9 9d 00 00 00 89 88 d0 00 00 00 <42> 80
3c 23 00 74 08 4c 89 ef e8 84 2c 65 fc 49 8b 45 00 8b 40 08
RSP: 0018:ffffc90000707840 EFLAGS: 00000246
RAX: ffffc900065c0000 RBX: 1ffff1100371f229 RCX: 000000000000009d
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000707ac8 R08: ffffffff856f35c6 R09: ffffed100371f2a7
R10: ffffed100371f2a7 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff88801b8f9148 R14: 0000000000004e20 R15: 1ffff920000e0f2c
 __napi_poll+0xbd/0x550
 net_rx_action+0x67b/0xfc0
 __do_softirq+0x372/0x783
 run_ksoftirqd+0xa2/0x100
 smpboot_thread_fn+0x570/0xa20
 kthread+0x419/0x510
 ret_from_fork+0x1f/0x30
NMI backtrace for cpu 0
CPU: 0 PID: 13 Comm: ksoftirqd/0 Not tainted 5.15.0-rc5+ #14
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.13.0-1ubuntu1.1 04/01/2014
Call Trace:
 <IRQ>
 dump_stack_lvl+0x1d8/0x2c4
 nmi_cpu_backtrace+0x452/0x480
 nmi_trigger_cpumask_backtrace+0x1a3/0x330
 rcu_dump_cpu_stacks+0x22d/0x390
 rcu_sched_clock_irq+0x1de9/0x2bc0
 update_process_times+0x1ab/0x220
 tick_sched_timer+0x2a0/0x440
 __hrtimer_run_queues+0x51a/0xae0
 hrtimer_interrupt+0x3c9/0x1130
 __sysvec_apic_timer_interrupt+0xf9/0x280
 sysvec_apic_timer_interrupt+0x8c/0xb0
 </IRQ>
 asm_sysvec_apic_timer_interrupt+0x12/0x20
RIP: 0010:e1000_clean+0x15ad/0x40b0
Code: c5 c8 04 00 00 4c 89 eb 48 c1 eb 03 42 80 3c 23 00 74 08 4c 89
ef e8 a2 2c 65 fc 49 8b 45 00 b9 9d 00 00 00 89 88 d0 00 00 00 <42> 80
3c 23 00 74 08 4c 89 ef e8 84 2c 65 fc 49 8b 45 00 8b 40 08
RSP: 0018:ffffc90000707840 EFLAGS: 00000246
RAX: ffffc900065c0000 RBX: 1ffff1100371f229 RCX: 000000000000009d
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc90000707ac8 R08: ffffffff856f35c6 R09: ffffed100371f2a7
R10: ffffed100371f2a7 R11: 0000000000000000 R12: dffffc0000000000
R13: ffff88801b8f9148 R14: 0000000000004e20 R15: 1ffff920000e0f2c
 __napi_poll+0xbd/0x550
 net_rx_action+0x67b/0xfc0
 __do_softirq+0x372/0x783
 run_ksoftirqd+0xa2/0x100
 smpboot_thread_fn+0x570/0xa20
 kthread+0x419/0x510
 ret_from_fork+0x1f/0x30
----------------
Code disassembly (best guess), 1 bytes skipped:
   0: c8 04 00 00          enterq $0x4,$0x0
   4: 4c 89 eb              mov    %r13,%rbx
   7: 48 c1 eb 03          shr    $0x3,%rbx
   b: 42 80 3c 23 00        cmpb   $0x0,(%rbx,%r12,1)
  10: 74 08                je     0x1a
  12: 4c 89 ef              mov    %r13,%rdi
  15: e8 a2 2c 65 fc        callq  0xfc652cbc
  1a: 49 8b 45 00          mov    0x0(%r13),%rax
  1e: b9 9d 00 00 00        mov    $0x9d,%ecx
  23: 89 88 d0 00 00 00    mov    %ecx,0xd0(%rax)
* 29: 42 80 3c 23 00        cmpb   $0x0,(%rbx,%r12,1) <-- trapping instruction
  2e: 74 08                je     0x38
  30: 4c 89 ef              mov    %r13,%rdi
  33: e8 84 2c 65 fc        callq  0xfc652cbc
  38: 49 8b 45 00          mov    0x0(%r13),%rax
  3c: 8b 40 08              mov    0x8(%rax),%eax

Best,
Wei