Message-ID: <CACT4Y+YkkgBM=VcAXe2bc0ijQrPZ4xyFOuSTELYGw1f1VHLc3w@mail.gmail.com>
Date: Tue, 7 Jan 2025 09:03:10 +0100
From: Dmitry Vyukov <dvyukov@...gle.com>
To: Kun Hu <huk23@...udan.edu.cn>
Cc: andreyknvl@...il.com, akpm@...ux-foundation.org, elver@...gle.com, 
	arnd@...db.de, nogikh@...gle.com, kasan-dev@...glegroups.com, 
	linux-kernel@...r.kernel.org
Subject: Re: Bug: Potential KCOV Race Condition in __sanitizer_cov_trace_pc
 Leading to Crash at kcov.c:217

On Thu, 2 Jan 2025 at 04:21, Kun Hu <huk23@...udan.edu.cn> wrote:
>
> Hello,
>
> When using our customed fuzzer tool to fuzz the latest Linux kernel, the following crash
> was triggered.
>
> HEAD commit: dbfac60febfa806abb2d384cb6441e77335d2799
> git tree: upstream
> Console output: https://drive.google.com/file/d/1rmVTkBzuTt0xMUS-KPzm9OafMLZVOAHU/view?usp=sharing
> Kernel config: https://drive.google.com/file/d/1m1mk_YusR-tyusNHFuRbzdj8KUzhkeHC/view?usp=sharing
> C reproducer: /
> Syzlang reproducer: /
>
> The crash in __sanitizer_cov_trace_pc at kernel/kcov.c:217 seems to be related to the handling of KCOV instrumentation when running in a preemption or IRQ-sensitive context. Specifically, the code might allow potential recursive invocations of __sanitizer_cov_trace_pc during early interrupt handling, which could lead to data races or inconsistent updates to the coverage area (kcov_area). It remains unclear whether this is a KCOV-specific issue or a rare edge case exposed by fuzzing.

Hi Kun,

How have you inferred this from the kernel oops?
I only see a stall that may have just happened to be caught inside of
__sanitizer_cov_trace_pc function since it's executed often in an
instrumented kernel.

Note: on syzbot we don't report stalls on instances that have
perf_event_open enabled, since perf have known bugs that lead to stall
all over the kernel.

> Could you please help check if this needs to be addressed?
>
> If you fix this issue, please add the following tag to the commit:
> Reported-by: Kun Hu <huk23@...udan.edu.cn>, Jiaji Qin <jjtan24@...udan.edu.cn>
>
> --------------------------------
> rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
> rcu:    0-....: (36 ticks this GP) idle=5a54/1/0x4000000000000000 softirq=28602/28602 fqs=20758
> rcu:    (detected by 2, t=105010 jiffies, g=53165, q=148274 ncpus=4)
> Sending NMI from CPU 2 to CPUs 0:
> NMI backtrace for cpu 0
> CPU: 0 UID: 0 PID: 2946 Comm: syz.1.149 Tainted: G    B              6.13.0-rc4 #1
> Tainted: [B]=BAD_PAGE
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
> RIP: 0010:__sanitizer_cov_trace_pc+0x22/0x60 kernel/kcov.c:217
> Code: 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 bf 02 00 00 00 53 48 8b 6c 24 10 65 48 8b 1d 78 11 7a 6b 48 89 de e8 40 ff ff ff 84 c0 <74> 27 48 8b 93 e0 14 00 00 8b 8b dc 14 00 00 48 8b 02 48 83 c0 01
> RSP: 0018:ffa0000000007698 EFLAGS: 00000046
> RAX: 0000000000000000 RBX: ff110000386f0000 RCX: ffffffff949eb81e
> RDX: 0000000000000000 RSI: ff110000386f0000 RDI: 0000000000000002
> RBP: ffffffff949eb8c3 R08: 0000000000000000 R09: fffffbfff4177aab
> R10: fffffbfff4177aaa R11: ffffffffa0bbd557 R12: ff11000002b9ab50
> R13: 0000000000000000 R14: 1ff4000000000edc R15: ff11000053a361a8
> FS:  00007f403e2c1700(0000) GS:ff11000053a00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000001 CR3: 0000000024264004 CR4: 0000000000771ef0
> PKRU: 80000000
> Call Trace:
>  <NMI>
>  </NMI>
>  <IRQ>
>  perf_prepare_sample+0x803/0x2580 kernel/events/core.c:7977
>  __perf_event_output kernel/events/core.c:8079 [inline]
>  perf_event_output_forward+0xd3/0x2c0 kernel/events/core.c:8100
>  __perf_event_overflow+0x1e4/0x8f0 kernel/events/core.c:9926
>  perf_swevent_overflow+0xac/0x150 kernel/events/core.c:10001
>  perf_swevent_event+0x1e9/0x2e0 kernel/events/core.c:10034
>  perf_tp_event+0x227/0xfe0 kernel/events/core.c:10535
>  perf_trace_run_bpf_submit+0xef/0x180 kernel/events/core.c:10471
>  do_perf_trace_preemptirq_template include/trace/events/preemptirq.h:14 [inline]
>  perf_trace_preemptirq_template+0x287/0x450 include/trace/events/preemptirq.h:14
>  trace_irq_enable include/trace/events/preemptirq.h:40 [inline]
>  trace_hardirqs_on+0xf2/0x160 kernel/trace/trace_preemptirq.c:73
>  irqentry_exit+0x3b/0x90 kernel/entry/common.c:357
>  asm_sysvec_irq_work+0x1a/0x20 arch/x86/include/asm/idtentry.h:738
> RIP: 0010:get_current arch/x86/include/asm/current.h:49 [inline]
> RIP: 0010:__rcu_read_unlock+0xc6/0x570 kernel/rcu/tree_plugin.h:440
> Code: b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e bf 01 00 00 8b 85 00 04 00 00 85 c0 75 57 <65> 48 8b 1d 02 af 92 6b 48 8d bb fc 03 00 00 48 b8 00 00 00 00 00
> RSP: 0018:ffa0000000007e08 EFLAGS: 00000206
> RAX: 0000000000000046 RBX: ff11000053a3d240 RCX: 1ffffffff4177c76
> RDX: 0000000000000000 RSI: 0000000000000101 RDI: ffffffff947103e2
> RBP: ffffffff9ed26380 R08: 0000000000000000 R09: 0000000000000000
> R10: fffffbfff4177aaa R11: ffffffffa0bbd557 R12: 0000000000000001
> R13: 0000000000000200 R14: ffa0000000007e00 R15: 1ff4000000000fc9
>  rcu_read_unlock include/linux/rcupdate.h:882 [inline]
>  ieee80211_rx_napi+0x117/0x410 net/mac80211/rx.c:5493
>  ieee80211_rx include/net/mac80211.h:5166 [inline]
>  ieee80211_handle_queued_frames+0xd9/0x130 net/mac80211/main.c:441
>  tasklet_action_common+0x279/0x810 kernel/softirq.c:811
>  handle_softirqs+0x1ad/0x870 kernel/softirq.c:561
>  __do_softirq kernel/softirq.c:595 [inline]
>  invoke_softirq kernel/softirq.c:435 [inline]
>  __irq_exit_rcu kernel/softirq.c:662 [inline]
>  irq_exit_rcu+0xee/0x140 kernel/softirq.c:678
>  instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1049 [inline]
>  sysvec_apic_timer_interrupt+0x94/0xb0 arch/x86/kernel/apic/apic.c:1049
>  </IRQ>
>  <TASK>
>  asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
> RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:210
> Code: 48 8b 05 b3 11 7a 6b 48 8b 80 f0 14 00 00 e9 32 a1 e6 07 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <f3> 0f 1e fa 55 bf 02 00 00 00 53 48 8b 6c 24 10 65 48 8b 1d 78 11
> RSP: 0018:ffa0000007f17db0 EFLAGS: 00000246
> RAX: 0000000000000000 RBX: 0000000000000200 RCX: ffffffff947854c4
> RDX: 0000000000000200 RSI: ff110000386f0000 RDI: 0000000000000002
> RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
> R10: fffffbfff4177aaa R11: ffffffffa0bbd557 R12: ffa0000007f17ec0
> R13: dffffc0000000000 R14: dffffc0000000000 R15: 1ff4000000fe2fd8
>  __seqprop_raw_spinlock_sequence include/linux/seqlock.h:226 [inline]
>  ktime_get_ts64+0xe4/0x3c0 kernel/time/timekeeping.c:952
>  posix_get_monotonic_timespec+0x78/0x260 kernel/time/posix-timers.c:156
>  __do_sys_clock_gettime kernel/time/posix-timers.c:1148 [inline]
>  __se_sys_clock_gettime kernel/time/posix-timers.c:1138 [inline]
>  __x64_sys_clock_gettime+0x15c/0x260 kernel/time/posix-timers.c:1138
>  do_syscall_x64 arch/x86/entry/common.c:52 [inline]
>  do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
> RIP: 0033:0x7f403f808ba5
> Code: c0 4c 89 63 08 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 83 f8 02 0f 84 e6 02 00 00 44 89 e7 48 89 de b8 e4 00 00 00 0f 05 <48> 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 81 7e 04 ff ff ff 7f
> RSP: 002b:00007f403e2c0b20 EFLAGS: 00000293 ORIG_RAX: 00000000000000e4
> RAX: ffffffffffffffda RBX: 00007f403e2c0ba0 RCX: 00007f403f808ba5
> RDX: 0000000000000002 RSI: 00007f403e2c0ba0 RDI: 0000000000000001
> RBP: 00007f403e2c0b70 R08: 00007f403f804010 R09: 0000000000032b26
> R10: 7fffffffffffffff R11: 0000000000000293 R12: 00007f403e2c0ba0
> R13: 00007f403f82ff8c R14: 00007f403f830018 R15: 00007f403e2c0d40
>  </TASK>
>
>
> ---------------
> thanks,
> Kun Hu

Powered by blists - more mailing lists