Message-ID: <aD2TspKH/7yvfYoO@e129823.arm.com>
Date: Mon, 2 Jun 2025 13:06:10 +0100
From: Yeoreum Yun <yeoreum.yun@....com>
To: 00107082@....com
Cc: peterz@...radead.org, mingo@...hat.com, acme@...nel.org,
	namhyung@...nel.org, mingo@...nel.org, yeoreum.yun@....com,
	leo.yan@....com, linux-perf-users@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: Re: [BUG][6.15][perf] Kernel panic not syncing: Fatal exception in
 interrupt

Sorry to make noise all.
I've forgotten to cc mailing list.
If you receive duplicate mail, Sorry again...

====================================================
Hi David,

> Hi,
>
> Caught a kernel panic when rebooting, system stuck until pressing power button.
> I have only a screenshot when it happens, following logs were extracted from a
> captured picture.
>
> 863.881960] sysved_call_function_sing le+0x4c/0xc0
> 863.881301] asm_sysvec_call_function_single+0x16/0x20
> 869.881344] RIP: 0633:0x7f9alcea3367
> 663.681373] Code: 00 66 99 b8 ff ff ff ff c3 66 ....
> 863.881524] RSP: 002b:00007fffa526fcf8 EFLAGS: 00000246
> 869.881567] RAX: 0000562060c962d0 RBX: 0000000000000002 RCX: 00007f9a1cff1c60
> 863.881625] RDX: 00007f9a0c000030 RSI: 00007f9alcff1c60 RDI: 00007f9a1ca91c20
> 863.081682] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f9a1d6217a0
> 869.881740] R10: 00007f9alca91c10 R11: 0000000000000246 R12: 00007f9a1d70c020
> 869.881798] R13: 00007fffa5270030 R14: 00007fffa526fd00 R15: 0000000000000000
> 863.881860] </TASK>
> 863.881876) Modules linked in: snd_seq_dummy (E) snd_hrtimer (E)...
> ...
> 863.887142] button (E)
> 863.912127] CR2: ffffe4afcc079650
> 863.914593] --- [ end trace 0000000000000000 1--
> 864.042750] RIP: 0010:ctx_sched_out+0x1ce/0x210
> 864.045214] Code: 89 c6 4c 8b b9 de 00 00 00 48 ...
> 864.050343] RSP: 0000:ffffaa4ec0f3fe60 EFLAGS: 00010086
> 864.052929] RAX: 0000000000000002 RBX: ffff8e8eeed2a580 RCX: ffff8e8bded9bf00
> 864.055518] RDX: 000000c92340b051 RSI: 000000c92340b051 RDI: ffff
> 864.058093] RBP: 0000000000000000 R08: 0000000000000002 R09: 00
> 864.060654] R10: 0000000000000000 R11: 0000000000000000 R12: 000
> 864.063183] R13: ffff8e8eeed2a580 R14: 0000000000000007 R15: ffffe4afcc079650
> 864.065729] FS: 00007f9a1ca91940 (0000) GS:ffff8e8f6b1c3000(0000) knIGS:0000000000000000
> 864.068312] CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033
> 864.070898] CR2: ffffe4afcc079650 CR3: 00000001136d8000 CR4: 0000000000350ef0
> 864.673523] Kernel panic - not syncing: Fatal exception in interrupt
> 864.076410] Kernel Offset: 0xc00000 from 0xffffffff81000000 (relocation range: 0xff
> 864.205401] --- [ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
> This happens ever since 6.15-rc1, from time to time, I would get kernel panic when
> reboot; it is only recently that I figured out a precedure reproducing
> this with *high* probability:
>
> 1. create a cgroup.
> 2. perf_event_open(PERF_FLAG_FD_CLOEXEC|PERF_FLAG_PID_CGROUP) for each cpu with following attrs:
> 	attr.type = PERF_TYPE_SOFTWARE;
> 	attr.size = sizeof(attr);
> 	attr.config = PERF_COUNT_SW_CPU_CLOCK;
> 	attr.sample_freq = 9999;
> 	attr.freq = 1;
> 	attr.wakeup_events = 16;
> 	attr.sample_type = PERF_SAMPLE_CALLCHAIN;
> 	attr.sample_max_stack = 32;
> 	attr.exclude_callchain_user = 1;
> 3. close all perf_event_open after several minutes
> 4. reboot
>
> And after an exhausting bisect on events/core.c, (I need 5 rounds to conclude a good bisect)
> I think I reach the conclusion, with very high probability, that this is caused by
>
> commit a3c3c66670cee11eb13aa43905904bf29cb92d32
> Author: Yeoreum Yun <yeoreum.yun@....com>
> Date:   Wed Mar 26 08:20:03 2025 +0000
>
>    perf/core: Fix child_total_time_enabled accounting bug at task exit
>
> Reverting this can fix it: I run the test 10 rounds, no kernel panic observed.
>
> The changes made to __perf_remove_from_context by commit a3c3c6667("perf/core:
> Fix child_total_time_enabled accounting bug at task exit") has wider effect
> than the callchain mentioned in commit message, and I think an esay fix would
> be just restricting the effect to that callchain only, and restore other changes back.
>
> I have test the patch below several rounds, and so far so good, and I will have
> more tests on it.
>
> Signed-off-by: David Wang <00107082@....com>

Thanks for your reporting and Sorry for my bad.
By my change, the tracking nr_cgorups is broken which could make a dangling
pointer for cpuctx->cgrp.

Could you test with below change please?

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 95e703891b24..d0a9096735b9 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2116,18 +2116,6 @@ list_del_event(struct perf_event *event, struct perf_event_context *ctx)
        if (event->group_leader == event)
                del_event_from_groups(event, ctx);

-       /*
-        * If event was in error state, then keep it
-        * that way, otherwise bogus counts will be
-        * returned on read(). The only way to get out
-        * of error state is by explicit re-enabling
-        * of the event
-        */
-       if (event->state > PERF_EVENT_STATE_OFF) {
-               perf_cgroup_event_disable(event, ctx);
-               perf_event_set_state(event, PERF_EVENT_STATE_OFF);
-       }
-
        ctx->generation++;
        event->pmu_ctx->nr_events--;
 }
@@ -2471,6 +2459,16 @@ __perf_remove_from_context(struct perf_event *event,

        ctx_time_update(cpuctx, ctx);

+       /*
+        * If event was in error state, then keep it
+        * that way, otherwise bogus counts will be
+        * returned on read(). The only way to get out
+        * of error state is by explicit re-enabling
+        * of the event
+        */
+       if (event->state > PERF_EVENT_STATE_OFF)
+               perf_cgroup_event_disable(event, ctx);
+
        /*
         * Ensure event_sched_out() switches to OFF, at the very least
         * this avoids raising perf_pending_task() at this time.

Thanks


--
Sincerely,
Yeoreum Yun

Powered by blists - more mailing lists