Message-ID: <CAHPNGSTmv7qxMYOs6h1uevB4Rjiy9R+-YTt0gZ2D1tVh7-cuxQ@mail.gmail.com>
Date: Tue, 14 Oct 2025 13:13:03 -0700
From: Octavia Togami <octavia.togami@...il.com>
To: "Mi, Dapeng" <dapeng1.mi@...ux.intel.com>
Cc: Octavia Togami <octavia.togami@...il.com>, stable@...r.kernel.org, 
	regressions@...ts.linux.dev, peterz@...radead.org, 
	linux-kernel@...r.kernel.org, linux-perf-users@...r.kernel.org
Subject: Re: [REGRESSION] bisected: perf: hang when using async-profiler
 caused by perf: Fix the POLL_HUP delivery breakage

That patch is also working fine.

On Mon, Oct 13, 2025 at 11:41 PM Mi, Dapeng <dapeng1.mi@...ux.intel.com> wrote:
>
>
> On 10/13/2025 2:55 PM, Octavia Togami wrote:
> > That change appears to fix the problem on my end. I ran my reproducer
> > and some other tests multiple times without issue.
>
> @Octavia Thanks for checking this patch. But following Peter's comments, we
> need to update the fix. So could you please re-test the below changes? Thanks.
>
> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index 7541f6f85fcb..ed236b8bbcaa 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -11773,7 +11773,8 @@ static enum hrtimer_restart
> perf_swevent_hrtimer(struct hrtimer *hrtimer)
>
>         event = container_of(hrtimer, struct perf_event, hw.hrtimer);
>
> -       if (event->state != PERF_EVENT_STATE_ACTIVE)
> +       if (event->state != PERF_EVENT_STATE_ACTIVE ||
> +           event->hw.state & PERF_HES_STOPPED)
>                 return HRTIMER_NORESTART;
>
>         event->pmu->read(event);
> @@ -11827,7 +11828,7 @@ static void perf_swevent_cancel_hrtimer(struct
> perf_event *event)
>                 ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
>                 local64_set(&hwc->period_left, ktime_to_ns(remaining));
>
> -               hrtimer_cancel(&hwc->hrtimer);
> +               hrtimer_try_to_cancel(&hwc->hrtimer);
>         }
>  }
>
> @@ -11871,12 +11872,14 @@ static void cpu_clock_event_update(struct
> perf_event *event)
>
>  static void cpu_clock_event_start(struct perf_event *event, int flags)
>  {
> +       event->hw.state = 0;
>         local64_set(&event->hw.prev_count, local_clock());
>         perf_swevent_start_hrtimer(event);
>  }
>
>  static void cpu_clock_event_stop(struct perf_event *event, int flags)
>  {
> +       event->hw.state = PERF_HES_STOPPED;
>         perf_swevent_cancel_hrtimer(event);
>         if (flags & PERF_EF_UPDATE)
>                 cpu_clock_event_update(event);
> @@ -11950,12 +11953,14 @@ static void task_clock_event_update(struct
> perf_event *event, u64 now)
>
>  static void task_clock_event_start(struct perf_event *event, int flags)
>  {
> +       event->hw.state = 0;
>         local64_set(&event->hw.prev_count, event->ctx->time);
>         perf_swevent_start_hrtimer(event);
>  }
>
>  static void task_clock_event_stop(struct perf_event *event, int flags)
>  {
> +       event->hw.state = PERF_HES_STOPPED;
>         perf_swevent_cancel_hrtimer(event);
>         if (flags & PERF_EF_UPDATE)
>                 task_clock_event_update(event, event->ctx->time);
>
>
> >
> > On Sun, Oct 12, 2025 at 7:34 PM Mi, Dapeng <dapeng1.mi@...ux.intel.com> wrote:
> >>
> >> On 10/11/2025 4:31 PM, Octavia Togami wrote:
> >>> Using async-profiler
> >>> (https://github.com/async-profiler/async-profiler/) on Linux
> >>> 6.17.1-arch1-1 causes a complete hang of the CPU. This has been
> >>> reported by many people at https://github.com/lucko/spark/issues/530.
> >>> spark is a piece of software that uses async-profiler internally.
> >>>
> >>> As seen in https://github.com/lucko/spark/issues/530#issuecomment-3339974827,
> >>> this was bisected to 18dbcbfabfffc4a5d3ea10290c5ad27f22b0d240 perf:
> >>> Fix the POLL_HUP delivery breakage. Reverting this commit on 6.17.1
> >>> fixed the issue for me.
> >>>
> >>> Steps to reproduce:
> >>> 1. Get a copy of async-profiler. I tested both v3 (affects older spark
> >>> versions) and v4.1 (latest at time of writing). Unarchive it, this is
> >>> <async-profiler-dir>.
> >>> 2. Set kernel parameters kernel.perf_event_paranoid=1 and
> >>> kernel.kptr_restrict=0 as instructed by
> >>> https://github.com/async-profiler/async-profiler/blob/fb673227c7fb311f872ce9566769b006b357ecbe/docs/GettingStarted.md
> >>> 3. Install a version of Java that comes with jshell, i.e. Java 9 or
> >>> newer. Note: jshell is used for ease of reproduction. Any Java
> >>> application that is actively running will work.
> >>> 4. Run `printf 'int acc; while (true) { acc++; }' | jshell -`. This
> >>> will start an infinitely running Java process.
> >>> 5. Run `jps` and take the PID next to the text RemoteExecutionControl
> >>> -- this is the process that was just started.
> >>> 6. Attach async-profiler to this process by running
> >>> `<async-profiler-dir>/bin/asprof -d 1 <PID>`. This will run for one
> >>> second, then the system should freeze entirely shortly thereafter.
> >>>
> >>> I triggered a sysrq crash while the system was frozen, and the output
> >>> I found in journalctl afterwards is at
> >>> https://gist.github.com/octylFractal/76611ee76060051e5efc0c898dd0949e
> >>> I'm not sure if that text is actually from the triggered crash, but it
> >>> seems relevant. If needed, please tell me how to get the actual crash
> >>> report, I'm not sure where it is.
> >>>
> >>> I'm using an AMD Ryzen 9 5900X 12-Core Processor. Given that I've seen
> >>> no Intel reports, it may be AMD specific. I don't have an Intel CPU on
> >>> hand to test with.
> >>>
> >>> /proc/version: Linux version 6.17.1-arch1-1 (linux@...hlinux) (gcc
> >>> (GCC) 15.2.1 20250813, GNU ld (GNU Binutils) 2.45.0) #1 SMP
> >>> PREEMPT_DYNAMIC Mon, 06 Oct 2025 18:48:29 +0000
> >>> Operating System: Arch Linux
> >>> uname -mi: x86_64 unknown
> >> It looks the issue described in the link
> >> (https://lore.kernel.org/all/20250606192546.915765-1-kan.liang@linux.intel.com/T/#u)
> >> happens again but in a different way. :(
> >>
> >> As the commit message above link described,  cpu-clock (and task-clock) is
> >> a specific SW event which rely on hrtimer. The hrtimer handler calls
> >> __perf_event_overflow() and then event_stop (cpu_clock_event_stop()) and
> >> eventually call hrtimer_cancel() which traps into a dead loop which waits
> >> for the calling hrtimer handler finishes.
> >>
> >> As the
> >> change (https://lore.kernel.org/all/20250606192546.915765-1-kan.liang@linux.intel.com/T/#u),
> >> it should be enough to just disable the event and don't need an extra event
> >> stop.
> >>
> >> @Octavia, could you please check if the change below can fix this issue?
> >> Thanks.
> >>
> >> diff --git a/kernel/events/core.c b/kernel/events/core.c
> >> index 7541f6f85fcb..883b0e1fa5d3 100644
> >> --- a/kernel/events/core.c
> >> +++ b/kernel/events/core.c
> >> @@ -10343,7 +10343,20 @@ static int __perf_event_overflow(struct perf_event
> >> *event,
> >>                 ret = 1;
> >>                 event->pending_kill = POLL_HUP;
> >>                 perf_event_disable_inatomic(event);
> >> -               event->pmu->stop(event, 0);
> >> +
> >> +               /*
> >> +                * The cpu-clock and task-clock are two special SW events,
> >> +                * which rely on the hrtimer. The __perf_event_overflow()
> >> +                * is invoked from the hrtimer handler for these 2 events.
> >> +                * Avoid to call event_stop()->hrtimer_cancel() for these
> >> +                * 2 events since hrtimer_cancel() waits for the hrtimer
> >> +                * handler to finish, which would trigger a deadlock.
> >> +                * Only disabling the events is enough to stop the hrtimer.
> >> +                * See perf_swevent_cancel_hrtimer().
> >> +                */
> >> +               if (event->attr.config != PERF_COUNT_SW_CPU_CLOCK &&
> >> +                   event->attr.config != PERF_COUNT_SW_TASK_CLOCK)
> >> +                       event->pmu->stop(event, 0);
> >>         }
> >>
> >>         if (event->attr.sigtrap) {
> >>
> >>

Powered by blists - more mailing lists