linux-kernel - perf regression. Was: [PATCH V4 01/16] perf: Fix the throttle logic for a group

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <djxlh5fx326gcenwrr52ry3pk4wxmugu4jccdjysza7tlc5fef@ktp4rffawgcw>
Date: Sun, 1 Jun 2025 17:30:01 -0700
From: Alexei Starovoitov <alexei.starovoitov@...il.com>
To: kan.liang@...ux.intel.com
Cc: peterz@...radead.org, mingo@...hat.com, namhyung@...nel.org, 
	irogers@...gle.com, mark.rutland@....com, linux-kernel@...r.kernel.org, 
	linux-perf-users@...r.kernel.org, eranian@...gle.com, ctshao@...gle.com, tmricht@...ux.ibm.com, 
	leo.yan@....com, bpf@...r.kernel.org, andrii@...nel.org, ihor.solodrai@...ux.dev, 
	song@...nel.org, jolsa@...nel.org
Subject: perf regression. Was: [PATCH V4 01/16] perf: Fix the throttle logic
 for a group

On Tue, May 20, 2025 at 11:16:29AM -0700, kan.liang@...ux.intel.com wrote:
> From: Kan Liang <kan.liang@...ux.intel.com>
> 
> The current throttle logic doesn't work well with a group, e.g., the
> following sampling-read case.
> 
> $ perf record -e "{cycles,cycles}:S" ...
> 
> $ perf report -D | grep THROTTLE | tail -2
>             THROTTLE events:        426  ( 9.0%)
>           UNTHROTTLE events:        425  ( 9.0%)
> 
> $ perf report -D | grep PERF_RECORD_SAMPLE -a4 | tail -n 5
> 0 1020120874009167 0x74970 [0x68]: PERF_RECORD_SAMPLE(IP, 0x1):
> ... sample_read:
> .... group nr 2
> ..... id 0000000000000327, value 000000000cbb993a, lost 0
> ..... id 0000000000000328, value 00000002211c26df, lost 0
> 
> The second cycles event has a much larger value than the first cycles
> event in the same group.
> 
> The current throttle logic in the generic code only logs the THROTTLE
> event. It relies on the specific driver implementation to disable
> events. For all ARCHs, the implementation is similar. Only the event is
> disabled, rather than the group.
> 
> The logic to disable the group should be generic for all ARCHs. Add the
> logic in the generic code. The following patch will remove the buggy
> driver-specific implementation.
> 
> The throttle only happens when an event is overflowed. Stop the entire
> group when any event in the group triggers the throttle.
> The MAX_INTERRUPTS is set to all throttle events.
> 
> The unthrottled could happen in 3 places.
> - event/group sched. All events in the group are scheduled one by one.
>   All of them will be unthrottled eventually. Nothing needs to be
>   changed.
> - The perf_adjust_freq_unthr_events for each tick. Needs to restart the
>   group altogether.
> - The __perf_event_period(). The whole group needs to be restarted
>   altogether as well.
> 
> With the fix,
> $ sudo perf report -D | grep PERF_RECORD_SAMPLE -a4 | tail -n 5
> 0 3573470770332 0x12f5f8 [0x70]: PERF_RECORD_SAMPLE(IP, 0x2):
> ... sample_read:
> .... group nr 2
> ..... id 0000000000000a28, value 00000004fd3dfd8f, lost 0
> ..... id 0000000000000a29, value 00000004fd3dfd8f, lost 0
> 
> Suggested-by: Peter Zijlstra (Intel) <peterz@...radead.org>
> Signed-off-by: Kan Liang <kan.liang@...ux.intel.com>
> ---
>  kernel/events/core.c | 66 ++++++++++++++++++++++++++++++--------------
>  1 file changed, 46 insertions(+), 20 deletions(-)

This patch breaks perf hw events somehow.

After merging this into bpf trees we see random "watchdog: BUG: soft lockup"
with various stack traces followed up:
[   78.620749] Sending NMI from CPU 8 to CPUs 0:
[   76.387722] NMI backtrace for cpu 0
[   76.387722] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G           O L      6.15.0-10818-ge0f0ee1c31de #1163 PREEMPT
[   76.387722] Tainted: [O]=OOT_MODULE, [L]=SOFTLOCKUP
[   76.387722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[   76.387722] RIP: 0010:_raw_spin_lock_irqsave+0xc/0x40
[   76.387722] Call Trace:
[   76.387722]  <IRQ>
[   76.387722]  hrtimer_try_to_cancel.part.0+0x24/0xe0
[   76.387722]  hrtimer_cancel+0x21/0x40
[   76.387722]  cpu_clock_event_stop+0x64/0x70
[   76.387722]  __perf_event_account_interrupt+0xcf/0x140
[   76.387722]  __perf_event_overflow+0x36/0x340
[   76.387722]  ? hrtimer_start_range_ns+0x2c1/0x420
[   76.387722]  ? kvm_sched_clock_read+0x11/0x20
[   76.387722]  perf_swevent_hrtimer+0xaf/0x100
[   76.387722]  ? cpu_clock_event_add+0x6e/0x90
[   76.387722]  ? event_sched_in+0xc3/0x190
[   76.387722]  ? update_load_avg+0x87/0x3d0
[   76.387722]  ? _raw_spin_unlock+0xe/0x20
[   76.387722]  ? sched_balance_update_blocked_averages+0x59b/0x6a0
[   76.387722]  ? ctx_sched_in+0x184/0x210
[   76.387722]  ? kvm_sched_clock_read+0x11/0x20
[   76.387722]  ? sched_clock_cpu+0x55/0x190
[   76.387722]  ? perf_exclude_event+0x50/0x50
[   76.387722]  __hrtimer_run_queues+0x111/0x290
[   76.387722]  hrtimer_interrupt+0xff/0x240
[   76.387722]  __sysvec_apic_timer_interrupt+0x4f/0x110
[   76.387722]  sysvec_apic_timer_interrupt+0x6c/0x90

After reverting:
commit e800ac51202f ("perf: Only dump the throttle log for the leader")
commit 9734e25fbf5a ("perf: Fix the throttle logic for a group")
everything is back to normal.

There are many ways to reproduce.
Any test that sets up perf hw event followed up by tests that IPIs all cpus.
One way:
selftests/bpf/test_progs -t stacktrace_build_id_nmi
selftests/bpf/test_progs -t unpriv_bpf_disabled

Please take a look.