linux-kernel - [perf] periodic performance monitoring interrupts causing heavy delays

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <c72df5db-914f-0e9b-5b63-d8164817b360@ericsson.com>
Date:   Tue, 21 Jan 2020 13:28:53 +0100
From:   Zoltán Kiss Z <zoltan.z.kiss@...csson.com>
To:     <linux-kernel@...r.kernel.org>
Subject: [perf] periodic performance monitoring interrupts causing heavy
 delays

Hi,

I'm running some benchmarking on a Dell Poweredge R630 server (Xeon 
E5-2650 v4 Broadwell) with Ubuntu 18.04 (4.15.0-62 kernel). The 
application is busy looping on core's reserved with isolcpus. However 
every 5 seconds there is a ~1 ms glitch where my cores seem to be 
stopped. After checking interrupts it seems these glitches coincide with 
an NMI and a "Performance monitoring interrupt" on all the busy looping 
cores, while the idle ones doesn't receive it. I was not running perf 
during these occasions, but the below post suggests that there is still 
some kernel perf activity in the background which causes these interrupts:

https://serverfault.com/questions/714648/how-to-disable-perf-subsystem-in-linux-kernel

I can also see these "perf interrupt took too long" messages even when I 
never started perf by myself since the last reboot. I've tried to adjust 
the /proc/sys/kernel/perf_* knobs, without any luck.

The above post suggests that these periodic interrupts cannot be 
disabled in any way, and I would like to figure out if this is really 
the case, and if yes, why? A 1 ms glitch in every 5 seconds is quite 
unacceptable not just because of benchmarking noise, but also for 
latency sensitive applications like mine.
Also, I couldn't find any info about these periodic perf background jobs 
generating the interrupts, could anyone point me to at least some 
function names where I could start digging, so I could verify if indeed 
these glitches are due to perf?

Thanks,

Zoltan Kiss