linux-kernel - Re: "perf: interrupt took too long" messages even with perf_cpu_time_max

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160402111726.GB305@x4>
Date:	Sat, 2 Apr 2016 13:17:26 +0200
From:	Markus Trippelsdorf <markus@...ppelsdorf.de>
To:	Peter Zijlstra <peterz@...radead.org>
Cc:	linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...hat.com>,
	Arnaldo Carvalho de Melo <acme@...nel.org>,
	Alexander Shishkin <alexander.shishkin@...ux.intel.com>
Subject: Re: "perf: interrupt took too long" messages even with
 perf_cpu_time_max_percent==0

On 2016.04.02 at 13:00 +0200, Peter Zijlstra wrote:
> On Sat, Apr 02, 2016 at 11:32:39AM +0200, Markus Trippelsdorf wrote:
> > Current git kernel sometimes shows:
> >
> >  perf: interrupt took too long (71 > 52), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (103 > 88), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (130 > 128), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (175 > 162), lowering kernel.perf_event_max_sample_rate to 300
> >  perf: interrupt took too long (219 > 218), lowering kernel.perf_event_max_sample_rate to 300
> > ...
> >
> > when running e.g. "perf top" even when
> > /proc/sys/kernel/perf_cpu_time_max_percent is set to 0.
>
>
> Ah, was 0 also meant to disable it?

Yes. From Documentation/sysctl/kernel.txt:

 perf_cpu_time_max_percent:
 
 Hints to the kernel how much CPU time it should be allowed to
 use to handle perf sampling events.  If the perf subsystem
 is informed that its samples are exceeding this limit, it
 will drop its sampling frequency to attempt to reduce its CPU
 usage.
 
 Some perf sampling happens in NMIs.  If these samples
 unexpectedly take too long to execute, the NMIs can become
 stacked up next to each other so much that nothing else is
 allowed to execute.
 
 0: disable the mechanism.  Do not monitor or correct perf's
    sampling rate no matter how CPU time it takes.
 
 1-100: attempt to throttle perf's sample rate to this
    percentage of CPU.  Note: the kernel calculates an
    "expected" length of each sample event.  100 here means
    100% of that expected length.  Even if this is set to
    100, you may still see sample throttling if this
    length is exceeded.  Set to 0 if you truly do not care
    how much CPU is consumed.

> Does the below help?

Thanks. I will test it later. But 91a612eea9a3 makes the assumption that only
sysctl_perf_cpu_time_max_percent==100 disables the feature also in
kernel/events/core.c.

--
Markus