[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAP045ArpMUYfT2GqdS931xG1nPqeh6z3zTq2RWJAkLXq3Ju6HA@mail.gmail.com>
Date: Wed, 28 Jun 2017 10:36:20 -0700
From: Kyle Huey <me@...ehuey.com>
To: Mark Rutland <mark.rutland@....com>
Cc: "Jin, Yao" <yao.jin@...ux.intel.com>,
Ingo Molnar <mingo@...nel.org>,
"Peter Zijlstra (Intel)" <peterz@...radead.org>,
stable@...r.kernel.org,
Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
Arnaldo Carvalho de Melo <acme@...hat.com>,
Jiri Olsa <jolsa@...hat.com>,
Linus Torvalds <torvalds@...ux-foundation.org>,
Namhyung Kim <namhyung@...nel.org>,
Stephane Eranian <eranian@...gle.com>,
Thomas Gleixner <tglx@...utronix.de>,
Vince Weaver <vincent.weaver@...ne.edu>, acme@...nel.org,
jolsa@...nel.org, kan.liang@...el.com,
Will Deacon <will.deacon@....com>, gregkh@...uxfoundation.org,
"Robert O'Callahan" <robert@...llahan.org>,
open list <linux-kernel@...r.kernel.org>
Subject: Re: [REGRESSION] perf/core: PMU interrupts dropped if we entered the
kernel in the "skid" region
On Wed, Jun 28, 2017 at 10:19 AM, Mark Rutland <mark.rutland@....com> wrote:
> On Wed, Jun 28, 2017 at 09:46:43AM -0700, Kyle Huey wrote:
>> On Wed, Jun 28, 2017 at 3:12 AM, Mark Rutland <mark.rutland@....com> wrote:
>> > On Tue, Jun 27, 2017 at 09:51:00PM -0700, Kyle Huey wrote:
>> >> My understanding of the situation is as follows:
>> >>
>> >> There is some time, call it t_0, where the hardware counter overflows.
>> >> The PMU triggers an interrupt, but this is not instantaneous. Call
>> >> the time when the interrupt is actually delivered t_1. Then t_1 - t_0
>> >> is the "skid".
>> >>
>> >> Note that if the counter is `exclude_kernel`, then at t_0 the CPU
>> >> *must* be running a userspace program. But by t_1, the CPU may be
>> >> doing something else. Your patch changed things so that if at t_1 the
>> >> CPU is in the kernel, then the interrupt is discarded. But rr has
>> >> programmed the counter to deliver a signal on overflow (via F_SETSIG
>> >> on the fd returned by perf_event_open). This change results in the
>> >> signal never being delivered, because the interrupt was ignored.
>> >> (More accurately, the signal is delivered the *next* time the counter
>> >> overflows, which is far past where we wanted to inject our
>> >> asynchronous event into our tracee.
>> >
>> > Yes, this is a bug.
>> >
>> > As we're trying to avoid smapling state, I think we can move the check
>> > into perf_prepare_sample() or __perf_event_output(), where that state is
>> > actually sampled. I'll take a look at that momentarily.
>> >
>> > Just to clarify, you don't care about the sample state at all? i.e. you
>> > don't need the user program counter?
>>
>> Right. `sample_regs_user`, `sample_star_user`, `branch_sample_type`,
>> etc are all 0.
>> https://github.com/mozilla/rr/blob/cf594dd01f07d96a61409e9f41a29f78c8c51693/src/PerfCounters.cc#L194
>> is what we do use.
>
> Given that, I must be missing something.
>
> In __perf_event_overflow(), we already bail out early if
> !is_sampling_event(event), i.e. when the sample_period is 0.
>
> Your attr has a sample_period of zero, so something must be initialising
> that.
>
> Do you always call PERF_EVENT_IOC_PERIOD, or is something in the core
> fiddling with the sample period behind your back?
We always either set sample_period or call PERF_EVENT_IOC_PERIOD (with
an enormous number if we don't actually want an interrupt. See
`PerfCounters::reset`, line 446.
> It seems odd that an event without any samples to take has a sample
> period. I'm surprised that there's not *some* sample_type set.
Perhaps sample_period is misleadingly named :) Alternatively, you
could imagine it as sampling where we're only interested in whether
the counter passed the sampling value or not.
>> > Is that signal delivered to the tracee, or to a different process that
>> > traces it? If the latter, what ensures that the task is stopped
>> > sufficiently quickly?
>>
>> It's delivered to the tracee (via an F_SETOWN_EX with the tracee tid).
>> In practice we've found that on modern Intel hardware that the
>> interrupt and resulting signal delivery delay is bounded by a
>> relatively small number of counter events.
>
> Ok.
>
> Thanks,
> Mark.
- Kyle
Powered by blists - more mailing lists