[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <87zhd4ut1y.fsf@toke.dk>
Date: Thu, 27 Feb 2020 13:13:29 +0100
From: Toke Høiland-Jørgensen <toke@...hat.com>
To: Luigi Rizzo <lrizzo@...gle.com>
Cc: linux-kernel@...r.kernel.org,
Masami Hiramatsu <mhiramat@...nel.org>,
Andrew Morton <akpm@...ux-foundation.org>,
Greg KH <gregkh@...uxfoundation.org>,
naveen.n.rao@...ux.ibm.com, ardb@...nel.org,
Luigi Rizzo <rizzo@....unipi.it>,
Paolo Abeni <pabeni@...hat.com>, giuseppe.lettieri@...pi.it,
Jesper Dangaard Brouer <hawk@...nel.org>, mingo@...hat.com,
acme@...nel.org, Steven Rostedt <rostedt@...dmis.org>,
Peter Zijlstra <peterz@...radead.org>
Subject: Re: [PATCH v3 0/2] kstats: kernel metric collector
Luigi Rizzo <lrizzo@...gle.com> writes:
> On Wed, Feb 26, 2020 at 3:11 PM Toke Høiland-Jørgensen <toke@...hat.com> wrote:
>>
>> Luigi Rizzo <lrizzo@...gle.com> writes:
>>
>> > - the runtime cost and complexity of hooking bpf code is still a bit
>> > unclear to me. kretprobe or tracepoints are expensive, I suppose that
>> > some lean hook replace register_kretprobe() may exist and the
>> > difference from inline annotations would be marginal (we'd still need
>> > to put in the hooks around the code we want to time, though, so it
>> > wouldn't be a pure bpf solution). Any pointers to this are welcome;
>> > Alexei mentioned fentry/fexit and bpf trampolines, but I haven't found
>> > an example that lets me do something equivalent to kretprobe (take a
>> > timestamp before and one after a function without explicit
>> > instrumentation)
>>
>> As Alexei said, with fentry/fexit the overhead should be on par with
>> your example. This functionality is pretty new, though, so I can
>> understand why it's not obvious how to do things with it yet :)
>>
>> I think the best place to look is currently in selftests/bpf in the
>> kernel sources. Grep for 'fexit' and 'fentry' in the progs/ subdir.
>> test_overhead.c and kfree_skb.c seem to have some examples you may be
>> able to work from.
>
> Thank you for the precise reference, Toke.
> I tweaked test_overhead.c to measure (using kstats) the cost of the various
> hooks and I can confirm that fentry and fexit are pretty fast. The
> following table
> shows the p90 runtime of __set_task_comm() at low (100/s) and high (1M/s) rates:
>
> 90 percentile of __set_task_comm() runtime
> (accuracy: 30ns)
> call rate base kprobe kretprobe tracepoint fentry fexit
> 100/sec 270 870 1220 500 400 450
> >1M/s 60 120 210 90
> 70 80
>
> For high rate operation, the overhead of fentry and fexit is quite good,
> even better than tracepoints, and well below the clock's accuracy
> (more detailed measurements indicate ~5ns for fentry, ~10ns for fexit).
> At very low call rates there is an extra 150-200ns
> but that is expected due to the out of line code.
Great, thank you for the performance numbers! This is indeed quite good :)
-Toke
Powered by blists - more mailing lists