linux-kernel - Re: Benchmarks of kernel tracing options (ftrace and ktrace)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <AANLkTi=Y2X5EuqiCncPeNefgHs6vvGmnZpsyyddRR2g_@mail.gmail.com>
Date:	Tue, 16 Nov 2010 15:27:58 -0800
From:	David Sharp <dhsharp@...gle.com>
To:	Steven Rostedt <rostedt@...dmis.org>
Cc:	linux-kernel@...r.kernel.org,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	Ingo Molnar <mingo@...e.hu>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Michael Rubin <mrubin@...gle.com>,
	Frederic Weisbecker <fweisbec@...il.com>
Subject: Re: Benchmarks of kernel tracing options (ftrace and ktrace)

On Wed, Oct 13, 2010 at 4:50 PM, Steven Rostedt <rostedt@...dmis.org> wrote:
> On Wed, 2010-10-13 at 16:19 -0700, David Sharp wrote:
>> Google uses kernel tracing aggressively in the its data centers. We
>
> Thanks!
>
>> wrote our own kernel tracer, ktrace. However ftrace, perf and LTTng
>> all have a better feature set than ktrace, so we are abandoning that
>> code.
>
> Cool!
>
>>
>> We see several implementations of tracing aimed at the mainline kernel
>> and wanted a fair comparison of each of them to make sure they will
>> not significantly impact performance. A tracing toolkit that is too
>> expensive is not usable in our environment.
>>
>
> [ snip for now (I'm traveling) ]
>
>> This first set of benchmark results compares ftrace to ktrace. The
>> numbers below are the "on" result minus the "off" result for each
>> configuration.
>>
>> ktrace: 200ns  (tracepoint: kernel_getuid)
>> ftrace: 224ns   (tracepoint: timer:sys_getuid)
>> ftrace: 587ns   (tracepoint: syscalls:sys_enter_getuid)
>
>
>> The last result shows that the syscall tracing is about twice as
>> expensive as a normal tracepoint, which is interesting.
>
> Argh, the syscall tracing has a lot of overhead. There is only one
> tracepoint that is hooked into the ptrace code, and will save all
> registers before calling the functions. It enables tracing on all
> syscalls and there's a table that decides whether or not to trace the
> syscall.
>
> So I'm not surprised with the result that the syscall trace point is so
> slow (note, perf uses the same infrastructure).
>
> -- Steve

Steve, Mike told me that you may have some patches to improve overhead
of syscall tracing. I'd be interested in testing them. Can you send
them my way?

Thanks,
d#
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/