linux-kernel - Re: [PATCH v9 00/22] tracing: 'hist' triggers

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 22 Jul 2015 20:55:19 -0500
From:	Tom Zanussi <tom.zanussi@...ux.intel.com>
To:	Brendan Gregg <brendan.d.gregg@...il.com>
Cc:	Steven Rostedt <rostedt@...dmis.org>, daniel.wagner@...-carit.de,
	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>,
	Namhyung Kim <namhyung@...nel.org>, josh@...htriplett.org,
	Andi Kleen <andi@...stfloor.org>,
	LKML <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v9 00/22] tracing: 'hist' triggers

Hi Brendan,

On Wed, 2015-07-22 at 11:29 -0700, Brendan Gregg wrote:
> G'Day Tom,
> 
> On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi
> <tom.zanussi@...ux.intel.com> wrote:
> >
> > This is v9 of the 'hist triggers' patchset.
> >
> [...]
> 
> I've browsed the functionality (sorry, catching up), and it looks like
> this will solve a number of common problems. But it seems
> tantalizingly close to solving a few more. These may already be on
> your future todo list.
> 
> A) CPU stack profiling
> 
> Kernel stacktrace as a key will be hugely useful; is there a way to
> enable this for a sampling profile? (eg, what perf record -F 99 does).
> I take CPU profiles daily, and would prefer to aggregate stacks
> in-kernel. Also, would like user stacktrace as a key (even if it's
> just the hex).
> 

This wasn't on my todo list but I can see how it would be useful.  On
the list now. ;-)

> B) Key buckets
> 
> Eg, imagine:
> 
> echo 'hist:keys=common_pid.execname,count.log2:val=count' >
> /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
> 
> to get a log2 bucketized histogram of syscall read request size. Same
> for any value where using the value as a key gets too verbose, and you
> just want a rough look at the distribution. (Would make it more
> readable if it could also be sorted by the log2 value.)
> 

This I actually had an early implementation of, which I plan on
reviving...

> C) Latency as a bucket key
> 
> With kprobes, we could then have a log2 histogram of any function call
> latency, collected efficiently. (There's already the function timers
> in ftrace, which I'm using from function_graph with filters sets to
> only match the target function.)
> 

My original thought for doing this kind of thing was to generalize the
concept of a 'difference'.  I used it in v1 as a way to calculate the
difference between requested and allocated sizes for memory allocations,
which was kind of pointless, though convenient.  The real value would of
course be in applying it to inter-event values rather than intra-event.
In this case it would be an inter-event difference between timestamps.

And in my previous patchset, I had a 'function_hist' tracer, similar to
and in fact based on the same code as function_graph, that would simply
aggregate hitcounts for every function call in the kernel, which indeed
was pretty efficient.  So the pieces are or have been there to do
something like this, just a matter of putting them together.

> ... Those are the other common use cases, that the hist functionality
> seemed suited for. Beyond that gets more custom, and we can use eBPF.
> 

Exactly.

Tom

> Brendan


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/