linux-kernel - Re: [PATCH v3 1/2] lib: stacklog_debug: Introduce helper tool for collecting and displaying stacktraces

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CANDhNCpGYkYMX+QUydMJJ4ED-DTVOHr-z21Wpgfkaj2a8Kqd1w@mail.gmail.com>
Date: Thu, 2 Jan 2025 14:18:15 -0800
From: John Stultz <jstultz@...gle.com>
To: Steven Rostedt <rostedt@...dmis.org>
Cc: LKML <linux-kernel@...r.kernel.org>, Alexander Potapenko <glider@...gle.com>, 
	Andrey Konovalov <andreyknvl@...il.com>, Andrew Morton <akpm@...ux-foundation.org>, 
	kernel-team@...roid.com
Subject: Re: [PATCH v3 1/2] lib: stacklog_debug: Introduce helper tool for
 collecting and displaying stacktraces

On Thu, Dec 19, 2024 at 6:21 AM Steven Rostedt <rostedt@...dmis.org> wrote:
>
> On Wed, 18 Dec 2024 22:06:05 -0800
> John Stultz <jstultz@...gle.com> wrote:
>
> > When debugging, its often useful to understand how a function is
> > called and the different paths taken to get there. Usually
> > dump_stack() can be used for this purpose. However there are a
> > number of cases where a function is called very frequently,
> > making dump_stack far too noisy to be useful.
> >
> > This is a little debug tool that utilizes stackdepot to capture
> > unique stack traces and store them in a circular buffer.
> >
> > In the code, the developer adds: stacklog_debug_save() calls
> > at points of interest (as they might with stack_dump()).
> >
> > Then after running the kernel, the developer can dump the unique
> > stack traces from the buffer via:
> >    cat /sys/kernel/debug/stacklog_debug
> >
> > This is pretty trivial, but I've had this hanging around for
> > awhile and recently hit another case where it was helpful, so I
> > figured it would be worth sending it out for feedback as to if
> > others thought it would be useful enough to merge upstream or to
> > possibly rework into stackdepot itself?
>
> Have you looked at using ftrace?
>
> You can even do histograms of stack traces from tracepoints (including
> kprobe events).
>
> Let's say we want to see all the calls to kmalloc:
>
>  # echo 'hist:keys=common_stacktrace' > /sys/kernel/tracing/events/kmem/kmalloc/trigger
>  # cat /sys/kernel/tracing/events/kmem/kmalloc/hist

Cool!  I guess it does depend on the trace-event being there (or added
via kprobe), but it indeed does overlap with my tool, so I'll probably
abandon efforts to upstream mine.

That said, for not-so-bright printk debugging focused developers like
myself, adding tracepoints (explicitly or via kprobes) to the points
we are interested in (along with remembering the advanced ftrace
commands) feels a like a bit more work than adding a dump_stack-like
one-liner, but maybe that's a better thing to try to improve on the
ftrace side instead of adding my new tool.  I'll think about that a
bit.

thanks
-john