linux-kernel - Re: [PATCH v3] perf trace: Implement syscall summary in BPF

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAH0uvogCka=hXsyPXboZS7znOgFHYhaMQ0H0VGMEM_z_AdBZYw@mail.gmail.com>
Date: Sun, 23 Mar 2025 21:47:45 -0700
From: Howard Chu <howardchu95@...il.com>
To: Namhyung Kim <namhyung@...nel.org>, Arnaldo Carvalho de Melo <acme@...nel.org>, 
	Ian Rogers <irogers@...gle.com>, Kan Liang <kan.liang@...ux.intel.com>
Cc: Jiri Olsa <jolsa@...nel.org>, Adrian Hunter <adrian.hunter@...el.com>, 
	Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...nel.org>, 
	LKML <linux-kernel@...r.kernel.org>, linux-perf-users@...r.kernel.org, 
	Song Liu <song@...nel.org>, bpf@...r.kernel.org
Subject: Re: [PATCH v3] perf trace: Implement syscall summary in BPF

Hello,

On Sun, Mar 23, 2025 at 6:49 PM Howard Chu <howardchu95@...il.com> wrote:
>
> Hello Namhyung,
>
> On Fri, Mar 21, 2025 at 11:42:55AM -0700, Namhyung Kim wrote:
> > When -s/--summary option is used, it doesn't need (augmented) arguments
> > of syscalls.  Let's skip the augmentation and load another small BPF
> > program to collect the statistics in the kernel instead of copying the
> > data to the ring-buffer to calculate the stats in userspace.  This will
> > be much more light-weight than the existing approach and remove any lost
> > events.
> >
> > Let's add a new option --bpf-summary to control this behavior.  I cannot
> > make it default because there's no way to get e_machine in the BPF which
> > is needed for detecting different ABIs like 32-bit compat mode.
> >
> > No functional changes intended except for no more LOST events. :)  But
> > it only works with -a/--all-cpus for now.
> >
> >   $ sudo ./perf trace -as --summary-mode=total --bpf-summary sleep 1
> >
> >    Summary of events:
> >
> >    total, 6194 events
> >
> >      syscall            calls  errors  total       min       avg       max       stddev
> >                                        (msec)    (msec)    (msec)    (msec)        (%)
> >      --------------- --------  ------ -------- --------- --------- ---------     ------
> >      epoll_wait           561      0  4530.843     0.000     8.076   520.941     18.75%
> >      futex                693     45  4317.231     0.000     6.230   500.077     21.98%
> >      poll                 300      0  1040.109     0.000     3.467   120.928     17.02%
> >      clock_nanosleep        1      0  1000.172  1000.172  1000.172  1000.172      0.00%
> >      ppoll                360      0   872.386     0.001     2.423   253.275     41.91%
> >      epoll_pwait           14      0   384.349     0.001    27.453   380.002     98.79%
> >      pselect6              14      0   108.130     7.198     7.724     8.206      0.85%
> >      nanosleep             39      0    43.378     0.069     1.112    10.084     44.23%
> >      ...
> >
> > Cc: Howard Chu <howardchu95@...il.com>
> > Signed-off-by: Namhyung Kim <namhyung@...nel.org>
> > ---
> > v3)
> >  * support -S/--with-summary option too  (Howard)
>
> It gave me segfault somehow.
>
> (gdb) bt
> #0  sighandler_dump_stack (sig=32767) at util/debug.c:322
> #1  <signal handler called>
> #2  0x00005555556d2383 in hashmap_find ()
> #3  0x000055555567474a in thread__update_stats (thread=0x5555564acc60, ttrace=0x5555564ad7a0, id=0, sample=0x7fffffff8f10, err=1,
>     trace=0x7fffffffb1a0) at builtin-trace.c:2616
> #4  0x00005555556757cb in trace__sys_exit (trace=0x7fffffffb1a0, evsel=0x5555561879d0, event=0x7fffed980000, sample=0x7fffffff8f10)
>     at builtin-trace.c:2924
> #5  0x0000555555677e1b in trace__handle_event (trace=0x7fffffffb1a0, event=0x7fffed980000, sample=0x7fffffff8f10) at builtin-trace.c:3619
> #6  0x00005555556796fb in __trace__deliver_event (trace=0x7fffffffb1a0, event=0x7fffed980000) at builtin-trace.c:4173
> #7  0x0000555555679859 in trace__deliver_event (trace=0x7fffffffb1a0, event=0x7fffed980000) at builtin-trace.c:4201
> #8  0x000055555567abed in trace__run (trace=0x7fffffffb1a0, argc=2, argv=0x7fffffffeb30) at builtin-trace.c:4590
> #9  0x000055555567f102 in cmd_trace (argc=2, argv=0x7fffffffeb30) at builtin-trace.c:5803
> #10 0x0000555555685252 in run_builtin (p=0x5555560eaf28 <commands+648>, argc=7, argv=0x7fffffffeb30) at perf.c:351
> #11 0x00005555556854fd in handle_internal_command (argc=7, argv=0x7fffffffeb30) at perf.c:404
> #12 0x000055555568565e in run_argv (argcp=0x7fffffffe91c, argv=0x7fffffffe910) at perf.c:448
> #13 0x00005555556859af in main (argc=7, argv=0x7fffffffeb30) at perf.c:556

the command I used is:

perf $ sudo ./perf trace -aS --summary-mode=total --bpf-summary -- sleep 1
[sudo] password for howard:
perf: Segmentation fault
Obtained 14 stack frames.
./perf(dump_stack+0x35) [0x591e26c8a735]
./perf(sighandler_dump_stack+0x2d) [0x591e26c8a7dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x45250) [0x737072045250]
./perf(+0x1543ed) [0x591e26ba43ed]
./perf(+0xff616) [0x591e26b4f616]
./perf(+0xfc3f4) [0x591e26b4c3f4]
./perf(+0x10193b) [0x591e26b5193b]
./perf(cmd_trace+0x205e) [0x591e26b56a9e]
./perf(+0x10b0e0) [0x591e26b5b0e0]
./perf(+0x10b3fb) [0x591e26b5b3fb]
./perf(main+0x2fb) [0x591e26ad63eb]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a3b8) [0x73707202a3b8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x73707202a47b]
./perf(_start+0x25) [0x591e26ad6a35]
Segmentation fault

Thanks,
Howard