Message-ID: <CAH0uvojdUT=OTBBZje--AJRd8gyHASqno5+Mhw2+_xMjvwqYPA@mail.gmail.com>
Date: Thu, 14 Nov 2024 12:34:24 -0800
From: Howard Chu <howardchu95@...il.com>
To: Arnaldo Carvalho de Melo <acme@...nel.org>
Cc: peterz@...radead.org, namhyung@...nel.org, irogers@...gle.com, 
	mingo@...hat.com, mark.rutland@....com, james.clark@...aro.org, 
	alexander.shishkin@...ux.intel.com, jolsa@...nel.org, adrian.hunter@...el.com, 
	kan.liang@...ux.intel.com, linux-perf-users@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [PATCH v2] perf trace: Fix perf trace tracing itself, creating
 feedback loops

Hello,

On Thu, Nov 14, 2024 at 12:03 PM Arnaldo Carvalho de Melo
<acme@...nel.org> wrote:
>
> On Thu, Nov 14, 2024 at 02:57:02PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Tue, Nov 12, 2024 at 02:07:36PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Oct 29, 2024 at 10:24:31PM -0700, Howard Chu wrote:
> > > > There exists a pids_filtered map in augmented_raw_syscalls.bpf.c that
> > > > ceases to provide functionality after the BPF skeleton migration:
> > > > commit 5e6da6be3082 ("perf trace: Migrate BPF augmentation to use a skeleton")
> > >
> > > Thanks, applied to perf-tools-next,
> >
> > There is a problem with this patch, I'm investigating it now:
> >
> > root@...ber:~# perf trace -e syscalls:sys_enter_exit_group true
> > perf: Segmentation fault
> > Obtained 12 stack frames.
> > perf() [0x6229a1]
> > perf() [0x622a8b]
> > /lib64/libc.so.6(+0x40d00) [0x7f704364fd00]
> > perf() [0x4b5f2e]
> > perf() [0x4ba3e6]
> > perf() [0x4c03bc]
> > perf() [0x4c0663]
> > perf() [0x4c07bc]
> > perf() [0x4c0b05]
> > /lib64/libc.so.6(+0x2a088) [0x7f7043639088]
> > /lib64/libc.so.6(__libc_start_main+0x8b) [0x7f704363914b]
> > perf() [0x412265]
> > Segmentation fault (core dumped)
> > root@...ber:~#
> >
> > (gdb) run trace -e syscalls:sys_enter_exit_group true
> > Starting program: /root/bin/perf trace -e syscalls:sys_enter_exit_group true
> >
> > This GDB supports auto-downloading debuginfo from the following URLs:
> >   <https://debuginfod.fedoraproject.org/>
> > Enable debuginfod for this session? (y or [n]) y
> > Debuginfod has been enabled.
> > To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > [Detaching after fork from child process 1965197]
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > trace__run (trace=0x7fffffffa5e0, argc=1, argv=0x7fffffffde90) at builtin-trace.c:4330
> > 4330          trace->filter_pids.map = trace->skel->maps.pids_filtered;
> > (gdb) bt
> > #0  trace__run (trace=0x7fffffffa5e0, argc=1, argv=0x7fffffffde90) at builtin-trace.c:4330
> > #1  0x00000000004ba3e6 in cmd_trace (argc=1, argv=0x7fffffffde90) at builtin-trace.c:5490
> > #2  0x00000000004c03bc in run_builtin (p=0xec4068 <commands+648>, argc=4, argv=0x7fffffffde90) at perf.c:351
> > #3  0x00000000004c0663 in handle_internal_command (argc=4, argv=0x7fffffffde90) at perf.c:404
> > #4  0x00000000004c07bc in run_argv (argcp=0x7fffffffdc7c, argv=0x7fffffffdc70) at perf.c:448
> > #5  0x00000000004c0b05 in main (argc=4, argv=0x7fffffffde90) at perf.c:560
> > (gdb) p trace->skel
> > $1 = (struct augmented_raw_syscalls_bpf *) 0x0
> > (gdb)
> >
> > I.e. when we specify a syscall event tracepoint, we expect to use
> > augmentation, but are not setting it up.
>
> We can't assume that trace->skel will be always set, as it is only
> initialized if trace->trace_syscalls is set, so I added this patch on
> top of yours:

Yeah, using syscall tracepoints won't load the BPF skeleton.

perf $ ./perf trace -e syscalls:sys_enter_exit_group
     0.000 dirname/3732242 syscalls:sys_enter_exit_group()
     0.268 bash/3732241 syscalls:sys_enter_exit_group()
     3.951 :3732246/3732246 syscalls:sys_enter_exit_group()
     4.756 tr/3732247 syscalls:sys_enter_exit_group()
     4.945 :3732245/3732245 syscalls:sys_enter_exit_group()
     5.235 bash/3732244 syscalls:sys_enter_exit_group()
     7.975 tmux/3732249 syscalls:sys_enter_exit_group()
     9.160 :3732252/3732252 syscalls:sys_enter_exit_group()
    10.062 tr/3732253 syscalls:sys_enter_exit_group()
    10.308 :3732251/3732251 syscalls:sys_enter_exit_group()
    10.619 bash/3732250 syscalls:sys_enter_exit_group()
    10.853 bash/3732248 syscalls:sys_enter_exit_group()
    11.143 bash/3732243 syscalls:sys_enter_exit_group()
    15.379 tmux: client/3732255 syscalls:sys_enter_exit_group()
    15.985 sh/3732254 syscalls:sys_enter_exit_group()
    20.111 tmux: client/3732257 syscalls:sys_enter_exit_group()
    20.697 sh/3732256 syscalls:sys_enter_exit_group()
    24.849 tmux: client/3732259 syscalls:sys_enter_exit_group()
    25.461 sh/3732258 syscalls:sys_enter_exit_group()
    27.087 date/3732261 syscalls:sys_enter_exit_group()
    27.333 sh/3732260 syscalls:sys_enter_exit_group()
    27.596 sh/3732240 syscalls:sys_enter_exit_group()
    27.850 tmux: server/3732239 syscalls:sys_enter_exit_group()
^Cperf $

>
> ⬢ [acme@...lbox perf-tools-next]$ git diff
> diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> index 0f8678142fbfdbcb..14794725669d49ee 100644
> --- a/tools/perf/builtin-trace.c
> +++ b/tools/perf/builtin-trace.c
> @@ -4327,7 +4327,8 @@ static int trace__run(struct trace *trace, int argc, const char **argv)
>                 }
>         }
>
> -       trace->filter_pids.map = trace->skel->maps.pids_filtered;
> +       if (trace->skel)
> +               trace->filter_pids.map = trace->skel->maps.pids_filtered;

LGTM.

>  #endif
>         err = trace__set_filter_pids(trace);
>         if (err < 0)

Tested-by: Howard Chu <howardchu95@...il.com>

Thanks,
Howard

Powered by blists - more mailing lists