| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20241105184812.e8256c3078505b63b83a6cf6@kernel.org>
Date: Tue, 5 Nov 2024 18:48:12 +0900
From: Masami Hiramatsu (Google) <mhiramat@...nel.org>
To: Arnaldo Carvalho de Melo <acme@...nel.org>
Cc: Namhyung Kim <namhyung@...nel.org>, Peter Zijlstra
<peterz@...radead.org>, Ingo Molnar <mingo@...hat.com>, Ian Rogers
<irogers@...gle.com>, Dima Kogan <dima@...retsauce.net>, Alexander Lobakin
<aleksander.lobakin@...el.com>, Przemek Kitszel
<przemyslaw.kitszel@...el.com>, linux-perf-users@...r.kernel.org,
linux-kernel@...r.kernel.org
Subject: Re: [PATCH 0/4] perf-probe: Improbe non-C language support
On Mon, 4 Nov 2024 15:56:00 -0300
Arnaldo Carvalho de Melo <acme@...nel.org> wrote:
> On Tue, Nov 05, 2024 at 01:17:08AM +0900, Masami Hiramatsu (Google) wrote:
> > Hi,
>
> > Here is a series of patches for perf probe to improve non-C language
> > (e.g. Rust, Go) support.
>
> > The non-C symbols are demangled style in debuginfo, e.g. golang stores
>
> > ----
> > $ ./perf probe -x /work/go/example/outyet/main -F main*
> > main.(*Server).ServeHTTP
> > main.(*Server).ServeHTTP.Print.func1
> > main.(*Server).poll
> > ...
> > -----
>
> I presented about this last year:
>
> https://tracingsummit.org/ts/2023/bpf-non-c/
> https://tracingsummit.org/ts/2023/files/Trying_to_use_uprobes_and_BPF_on_non-C_userspace.pdf
> https://www.youtube.com/watch?v=RDFRy1vWyHg&feature=youtu.be
Nice!
>
> So trying to do some of the things I did while playing with golang, and
> with your patches applied, I only had to cope with a minor clash with a
> patch by Ian Rogers that is present on perf-tools-next, related to:
>
> char buf[MAX_EVENT_NAME_LEN];
>
> That in your patch expects the old 64 hard coded value, which will
> appear in the my tests:
Ah, OK let me rebase on it.
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -x main -F github*counter*
> github.com/prometheus/client_golang/prometheus.(*counter).Add
> github.com/prometheus/client_golang/prometheus.(*counter).AddWithExemplar
> github.com/prometheus/client_golang/prometheus.(*counter).Collect
> github.com/prometheus/client_golang/prometheus.(*counter).Desc
> github.com/prometheus/client_golang/prometheus.(*counter).Describe
> github.com/prometheus/client_golang/prometheus.(*counter).Inc
> github.com/prometheus/client_golang/prometheus.(*counter).Write
> github.com/prometheus/client_golang/prometheus.(*counter).updateExemplar
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -x main -F github*counter*
>
> Then trying to add for all those:
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -d *:*
> "*:*" does not hit any event.
> Error: Failed to delete events.
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -l
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -x main github*counter*
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> snprintf() failed: -7; the event name 'github_com_prometheus_client_golang_prometheus_counter_AddWithExemplar' is too long
> Hint: Set a shorter event with syntax "EVENT=PROBEDEF"
> EVENT: Event name (max length: 64 bytes).
> Error: Failed to add events.
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> But:
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -l
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Desc (on github.com/prometheus/client_golang/prometheus.(*counter).Des>
> (END)
>
> That pager thing looks odd as well, since there is just one line in the
> output...
>
> So it failed to do all those, added just one, maybe state that some were
> added but from the problematic one onwards it stopped? Or try all of
> them and just state the ones that couldn't be added?
OK, yes, it should show what events are actually added.
>
> I.e. something like:
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -x main -F github*counter* | while read probename ; do perf probe -x main $probename ; done
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> Failed to find debug information for address 0x3287e0
> Probe point 'github.com/prometheus/client_golang/prometheus.(*counter).Add' not found.
> Error: Failed to add events.
> snprintf() failed: -7; the event name 'github_com_prometheus_client_golang_prometheus_counter_AddWithExemplar' is too long
> Hint: Set a shorter event with syntax "EVENT=PROBEDEF"
> EVENT: Event name (max length: 64 bytes).
> Error: Failed to add events.
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> Failed to find debug information for address 0x33ab40
> Probe point 'github.com/prometheus/client_golang/prometheus.(*counter).Collect' not found.
> Error: Failed to add events.
> Error: event "github_com_prometheus_client_golang_prometheus_counter_Desc" already exists.
> Hint: Remove existing event by 'perf probe -d'
> or force duplicates by 'perf probe -f'
> or set 'force=yes' in BPF source.
> Error: Failed to add events.
> A function DIE doesn't have decl_line. Maybe broken DWARF?
> Failed to find debug information for address 0x33aba0
> Probe point 'github.com/prometheus/client_golang/prometheus.(*counter).Describe' not found.
> Error: Failed to add events.
> Added new event:
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc (on github.com/prometheus/client_golang/prometheus.(*counter).Inc in /home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc -aR sleep 1
>
> Added new event:
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Write (on github.com/prometheus/client_golang/prometheus.(*counter).Write in /home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
>
> You can now use it in all perf tools, such as:
>
> perf record -e probe_main:github_com_prometheus_client_golang_prometheus_counter_Write -aR sleep 1
>
> snprintf() failed: -7; the event name 'github_com_prometheus_client_golang_prometheus_counter_updateExemplar' is too long
> Hint: Set a shorter event with syntax "EVENT=PROBEDEF"
> EVENT: Event name (max length: 64 bytes).
> Error: Failed to add events.
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> In the end we get:
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf probe -l
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Desc (on github.com/prometheus/client_golang/prometheus.(*counter).Desc@...metheus/counter.go in /home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc (on github.com/prometheus/client_golang/prometheus.(*counter).Inc@...metheus/counter.go in /home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> probe_main:github_com_prometheus_client_golang_prometheus_counter_Write (on github.com/prometheus/client_golang/prometheus.(*counter).Write@...metheus/counter.go in /home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> That also explains the pager kicking in: I had to reduce font size in my
> xterm (gnome-terminal really) and then the perf pager wasn't used (no
> (END) at the last line waiting for me to press 'q').
>
> The ones that got installed are working:
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf trace -e probe_main:*
> 0.000 main/616840 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> 1001.043 main/616926 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> 1001.080 main/616926 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> 4000.994 main/616926 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> ^Croot@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# perf trace -e probe_main:*/max-stack=8/
> 0.000 main/616926 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> github.com/prometheus/client_golang/prometheus.(*counter).Inc (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> runtime.goexit.abi0 (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> 0.030 main/616926 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> github.com/prometheus/client_golang/prometheus.(*counter).Inc (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> runtime.goexit.abi0 (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> 3000.166 main/616840 probe_main:github_com_prometheus_client_golang_prometheus_counter_Inc(__probe_ip: 7506464)
> github.com/prometheus/client_golang/prometheus.(*counter).Inc (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> runtime.goexit.abi0 (/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus/main)
> ^Croot@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> I'll test this some more later/tomorrow, just wanted to give this first
> reaction, thanks for working on this!
Thanks for testing and the information!
Since the event name limitation is in the kernel too, user should specify
abbreviated event name in this case (or use only the last few words).
Thank you,
>
> Btw, some more info about the environment (fedora 40):
>
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus# readelf -wi main | head -20
> Contents of the .debug_info section:
>
> Compilation Unit @ offset 0:
> Length: 0x506 (32-bit)
> Version: 4
> Abbrev Offset: 0
> Pointer Size: 8
> <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
> <c> DW_AT_name : google.golang.org/protobuf/internal/strs
> <35> DW_AT_language : 22 (Go)
> <36> DW_AT_stmt_list : 0
> <3a> DW_AT_low_pc : 0x667c40
> <42> DW_AT_ranges : 0
> <46> DW_AT_comp_dir : .
> <48> DW_AT_producer : Go cmd/compile go1.22.7; regabi
> <68> Unknown AT value: 2905: strs
> <1><6d>: Abbrev Number: 5 (DW_TAG_subprogram)
> <6e> DW_AT_name : google.golang.org/protobuf/internal/strs.isASCIILower
> <a4> DW_AT_inline : 1 (inlined)
> <a5> DW_AT_decl_line : 188
> root@x1:/home/acme/git/libbpf-bootstrap/examples/c/tests/prometheus#
>
> - Arnaldo
>
> > And Rust stores
> > -----
> > $ ./perf probe -x /work/cro3/target/x86_64-unknown-linux-gnu/debug/cro3 -F cro3::cmd::servo*
> > cro3::cmd::servo::run
> > cro3::cmd::servo::run::CALLSITE
> > cro3::cmd::servo::run::CALLSITE::META
> > cro3::cmd::servo::run_control
> > -----
> >
> > These symbols are not parsed correctly because it looks like a file name or
> > including line numbers (`:` caused it.) So, I decided to introduce the changes
> >
> > - filename MUST start from '@'. (so it is able to distinguish the filename
> > and the function name)
> > - Fix to allow backslash to escape to --lines option.
> > - Introduce quotation mark support.
> > - Replace non-alnum character to '_' for event name (for non-C symbols).
> >
> > With these changes, we can run -L (--lines) on golang;
> >
> > ------
> > $ perf probe -x goexample/hello/hello -L \"main.main\"
> > <main.main@...rk/goexample/hello/hello.go:0>
> > 0 func main() {
> > // Configure logging for a command-line program.
> > 2 log.SetFlags(0)
> > 3 log.SetPrefix("hello: ")
> >
> > // Parse flags.
> > 6 flag.Usage = usage
> > 7 flag.Parse()
> > ------
> >
> > And Rust
> > ------
> > $ perf probe -x cro3 -L \"cro3::cmd::servo::run_show\"
> > <run_show@...rk/cro3/src/cmd/servo.rs:0>
> > 0 fn run_show(args: &ArgsShow) -> Result<()> {
> > 1 let list = ServoList::discover()?;
> > 2 let s = list.find_by_serial(&args.servo)?;
> > 3 if args.json {
> > 4 println!("{s}");
> > ------
> >
> > And event name are created automatically like below;
> >
> > $ ./perf probe -x /work/go/example/outyet/main -D 'main.(*Server).poll'
> > p:probe_main/main_Server_poll /work/go/example/outyet/main:0x353040
> >
> > $ ./perf probe -x cro3 -D \"cro3::cmd::servo::run_show\"
> > p:probe_cro3/cro3_cmd_servo_run_show /work/cro3/target/x86_64-unknown-linux-gnu/debug/cro3:0x197530
> >
> > We still need some more work, but these shows how perf-probe can work
> > with other languages.
> >
> > Thank you,
> >
> > ---
> >
> > Masami Hiramatsu (Google) (4):
> > perf-probe: Fix to ignore escaped characters in --lines option
> > perf-probe: Require '@' prefix for filename always
> > perf-probe: Introduce quotation marks support
> > perf-probe: Replace unacceptable characters when generating event name
> >
> >
> > tools/perf/util/probe-event.c | 136 ++++++++++++++++++++++------------------
> > tools/perf/util/probe-finder.c | 3 +
> > tools/perf/util/string.c | 100 +++++++++++++++++++++++++++++
> > tools/perf/util/string2.h | 2 +
> > 4 files changed, 180 insertions(+), 61 deletions(-)
> >
> > --
> > Masami Hiramatsu (Google) <mhiramat@...nel.org>
--
Masami Hiramatsu (Google) <mhiramat@...nel.org>
Powered by blists - more mailing lists