linux-kernel - Re: perf is unable to read dward from go programs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20191203102234.GE17468@krava>
Date:   Tue, 3 Dec 2019 11:22:34 +0100
From:   Jiri Olsa <jolsa@...hat.com>
To:     Ivan Babrou <ivan@...udflare.com>
Cc:     Arnaldo Carvalho de Melo <arnaldo.melo@...il.com>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        linux-perf-users@...r.kernel.org,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Namhyung Kim <namhyung@...nel.org>,
        kernel-team <kernel-team@...udflare.com>
Subject: Re: perf is unable to read dward from go programs

On Mon, Dec 02, 2019 at 11:49:55AM -0800, Ivan Babrou wrote:
> I've tried building with libdw with mixed results:
> 
> 1. I can see stacks from some Go programs that were invisible before (yay!)
> 
> 2. Warnings like below still appear in great numbers for a system-wide
> flamegraph:
> 
> BFD: Dwarf Error: found dwarf version '18345', this reader only
> handles version 2, 3 and 4 information.
> 
> I'm not sure how to pinpoint this to a particular binary and would
> appreciate some help with this.

I'd need some way of reproducing this, could you please
paste me command lines you used?

> 
> 3. It takes minutes to produce a flamegraph of a running system
> whereas before it only took seconds. See this flamegraph of "perf
> script" itself:
> 
> * https://gist.github.com/bobrik/a9c46cffe9daa5840abd137443d8bab0#file-flamegraph-perf-svg
> 
> Seems like there is no caching and debug info is getting reparsed
> continuously for every stack. It's possible that it was not an issue
> before, because we spent no time decompressing dwarf.

possibly, if we have some clear reproducer we can hand it
to the libdw guy that helped us develop this

> 
> 4. Pretty much all luajit frames stacks that were marked as unknown
> before are now gone.
> 
> See before and after here: https://imgur.com/a/1LNfqAk
> 
> Before:
> 
> nginx-cache 94572 446642.722028:   10101010 cpu-clock:
>             5607d8d56718 ngx_http_lua_shdict_lookup+0x48 (inlined)
>             5607d8d5a09d ngx_http_lua_ffi_shdict_incr+0xcd
> (/usr/local/nginx-cache/sbin/nginx-cache)
>             560802fe58e4 [unknown] (/tmp/perf-94572.map)
> 
> After:
> 
> nginx-cache 94572 446543.008703:   10101010 cpu-clock:
>             5607d8d56718 ngx_http_lua_shdict_lookup+0x48 (inlined)
>             5607d8d59da7 ngx_http_lua_ffi_shdict_get+0x197
> (/usr/local/nginx-cache/sbin/nginx-cache)
> 
> The key is /tmp/perf-*.map frame at the bottom. I don't know if it's
> expected, but we grew dependent on knowing this.
> 
> 5. Special [[stack]], [[heap]] and [anon] frames are also gone, and
> you can see the following during "perf script" run:
> 
> open("[stack]", O_RDONLY)               = -1 ENOENT (No such file or directory)
> open("[heap]", O_RDONLY)                = -1 ENOENT (No such file or directory)
> open("//anon", O_RDONLY)                = -1 ENOENT (No such file or directory)

strange, let's start with the reproducer and I'll check
on this if I see it

thanks,
jirka