linux-kernel - Re: perf test BPF failing on f24: fix

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20160804153221.8aa29dfc5533b14089bd7470@kernel.org>
Date:	Thu, 4 Aug 2016 15:32:21 +0900
From:	Masami Hiramatsu <mhiramat@...nel.org>
To:	Arnaldo Carvalho de Melo <acme@...nel.org>
Cc:	"Wangnan (F)" <wangnan0@...wei.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: perf test BPF failing on f24: fix

On Wed, 3 Aug 2016 17:04:15 -0300
Arnaldo Carvalho de Melo <acme@...nel.org> wrote:

> Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu:
> > > If we remove vmlinux, perf should use /proc/kallsyms. I think
> 
> I am not removing vmlinux, it is being used, its just that the function
> chosen by the 'perf test BPF' testcase isn't there.
> 
> So lets go again trying to chase this without missing a single step of
> the way:
> 
> We start with:
> 
> [root@...et ~]# perf test bpf
> 37: Test BPF filter                                          :
> 37.1: Test basic BPF filtering                               : FAILED!
> 37.2: Test BPF prologue generation                           : Skip
> 37.3: Test BPF relocation checker                            : Skip
> [root@...et ~]# 
> 
> Ok, so we add -v to get more information:
> 
> [root@...et ~]# perf test -v bpf
> <BIG SNIP>
> bpf: config program 'func=sys_epoll_wait'
> symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> bpf: config 'func=sys_epoll_wait' is ok
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/4.7.0+/build/vmlinux for symbols
> Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
> Try to find probe point from debuginfo.
> Symbol sys_epoll_wait address found : ffffffffbd295b50
> Failed to find debug information for address ffffffffbd295b50
> Probe point 'sys_epoll_wait' not found.
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 0: FAILED!
> 
> --------------
> 
> See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should
> because:
> 
> [acme@...et linux]$ file /lib/modules/4.7.0+/build/vmlinux
> /lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64,
> version 1 (SYSV), statically linked,
> BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped
> [acme@...et linux]$
> 
> Its the kernel that is in use:
> 
> [acme@...et linux]$ perf buildid-list --kernel
>               a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a

> [acme@...et linux]$ perf buildid-list -h --kernel
> 
>  Usage: perf buildid-list [<options>]
> 
>     -k, --kernel          Show current kernel build id
> 
> [acme@...et linux]$
> 
> And, in this vmlinux file, there is _no_ such function:
> 
> [acme@...et linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
> [acme@...et linux]$ 
> 
> Exactly like the 'perf probe -v bpf' says:
> 
> Symbol sys_epoll_wait address found : ffffffffbd295b50
> Failed to find debug information for address ffffffffbd295b50
> 
> -----
> 
> It mapped it to an address, sure, it found it in /proc/kallsyms, but
> then it didn't find it in the matching vmlinux file.
> 
> Since the test was working before, when did it stop to be available on
> vmlinux?

Would you remember the previous test was done with this kernel or other one?

Actually, I could put probes on sys_epoll_wait on F24 standard kernel.

-----
[mhiramat@...box perf]$ sudo ./perf probe -vnf sys_epoll_wait
probe-definition(0): sys_epoll_wait 
symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols
Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_wait address found : ffffffff81292d60
Matched function: SyS_epoll_wait
found inline addr: 0xffffffff812930f3
Probe point found: compat_SyS_epoll_pwait+147
found inline addr: 0xffffffff81292ed6
Probe point found: SyS_epoll_pwait+134
found inline addr: 0xffffffff81292d60
Probe point found: SyS_epoll_wait+0
Found 3 probe_trace_events.
Opening /sys/kernel/debug/tracing//kprobe_events write=1
Writing event: p:probe/sys_epoll_wait _text+2699507
Writing event: p:probe/sys_epoll_wait_1 _text+2698966
Writing event: p:probe/sys_epoll_wait_2 _text+2698592
Added new events:
  probe:sys_epoll_wait (on sys_epoll_wait)
  probe:sys_epoll_wait_1 (on sys_epoll_wait)
  probe:sys_epoll_wait_2 (on sys_epoll_wait)

You can now use it in all perf tools, such as:

	perf record -e probe:sys_epoll_wait_2 -aR sleep 1

-----
FYI, if perf-probe failed to find the symbol in debuginfo, it tries to find its address
from symbol table(kallsyms, here), and then tries to convert the address to the symbol
in debuginfo again. It seems that in your case this convert process is failed.

Then, could you grep DEBUG_INFO in .config? I guess your kernel enables some
reduced debuginfo related config enabled...

Thank you,


-- 
Masami Hiramatsu <mhiramat@...nel.org>