Date:	Wed, 3 Aug 2016 17:04:15 -0300
From:	Arnaldo Carvalho de Melo <acme@...nel.org>
To:	Masami Hiramatsu <mhiramat@...nel.org>
Cc:	"Wangnan (F)" <wangnan0@...wei.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: perf test BPF failing on f24: fix

Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu:
> > If we remove vmlinux, perf should use /proc/kallsyms. I think

I am not removing vmlinux, it is being used, its just that the function
chosen by the 'perf test BPF' testcase isn't there.

So lets go again trying to chase this without missing a single step of
the way:

We start with:

[root@...et ~]# perf test bpf
37: Test BPF filter                                          :
37.1: Test basic BPF filtering                               : FAILED!
37.2: Test BPF prologue generation                           : Skip
37.3: Test BPF relocation checker                            : Skip
[root@...et ~]# 

Ok, so we add -v to get more information:

[root@...et ~]# perf test -v bpf
<BIG SNIP>
bpf: config program 'func=sys_epoll_wait'
symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
bpf: config 'func=sys_epoll_wait' is ok
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_wait address found : ffffffffbd295b50
Failed to find debug information for address ffffffffbd295b50
Probe point 'sys_epoll_wait' not found.
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 0: FAILED!

--------------

See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should
because:

[acme@...et linux]$ file /lib/modules/4.7.0+/build/vmlinux
/lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64,
version 1 (SYSV), statically linked,
BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped
[acme@...et linux]$

Its the kernel that is in use:

[acme@...et linux]$ perf buildid-list --kernel
              a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a
[acme@...et linux]$ perf buildid-list -h --kernel

 Usage: perf buildid-list [<options>]

    -k, --kernel          Show current kernel build id

[acme@...et linux]$

And, in this vmlinux file, there is _no_ such function:

[acme@...et linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
[acme@...et linux]$ 

Exactly like the 'perf probe -v bpf' says:

Symbol sys_epoll_wait address found : ffffffffbd295b50
Failed to find debug information for address ffffffffbd295b50

-----

It mapped it to an address, sure, it found it in /proc/kallsyms, but
then it didn't find it in the matching vmlinux file.

Since the test was working before, when did it stop to be available on
vmlinux?

Looking at a distro kernel vmlinux file, that comes in the
kernel-debuginfo package...

[acme@...et linux]$ readelf -wi /usr/lib/debug/usr/lib/modules/4.6.3-300.fc24.x86_64/vmlinux | grep -w sys_epoll_wait
[acme@...et linux]$
[acme@...et linux]$ readelf -wi /usr/lib/debug/usr/lib/modules/4.6.3-300.fc24.x86_64/vmlinux | grep -w SyS_epoll_wait
    <2cb9655>   DW_AT_name        : (indirect string, offset: 0xe0d53): SyS_epoll_wait
[acme@...et linux]$

So the situation is the same, i.e. it seems that we were doing the fallback
from vmlinux to kallsyms and this somehow regressed, so now lets do a tools
bisect:

Ok, I tried this on a RHEL7 machine and got this:

Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.7.0-rc3+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_pwait address found : ffffffff8128b4a0
Matched function: SyS_epoll_pwait
Probe point found: SyS_epoll_pwait+0

So it seems that sys_epoll_wait (of sys_epoll_pwait) was never in the debuginfo:

But tooling was finding it in kallsyms, getting its address, then being able to
"reverse lookup" it to the SyS_ alias, and now this isn't happening :-\

[root@...icio ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@...icio ~]# gcc --version | head -1
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
[root@...icio ~]#
[root@...icio ~]# readelf -wi /lib/modules/4.7.0-rc3+/build/vmlinux | grep -wi sys_epoll_pwait
    <23e4444>   DW_AT_name        : (indirect string, offset: 0xe089f): SyS_epoll_pwait	
[root@...icio ~]# 

So I'll concentrate on this path of investigation...

- Arnaldo

> > the failure would gone. The problem is: when symbol searching
> > fail using vmlinux, should we fallback to kallsyms? However,
> > this is another question.
> 
> As below result shown, perf probe tries to fallback, but it seems to
> fail to find corresponding debuginfo entry...
> 
> > > [root@...et ~]# perf probe sys_epoll_wait
> > > Failed to find debug information for address ffffffffbd295b50
> > > Probe point 'sys_epoll_wait' not found.
> > >    Error: Failed to add events.
> 
> So, at least I should investigate it.
> 
> Thanks,
> 
> > > [root@...et ~]# perf probe SyS_epoll_wait
> > > Added new events:
> > >    probe:SyS_epoll_wait (on SyS_epoll_wait)
> > >    probe:SyS_epoll_wait_1 (on SyS_epoll_wait)
> > >    probe:SyS_epoll_wait_2 (on SyS_epoll_wait)
> > >
> > > You can now use it in all perf tools, such as:
> > >
> > > 	perf record -e probe:SyS_epoll_wait_2 -aR sleep 1
> > >
> > > [root@...et ~]#
> > >
> > > So I am changing the BPF perf test to use the CamelCase notation alias:
> > 
> > Changing to SyS_xxx is okay, byt we still need to know the root
> > cause.
> > 
> > Thank you.
> > 
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@...nel.org>

Powered by blists - more mailing lists