Date:	Fri, 5 Aug 2016 12:55:04 -0300
From:	Arnaldo Carvalho de Melo <acme@...nel.org>
To:	Masami Hiramatsu <mhiramat@...nel.org>
Cc:	Arnaldo Carvalho de Melo <arnaldo.melo@...il.com>,
	"Wangnan (F)" <wangnan0@...wei.com>,
	Alexei Starovoitov <alexei.starovoitov@...il.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: perf test BPF failing on f24: fix

Em Fri, Aug 05, 2016 at 11:35:32AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Fri, Aug 05, 2016 at 06:45:50PM +0900, Masami Hiramatsu escreveu:
> > See the symbol address calcurated from symbol map, in successful case
> > the address exactly same address of SyS_epoll_wait. This indicates
> > something might wrong in the symbol map. (maybe KASLR?)
> > Could you check what happen if nokaslr is passed to your kernel?
 
> [acme@...et linux]$ grep RANDOMIZE_BASE ../build/v4.7.0+/.config
> CONFIG_RANDOMIZE_BASE=y
 
> Yes, it is in place, I'll check rebooting with nokaslr.
 
> Can you think about how to properly warn the user about this
> possibility, if it is confirmed to cause this issue?
 
> I wonder why this doesn't seem to affect annotation, that also uses vmlinux,
> for instance:
 
> [root@...et ~]# perf record --all-kernel -F 10000 ping -f -c 100000 localhost
> PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
  
> --- localhost.localdomain ping statistics ---
> 100000 packets transmitted, 100000 received, 0% packet loss, time 732ms
> rtt min/avg/max/mdev = 0.003/0.003/0.048/0.002 ms, ipg/ewma 0.007/0.003 ms
> [ perf record: Woken up 2 times to write data ]
> [ perf record: Captured and wrote 0.298 MB perf.data (7340 samples) ]
> [root@...et ~]# perf report -v | grep '\[' | head -10
> Using /lib/modules/4.7.0+/build/vmlinux for symbols
>    20.42%  ping   /lib/modules/4.7.0+/build/vmlinux  0x960374  v [k] ipt_do_table
>     2.55%  ping   /lib/modules/4.7.0+/build/vmlinux  0x900b8a  v [k] nf_iterate
>     1.94%  ping   /lib/modules/4.7.0+/build/vmlinux  0x95190b  v [k] fib_table_lookup
>     1.68%  ping   /lib/modules/4.7.0+/build/vmlinux  0x2accfa  v [k] __local_bh_enable_ip
>     1.68%  ping   /lib/modules/4.7.0+/build/vmlinux  0x9e41ab  v [k] _raw_spin_lock_irqsave
>     1.66%  ping   /lib/modules/4.7.0+/build/vmlinux  0x55fd94  v [k] avc_has_perm
>     1.65%  ping   /lib/modules/4.7.0+/build/vmlinux  0x90e3e2  v [k] __ip_append_data.isra.44
>     1.31%  ping   /lib/modules/4.7.0+/build/vmlinux  0x5ead59  v [k] copy_user_enhanced_fast_string
>     1.21%  ping   /lib/modules/4.7.0+/build/vmlinux  0x938820  v [k] raw_local_deliver
>     1.14%  ping   /lib/modules/4.7.0+/build/vmlinux  0x6c6cea  v [k] n_tty_write
> [root@...et ~]# 
 
> [root@...et ~]# perf annotate -v --stdio ipt_do_table
> Looking at the vmlinux_path (8 entries long)
> Using /lib/modules/4.7.0+/build/vmlinux for symbols
> symbol__disassemble: filename=/lib/modules/4.7.0+/build/vmlinux, sym=ipt_do_table, start=0xffffffffbd7601c0, end=0xffffffffbd7607b0
> annotating [0x3cef9b0] /lib/modules/4.7.0+/build/vmlinux : [0x4159750]                   ipt_do_table
> Executing: objdump  --start-address=0xffffffff817601c0 --stop-address=0xffffffff817607b0 -l -d --no-show-raw -S -C /lib/modules/4.7.0+/build/vmlinux 2>/dev/nu
>  Percent |      Source code & Disassembly of vmlinux for cycles:ppp (1498 samples)
> ----------------------------------------------------------------------------------
> <SNIP>
 
> [root@...et ~]# grep ffffffffbd7601c0 /proc/kallsyms 
> ffffffffbd7601c0 T ipt_do_table
> [root@...et ~]# 
 
> Anyway, will reboot this machine with nokaslr to check that possibility.

So, more info, but executive summary: distro kernels don't exhibit this
problem, but they have CONFIG_RANDOMIZE_BASE=y, the kernel where I
noticed this problem (4.7.0+) doesn't exhibit this problem when used
with 'nokaslr', but I think more investigation is needed to get to the
bottom of this, i.e. to emit meaningful messages when kaslr is in place
(or is suspected to be):

Using a distro kernel:

  [root@...et ~]# uname -r
  4.6.4-301.fc24.x86_64
  [root@...et ~]# perf probe sys_epoll_wait
  Added new event:
    probe:sys_epoll_wait (on sys_epoll_wait)

  You can now use it in all perf tools, such as:

	perf record -e probe:sys_epoll_wait -aR sleep 1

  [root@...et ~]# rpm -q kernel-debuginfo
  kernel-debuginfo-4.6.3-300.fc24.x86_64
  [root@...et ~]# uname -r
  4.6.4-301.fc24.x86_64
  [root@...et ~]# 

So it must be using just /proc/kallsyms, since there is no matching vmlinux:

  [root@...et ~]# perf probe -d *:*
  Removed event: probe:sys_epoll_wait
  [root@...et ~]# perf probe -v sys_epoll_wait
  probe-definition(0): sys_epoll_wait 
  symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
  0 arguments
  Looking at the vmlinux_path (8 entries long)
  symsrc__init: cannot get elf header.
  Could not open debuginfo. Try to use symbols.
  Looking at the vmlinux_path (8 entries long)
  symsrc__init: cannot get elf header.
  Using /proc/kcore for kernel object code
  Using /proc/kallsyms for symbols
  Opening /sys/kernel/debug/tracing//kprobe_events write=1
  Writing event: p:probe/sys_epoll_wait _text+2698592
  Added new event:
    probe:sys_epoll_wait (on sys_epoll_wait)

  You can now use it in all perf tools, such as:

	perf record -e probe:sys_epoll_wait -aR sleep 1

  [root@...et ~]#

Ok, installing a matching kernel vmlinux:

  [root@...et ~]# perf probe -d *:*
  Removed event: probe:sys_epoll_wait
  [root@...et ~]# dnf debuginfo-install kernel

Trying again:

  [root@...et ~]# perf probe -v sys_epoll_wait
  probe-definition(0): sys_epoll_wait 
  symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
  0 arguments
  Looking at the vmlinux_path (8 entries long)
  Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols
  Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux
  Try to find probe point from debuginfo.
  Symbol sys_epoll_wait address found : ffffffff81292d60
  Matched function: SyS_epoll_wait
  found inline addr: 0xffffffff812930f3
  Probe point found: compat_SyS_epoll_pwait+147
  found inline addr: 0xffffffff81292ed6
  Probe point found: SyS_epoll_pwait+134
  found inline addr: 0xffffffff81292d60
  Probe point found: SyS_epoll_wait+0
  Found 3 probe_trace_events.
  Opening /sys/kernel/debug/tracing//kprobe_events write=1
  Writing event: p:probe/sys_epoll_wait _text+2699507
  Writing event: p:probe/sys_epoll_wait_1 _text+2698966
  Writing event: p:probe/sys_epoll_wait_2 _text+2698592
  Added new events:
    probe:sys_epoll_wait (on sys_epoll_wait)
    probe:sys_epoll_wait_1 (on sys_epoll_wait)
    probe:sys_epoll_wait_2 (on sys_epoll_wait)

  You can now use it in all perf tools, such as:

	perf record -e probe:sys_epoll_wait_2 -aR sleep 1

 [root@...et ~]#

So, a distro kernel works with or without vmlinux, but then...

  [root@...et ~]# grep RANDOMIZE_BASE /boot/config-4.6.4-301.fc24.x86_64 
  CONFIG_RANDOMIZE_BASE=y
  CONFIG_RANDOMIZE_BASE_MAX_OFFSET=0x40000000
  [root@...et ~]#

It also has kaslr :-\ Rebooting with the kernel where the problem happened,
this time with nokaslr:

  [root@...et ~]# uname -r
  4.7.0+
  [root@...et ~]# cat /proc/cmdline 
  BOOT_IMAGE=/vmlinuz-4.7.0+ root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet LANG=en_US.UTF-8 nokaslr
  [root@...et ~]# perf probe -v sys_epoll_wait
  probe-definition(0): sys_epoll_wait 
  symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
  0 arguments
  Looking at the vmlinux_path (8 entries long)
  Using /lib/modules/4.7.0+/build/vmlinux for symbols
  Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
  Try to find probe point from debuginfo.
  Symbol sys_epoll_wait address found : ffffffff81295b50
  Matched function: SyS_epoll_wait
  found inline addr: 0xffffffff81295ee7
  Probe point found: compat_SyS_epoll_pwait+151
  found inline addr: 0xffffffff81295cca
  Probe point found: SyS_epoll_pwait+138
  found inline addr: 0xffffffff81295b50
  Probe point found: SyS_epoll_wait+0
  Found 3 probe_trace_events.
  Opening /sys/kernel/debug/tracing//kprobe_events write=1
  Writing event: p:probe/sys_epoll_wait _text+2711271
  Writing event: p:probe/sys_epoll_wait_1 _text+2710730
  Writing event: p:probe/sys_epoll_wait_2 _text+2710352
  Added new events:
    probe:sys_epoll_wait (on sys_epoll_wait)
    probe:sys_epoll_wait_1 (on sys_epoll_wait)
    probe:sys_epoll_wait_2 (on sys_epoll_wait)
  
  You can now use it in all perf tools, such as:
  
	  perf record -e probe:sys_epoll_wait_2 -aR sleep 1

  [root@...et ~]# 

It works, but why was annotation worked when "nokaslr" wasn't used?

- Arnaldo

Powered by blists - more mailing lists