linux-kernel - Re: [perf] perf probe fails sometimes on 4.9

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <20180129231723.524a8c32e1456358f7cb88ce@kernel.org>
Date:   Mon, 29 Jan 2018 23:17:23 +0900
From:   Masami Hiramatsu <mhiramat@...nel.org>
To:     Pintu Kumar <pintu.ping@...il.com>
Cc:     "Wangnan (F)" <wangnan0@...wei.com>, linux-kernel@...r.kernel.org,
        acme@...hat.com, mhiramat@...nel.org, quentin.monnet@...ronome.com,
        daniel@...earbox.net, davem@...emloft.net,
        alexander.shishkin@...ux.intel.com, mingo@...hat.com,
        huawei.libin@...wei.com, alexis.berlemont@...il.com,
        maddy@...ux.vnet.ibm.com, naveen.n.rao@...ux.vnet.ibm.com,
        peterz@...radead.org, kernelnewbies@...nelnewbies.org
Subject: Re: [perf] perf probe fails sometimes on 4.9

On Mon, 29 Jan 2018 13:40:34 +0530
Pintu Kumar <pintu.ping@...il.com> wrote:

> Hi All,
> 
> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> # perf probe --add schedule
> schedule is out of .text, skip it.
>   Error: Failed to add events.
> 
> If any one have come across this problem please let me know the cause.

Hi Pintu,

Could you run it with --vv?

> 
> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?

Without it, you can not probe source-level probe nor trace local variable.

> However, I manually copied the vmlinux file to /boot/ directory, but
> still it does not work.

That doesn't work.
CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
Without that option, debuginfo is not generated with vmlinux.

> 
> I checked upstream patches until 4.15 but could not find any clue.
> Please let me know if there is any fixes available for this.

Could you also ensure that you run perf by root user?

Thank you,


> 
> 
> Thank You!
> Regards,
> Pintu
> 
> 
> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
> > Hi,
> >
> > ** Changed the subject now, since these issues are related to general
> > perf commands.
> >
> > Following are the issues:
> >
> > 1) perf probe --add schedule - FAILED
> > output:
> > schedule is out of .text, skip it.
> >   Error: Failed to add events.
> >
> > what is the issue here?
> > Sometimes it pass and sometimes it fails...
> > Similar is the case of 'perf inject' as well.
> >
> > 2) perf test - 1 FAILURE
> > 37.1: Test basic BPF filtering                               : FAILED!
> > 37.2: Test BPF prologue generation                           : Skip
> > 37.3: Test BPF relocation checker                            : Skip
> >
> > bpf: config program 'func=SyS_epoll_wait'
> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > bpf: config 'func=SyS_epoll_wait' is ok
> > Looking at the vmlinux_path (8 entries long)
> > Using /boot/vmlinux for symbols
> > Could not open debuginfo. Try to use symbols.
> > SyS_epoll_wait is out of .text, skip it.
> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > selected by BPF
> > test child finished with -1
> > ---- end ----
> > Test BPF filter subtest 0: FAILED!
> >
> > Looks like both 1,2 are related.
> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > to /boot/ folder.
> >
> > -------
> > Some more info:
> >
> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > unset env: KBUILD_OPTS
> > include option is set to  -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: NR_CPUS=8
> > set env: LINUX_VERSION_CODE=0x40914
> > set env: CLANG_EXEC=/usr/bin/clang
> > set env: CLANG_OPTIONS=-xc
> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >
> >
> > If you have any clue about these failure please help me.
> >
> >
> > Thanks,
> > Pintu
> >
> >
> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
> >> Hi,
> >>
> >> Thanks for your help.
> >> Yes it was a sub version issue.
> >>
> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >>
> >> make -j8 bindeb-pkg SUBLEVEL=20
> >>
> >> After passing the correct sublevel now the test is working.
> >>
> >> But still following are failing:
> >>
> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> >> 37.2: Test BPF prologue generation                           : FAILED!
> >>
> >>
> >> This is the error I get:
> >>
> >> 16: Try 'import perf' in python, checking link problems      :
> >> --- start ---
> >> test child forked, pid 7637
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> ImportError: No module named perf
> >> test child finished with -1
> >> ---- end ----
> >> Try 'import perf' in python, checking link problems: FAILED!
> >>
> >>
> >> Looking at the vmlinux_path (8 entries long)
> >> symsrc__init: cannot get elf header.
> >> Failed to find the path for kernel: Invalid ELF file
> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> selected by BPF
> >> test child finished with -1
> >> ---- end ----
> >> Test BPF filter subtest 1: FAILED!
> >>
> >>
> >>
> >> Thanks,
> >> Pintu
> >>
> >>
> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@...wei.com> wrote:
> >>>
> >>>
> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >>>> kernel version 4.9.20.
> >>>> I have installed several others packages including: clang, llvm
> >>>>
> >>>> But, when I run 'perf test' I get some FAILURE.
> >>>> Specially, 'perf test LLVM' is failing.
> >>>>
> >>>> Please check the below error logs:
> >>>>
> >>>> ----------------------------------------------------
> >>>> # perf test LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> >>>> 35.2: Test kbuild searching                                  : Skip
> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> >>>> ----------------------------------------------------
> >>>>
> >>>> When I run with -v I get this:
> >>>>
> >>>> -----------------------------------------------------
> >>>> # perf test -v LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          :
> >>>> --- start ---
> >>>> test child forked, pid 3304
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>> WARNING:        unable to get correct kernel building directory.
> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> >>>>          detection.
> >>>>
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>>
> >>>>
> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >>>> -
> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >>>> test child finished with -1
> >>>> ---- end ----
> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >>>> -----------------------------------------------------
> >>>>
> >>>>
> >>>> Basic BPF LLVM compiling test is failing.
> >>>> It reports that bpf could not able to read the kernel version from uname.
> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >>>>
> >>>> My uname says:
> >>>> 4.9-xxxx-amd-x86-64
> >>>
> >>>
> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >>> report
> >>> kernel version number (from your uname string, sublevel can't be
> >>> determined),
> >>> please help us improve that function like what we have done for ubuntu
> >>> (checking /proc/version_signature).
> >>>
> >>>
> >>>
> >>>> My kernel version is: 4.9.20 (from Makefile)
> >>>>
> >>>> So, I dont think there is any problem with kernel version.
> >>>>
> >>>> If any body have come across this problem please let me know how to
> >>>> resolve this issue.
> >>>>
> >>>>
> >>>>
> >>>> Thank You!
> >>>>
> >>>> Regards,
> >>>> Pintu
> >>>
> >>>
> >>>


-- 
Masami Hiramatsu <mhiramat@...nel.org>