Message-ID: <CAOuPNLgnazHNKoMozZq7Ef=rEwRG=rwktsnp8jCKCD0gMH+hbg@mail.gmail.com>
Date:   Mon, 5 Feb 2018 15:17:41 +0530
From:   Pintu Kumar <pintu.ping@...il.com>
To:     Masami Hiramatsu <mhiramat@...nel.org>
Cc:     linux-kernel@...r.kernel.org, "Wangnan (F)" <wangnan0@...wei.com>,
        acme@...hat.com, quentin.monnet@...ronome.com,
        daniel@...earbox.net, davem@...emloft.net,
        alexander.shishkin@...ux.intel.com, mingo@...hat.com,
        huawei.libin@...wei.com,
        Alexis Berlemont <alexis.berlemont@...il.com>,
        maddy@...ux.vnet.ibm.com, naveen.n.rao@...ux.vnet.ibm.com,
        peterz@...radead.org, kernelnewbies@...nelnewbies.org
Subject: Re: [perf] perf test BPF fails on 4.9.20

On Sat, Feb 3, 2018 at 6:49 PM, Masami Hiramatsu <mhiramat@...nel.org> wrote:
> Hi Pintu,
>
> On Fri, 2 Feb 2018 19:03:30 +0530
> Pintu Kumar <pintu.ping@...il.com> wrote:
>
>> Hi All,
>>
>> I have fixed both the issue with perf test.
>>
>> 1) 16: Try 'import perf' in python, checking link problems      : FAILED!
>> To fix this issue, I have to install:
>> pip install --upgrade pip
>> pip2.7 install perf
>>
>> 2) 37.2: Test BPF prologue generation                           : FAILED!
>> To fix this issue I had to enable full version of CONFIG_DEBUG_INFO in kernel.
>>
>> Now, my question is:
>> Is it possible to pass BPF test without enabling CONFIG_DEBUG_INFO in kernel ?
>
> It depends on what BPF test does, but it seems to access local vars. This means
> you need to enable CONFIG_DEBUG_INFO. With debuginfo we can find the assignment
> of register/stack for each local variable.
>

OK. Thank you so much for your reply.
I think we don't need BPF test at this time.
Is it possible to disable/skip it ?


> Thank you,
>>
>>
>> Thanks,
>> Pintu
>>
>>
>> On Fri, Feb 2, 2018 at 1:25 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
>> > Hi,
>> >
>> > perf test bpf prologue generation is failing.
>> > 37.2: Test BPF prologue generation                           : FAILED!
>> >
>> > Try to find probe point from debuginfo.
>> > Matched function: null_lseek [105be32]
>> > Probe point found: null_lseek+0
>> > Searching 'file' variable in context.
>> > Converting variable file into trace event.
>> > converting f_mode in file
>> > file(type:file) has no member f_mode.
>> > An error occurred in debuginfo analysis (-22).
>> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > selected by BPF
>> > test child finished with -1
>> > ---- end ----
>> > Test BPF filter subtest 1: FAILED!
>> >
>> >
>> > Is there any fix available for this issue?
>> > I searched 4.15, but could not relate any of the patches to this.
>> >
>> >
>> > Thanks,
>> > Pintu
>> >
>> >
>> >
>> > On Thu, Feb 1, 2018 at 7:34 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
>> >> Hi,
>> >>
>> >> After enabling DEBUG_INFO in kernel I still get this error for BPF test.
>> >> Please help.
>> >>
>> >> # perf test BPF -v
>> >> .....
>> >> Looking at the vmlinux_path (8 entries long)
>> >> Using /usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
>> >> for symbols
>> >> Open Debuginfo file:
>> >> /usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
>> >> Try to find probe point from debuginfo.
>> >> Matched function: null_lseek [105be32]
>> >> Probe point found: null_lseek+0
>> >> Searching 'file' variable in context.
>> >> Converting variable file into trace event.
>> >> converting f_mode in file
>> >> file(type:file) has no member f_mode.
>> >> An error occurred in debuginfo analysis (-22).
>> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >> selected by BPF
>> >> test child finished with -1
>> >> ---- end ----
>> >> Test BPF filter subtest 1: FAILED!
>> >>
>> >>
>> >>
>> >> On Thu, Feb 1, 2018 at 10:50 AM, Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>> Dear Masami,
>> >>>
>> >>> Now I am stuck again with 'perf test' failure on 4.9
>> >>>
>> >>> # perf --version
>> >>> perf version 4.9.20-<product name>
>> >>>
>> >>> # perf test
>> >>> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> >>> 37.2: Test BPF prologue generation                           : FAILED!
>> >>>
>> >>> If you have any clue about these failure please hep me.
>> >>>
>> >>> Here are the verbose output:
>> >>> -----------------------------------------------------------------------------
>> >>> 1) # perf test python -v
>> >>> 16: Try 'import perf' in python, checking link problems      :
>> >>> --- start ---
>> >>> test child forked, pid 24562
>> >>> Traceback (most recent call last):
>> >>>   File "<stdin>", line 1, in <module>
>> >>> ImportError: No module named perf
>> >>> test child finished with -1
>> >>> ---- end ----
>> >>> Try 'import perf' in python, checking link problems: FAILED!
>> >>> ------------------------------------------------------------------------------
>> >>>
>> >>> 2) # perf test BPF -v
>> >>> -------------------------------------------------------------------------------
>> >>> .....
>> >>> bpf: config 'func=null_lseek file->f_mode offset orig' is ok
>> >>> Looking at the vmlinux_path (8 entries long)
>> >>> symsrc__init: cannot get elf header.
>> >>> Failed to find the path for kernel: Invalid ELF file
>> >>> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >>> selected by BPF
>> >>> test child finished with -1
>> >>> ---- end ----
>> >>> Test BPF filter subtest 1: FAILED!
>> >>>
>> >>> -------------------------------------------------------------------------------
>> >>>
>> >>>
>> >>> Thanks,
>> >>> Pintu
>> >>>
>> >>>
>> >>> On Wed, Jan 31, 2018 at 9:01 AM, Masami Hiramatsu <mhiramat@...nel.org> wrote:
>> >>>> On Tue, 30 Jan 2018 19:20:36 +0530
>> >>>> Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>>>
>> >>>>> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@...nel.org> wrote:
>> >>>>> >
>> >>>>> > On Mon, 29 Jan 2018 22:00:52 +0530
>> >>>>> > Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>>>> >
>> >>>>> > > Dear Masami,
>> >>>>> > >
>> >>>>> > > Thank you so much for your reply.
>> >>>>> > > Please find some of my answers inline.
>> >>>>> > >
>> >>>>> > >
>> >>>>> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@...nel.org> wrote:
>> >>>>> > > > On Mon, 29 Jan 2018 13:40:34 +0530
>> >>>>> > > > Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>>>> > > >
>> >>>>> > > >> Hi All,
>> >>>>> > > >>
>> >>>>> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> >>>>> > > >> # perf probe --add schedule
>> >>>>> > > >> schedule is out of .text, skip it.
>> >>>>> > > >>   Error: Failed to add events.
>> >>>>> > > >>
>> >>>>> > > >> If any one have come across this problem please let me know the cause.
>> >>>>> > > >
>> >>>>> > > > Hi Pintu,
>> >>>>> > > >
>> >>>>> > > > Could you run it with --vv?
>> >>>>> > > >
>> >>>>> > > Ok, I will send verbose output by tomorrow.
>> >>>>> > >
>> >>>>> > > >>
>> >>>>> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>> >>>>> > > >
>> >>>>> > > > Without it, you can not probe source-level probe nor trace local variable.
>> >>>>> > > >
>> >>>>> > >
>> >>>>> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
>> >>>>> > > However, I will try to manually include "-g" option during compilation.
>> >>>>> > >
>> >>>>> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
>> >>>>> > > >> still it does not work.
>> >>>>> > > >
>> >>>>> > > > That doesn't work.
>> >>>>> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
>> >>>>> > > > Without that option, debuginfo is not generated with vmlinux.
>> >>>>> > > >
>> >>>>> > > >>
>> >>>>> > > >> I checked upstream patches until 4.15 but could not find any clue.
>> >>>>> > > >> Please let me know if there is any fixes available for this.
>> >>>>> > > >
>> >>>>> > > > Could you also ensure that you run perf by root user?
>> >>>>> > > >
>> >>>>> > >
>> >>>>> > > Yes I am running with root user.
>> >>>>> > >
>> >>>>> > > My concern is sometimes it works but sometimes it fails.
>> >>>>> >
>> >>>>> > What I thought was that your kernel enables kptr_strict(but maybe not.)
>> >>>>> > Can you also try to find "schedule" and "_etext" functions in
>> >>>>> > /proc/kallsyms?
>> >>>>> >
>> >>>>>
>> >>>>> Yes these functions are present in kallsyms.
>> >>>>>
>> >>>>> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
>> >>>>> ffffffff829328e0 T schedule
>> >>>>> ffffffff8293e751 T _etext
>> >>>>>
>> >>>>>
>> >>>>> > > I still needs to figure out, in which condition it works and which
>> >>>>> > > condition it fails.
>> >>>>> >
>> >>>>> > Yeah, it is important.
>> >>>>> >
>> >>>>> > > Usually I noticed that in fresh reboot case it works.
>> >>>>> >
>> >>>>> > So, after a while, it doesn't work again? If so, it sounds like a daemon
>> >>>>> > process changes settings in background.
>> >>>>> >
>> >>>>>
>> >>>>> Fortunately or unfortunately, now it is passing every time.
>> >>>>> Still not sure about the cause.
>> >>>>> If it fails again, I will collect info with -v and report again.
>> >>>>> Any specific info that I need to check when it fails ?
>> >>>>
>> >>>> OK, if you hit it again, please run perf-probe with -vvv option
>> >>>> (-v is enough usually, if it is unknown bug, -vvv will be better)
>> >>>>
>> >>>> Thank you,
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> Thank You,
>> >>>>> Pintu
>> >>>>>
>> >>>>>
>> >>>>> > Thank you,
>> >>>>> >
>> >>>>> > >
>> >>>>> > >
>> >>>>> > > > Thank you,
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > > >>
>> >>>>> > > >>
>> >>>>> > > >> Thank You!
>> >>>>> > > >> Regards,
>> >>>>> > > >> Pintu
>> >>>>> > > >>
>> >>>>> > > >>
>> >>>>> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>>>> > > >> > Hi,
>> >>>>> > > >> >
>> >>>>> > > >> > ** Changed the subject now, since these issues are related to general
>> >>>>> > > >> > perf commands.
>> >>>>> > > >> >
>> >>>>> > > >> > Following are the issues:
>> >>>>> > > >> >
>> >>>>> > > >> > 1) perf probe --add schedule - FAILED
>> >>>>> > > >> > output:
>> >>>>> > > >> > schedule is out of .text, skip it.
>> >>>>> > > >> >   Error: Failed to add events.
>> >>>>> > > >> >
>> >>>>> > > >> > what is the issue here?
>> >>>>> > > >> > Sometimes it pass and sometimes it fails...
>> >>>>> > > >> > Similar is the case of 'perf inject' as well.
>> >>>>> > > >> >
>> >>>>> > > >> > 2) perf test - 1 FAILURE
>> >>>>> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
>> >>>>> > > >> > 37.2: Test BPF prologue generation                           : Skip
>> >>>>> > > >> > 37.3: Test BPF relocation checker                            : Skip
>> >>>>> > > >> >
>> >>>>> > > >> > bpf: config program 'func=SyS_epoll_wait'
>> >>>>> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> >>>>> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
>> >>>>> > > >> > Looking at the vmlinux_path (8 entries long)
>> >>>>> > > >> > Using /boot/vmlinux for symbols
>> >>>>> > > >> > Could not open debuginfo. Try to use symbols.
>> >>>>> > > >> > SyS_epoll_wait is out of .text, skip it.
>> >>>>> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> >>>>> > > >> > selected by BPF
>> >>>>> > > >> > test child finished with -1
>> >>>>> > > >> > ---- end ----
>> >>>>> > > >> > Test BPF filter subtest 0: FAILED!
>> >>>>> > > >> >
>> >>>>> > > >> > Looks like both 1,2 are related.
>> >>>>> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> >>>>> > > >> > to /boot/ folder.
>> >>>>> > > >> >
>> >>>>> > > >> > -------
>> >>>>> > > >> > Some more info:
>> >>>>> > > >> >
>> >>>>> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> >>>>> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> >>>>> > > >> > unset env: KBUILD_OPTS
>> >>>>> > > >> > include option is set to  -nostdinc -isystem
>> >>>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> >>>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> >>>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> >>>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> >>>>> > > >> > set env: NR_CPUS=8
>> >>>>> > > >> > set env: LINUX_VERSION_CODE=0x40914
>> >>>>> > > >> > set env: CLANG_EXEC=/usr/bin/clang
>> >>>>> > > >> > set env: CLANG_OPTIONS=-xc
>> >>>>> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> >>>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> >>>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> >>>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> >>>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> >>>>> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> >>>>> > > >> >
>> >>>>> > > >> >
>> >>>>> > > >> > If you have any clue about these failure please help me.
>> >>>>> > > >> >
>> >>>>> > > >> >
>> >>>>> > > >> > Thanks,
>> >>>>> > > >> > Pintu
>> >>>>> > > >> >
>> >>>>> > > >> >
>> >>>>> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@...il.com> wrote:
>> >>>>> > > >> >> Hi,
>> >>>>> > > >> >>
>> >>>>> > > >> >> Thanks for your help.
>> >>>>> > > >> >> Yes it was a sub version issue.
>> >>>>> > > >> >>
>> >>>>> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> >>>>> > > >> >>
>> >>>>> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
>> >>>>> > > >> >>
>> >>>>> > > >> >> After passing the correct sublevel now the test is working.
>> >>>>> > > >> >>
>> >>>>> > > >> >> But still following are failing:
>> >>>>> > > >> >>
>> >>>>> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> >>>>> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
>> >>>>> > > >> >>
>> >>>>> > > >> >>
>> >>>>> > > >> >> This is the error I get:
>> >>>>> > > >> >>
>> >>>>> > > >> >> 16: Try 'import perf' in python, checking link problems      :
>> >>>>> > > >> >> --- start ---
>> >>>>> > > >> >> test child forked, pid 7637
>> >>>>> > > >> >> Traceback (most recent call last):
>> >>>>> > > >> >>   File "<stdin>", line 1, in <module>
>> >>>>> > > >> >> ImportError: No module named perf
>> >>>>> > > >> >> test child finished with -1
>> >>>>> > > >> >> ---- end ----
>> >>>>> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
>> >>>>> > > >> >>
>> >>>>> > > >> >>
>> >>>>> > > >> >> Looking at the vmlinux_path (8 entries long)
>> >>>>> > > >> >> symsrc__init: cannot get elf header.
>> >>>>> > > >> >> Failed to find the path for kernel: Invalid ELF file
>> >>>>> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >>>>> > > >> >> selected by BPF
>> >>>>> > > >> >> test child finished with -1
>> >>>>> > > >> >> ---- end ----
>> >>>>> > > >> >> Test BPF filter subtest 1: FAILED!
>> >>>>> > > >> >>
>> >>>>> > > >> >>
>> >>>>> > > >> >>
>> >>>>> > > >> >> Thanks,
>> >>>>> > > >> >> Pintu
>> >>>>> > > >> >>
>> >>>>> > > >> >>
>> >>>>> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@...wei.com> wrote:
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Hi All,
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> >>>>> > > >> >>>> kernel version 4.9.20.
>> >>>>> > > >> >>>> I have installed several others packages including: clang, llvm
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
>> >>>>> > > >> >>>> Specially, 'perf test LLVM' is failing.
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Please check the below error logs:
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> ----------------------------------------------------
>> >>>>> > > >> >>>> # perf test LLVM
>> >>>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> >>>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>> >>>>> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
>> >>>>> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>> >>>>> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>> >>>>> > > >> >>>> ----------------------------------------------------
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> When I run with -v I get this:
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> -----------------------------------------------------
>> >>>>> > > >> >>>> # perf test -v LLVM
>> >>>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> >>>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
>> >>>>> > > >> >>>> --- start ---
>> >>>>> > > >> >>>> test child forked, pid 3304
>> >>>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>>> > > >> >>>> WARNING:        unable to get correct kernel building directory.
>> >>>>> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> >>>>> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>> >>>>> > > >> >>>>          detection.
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> >>>>> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> >>>>> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> >>>>> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> >>>>> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> >>>>> > > >> >>>> -
>> >>>>> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> >>>>> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> >>>>> > > >> >>>> test child finished with -1
>> >>>>> > > >> >>>> ---- end ----
>> >>>>> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> >>>>> > > >> >>>> -----------------------------------------------------
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Basic BPF LLVM compiling test is failing.
>> >>>>> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
>> >>>>> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> My uname says:
>> >>>>> > > >> >>>> 4.9-xxxx-amd-x86-64
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> >>>>> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> >>>>> > > >> >>> report
>> >>>>> > > >> >>> kernel version number (from your uname string, sublevel can't be
>> >>>>> > > >> >>> determined),
>> >>>>> > > >> >>> please help us improve that function like what we have done for ubuntu
>> >>>>> > > >> >>> (checking /proc/version_signature).
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> So, I dont think there is any problem with kernel version.
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> If any body have come across this problem please let me know how to
>> >>>>> > > >> >>>> resolve this issue.
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Thank You!
>> >>>>> > > >> >>>>
>> >>>>> > > >> >>>> Regards,
>> >>>>> > > >> >>>> Pintu
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >> >>>
>> >>>>> > > >
>> >>>>> > > >
>> >>>>> > > > --
>> >>>>> > > > Masami Hiramatsu <mhiramat@...nel.org>
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > Masami Hiramatsu <mhiramat@...nel.org>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Masami Hiramatsu <mhiramat@...nel.org>
>
>
> --
> Masami Hiramatsu <mhiramat@...nel.org>

Powered by blists - more mailing lists