Message-ID: <tencent_525B0DBF67E67116595BAF23@qq.com>
Date: Wed, 4 Sep 2024 15:47:25 +0800
From: "虞陆铭" <luming.yu@...ngroup.cn>
To: "Yang Jialong 杨佳龙" <jialong.yang@...ngroup.cn>, "mpe" <mpe@...erman.id.au>, "npiggin" <npiggin@...il.com>, "christophe.leroy" <christophe.leroy@...roup.eu>, "Naveen N. Rao" <naveen.n.rao@...ux.ibm.com>
Cc: "shenghui.qu" <shenghui.qu@...ngroup.cn>, "linuxppc-dev" <linuxppc-dev@...ts.ozlabs.org>, "linux-kernel" <linux-kernel@...r.kernel.org>, "Steven Rostedt" <rostedt@...dmis.org>, "Masami Hiramatsu" <mhiramat@...nel.org>, "Karol Herbst" <karolherbst@...il.com>, "Pekka Paalanen" <ppaalanen@...il.com>, "nouveau" <nouveau@...ts.freedesktop.org>
Subject: Re: [PATCH v1 1/2] powerpc/mmiotrace: Add MMIO Tracing tool for PowerPC

Hi,

the patch set and mmio trace capability helped us to track down to the root cause of  a mysterious EEH only on powernv platform
 triggered by a testpmd dpdk user mode driver (UIO) by comparing the mmio trace from UIO and native kernel driver from the same nic. 
And the problem is solved by switch to VFIO driver as backend.

the problem is the mmiotrace could not capture user mode mmio which I think it could be limitation by its current design.

so, I think we could tidy up the patch and do some feature enhancement then we can submit again for the debug value of the patch.
As it really is painful when EEH A/B record doesn't point to a clear root cause or suffient details that can help us to fix  driver code for the unaligned
MMIO for a 3rd party nic vendor on powernv platform.
 
Cheers,
Luming
------------------ Original ------------------
From:  "Yang Jialong 杨佳龙"<jialong.yang@...ngroup.cn>;
Date:  Fri, Jun 28, 2024 04:21 PM
To:  "mpe"<mpe@...erman.id.au>; "npiggin"<npiggin@...il.com>; "christophe.leroy"<christophe.leroy@...roup.eu>; "Naveen N. Rao"<naveen.n.rao@...ux.ibm.com>; 
Cc:  "虞陆铭"<luming.yu@...ngroup.cn>; "shenghui.qu"<shenghui.qu@...ngroup.cn>; "linuxppc-dev"<linuxppc-dev@...ts.ozlabs.org>; "linux-kernel"<linux-kernel@...r.kernel.org>; "Steven Rostedt"<rostedt@...dmis.org>; "Masami Hiramatsu"<mhiramat@...nel.org>; "Karol Herbst"<karolherbst@...il.com>; "Pekka Paalanen"<ppaalanen@...il.com>; "nouveau"<nouveau@...ts.freedesktop.org>; 
Subject:  Re: [PATCH v1 1/2] powerpc/mmiotrace: Add MMIO Tracing tool for PowerPC

 


在 2024/6/28 15:02, Michael Ellerman 写道:
> Jialong Yang <jialong.yang@...ngroup.cn> writes:
>> mmiotrace is a useful tool to trace MMIO accesses. Nowadays, it only
>> supported on x86 and x86_64 platforms.
> I've never used mmiotrace, and don't know it well.
>
> I'm not necessarily opposed to merging it, but AFAIK it was mostly used
> for reverse engineering proprietary drivers, where the driver itself
> couldn't be easily instrumented. Is that what you're using it for?

Yes. Just like you think. We have used it for network stack debug in 
ppc64le.


>
> For drivers where we have the source wouldn't it be easier to just use
> tracepoints in the MMIO accessors?


Tracepoints need pre-defined. And in some big driver, it's not easy to 
overwrite

all points where access registers in io area. And tracepoint is C 
function level filter.

mmiotrace is similar to set tracepoints in writel/readl... But it can do 
deeperly.

mmiotrace is a asm level filter tool. It doesn't care what have done in 
C level. It will

only find what have done by asm, such as stw(store word)/lw(load word),  
just like standing

in the view of device.


>
> Is it still in-use/maintained on the x86 side?


Here is some core file patches number in x86:

|      | mmio_mod.c | kmmio.c | pf_in.* | testmmiotrace.c |
|------+------------+---------+---------+-----------------|
| 2022 |               1 |            3 | |                  |
| 2021 |               2 |            1 |               |         |
| 2020 |               4 |            4 | |               1 |
| 2019 |               2 |            1 |            1 |               4 |
| 2018 |                  |            2 |               |          |
| 2017 |               2 |            2 | |               1 |
| 2016 |               1 |            2 |            1 |        |
| 2014 |                  |            1 |               |          |
| 2013 |               1 |               |               |          |
| 2012 |               1 |               | |                  |
| 2011 |               3 |               |            1 |         |
| 2010 |               1 |            3 |            2 |               1 |
| 2009 |               4 |          19 | |               3 |
| 2008 |             13 |            5 |            2 |               3 |

>
>> Here is a support for powerpc.
>> The manual is located at Documentation/trace/mmiotrace.rst which means
>> I have not changed user API. People will be easy to use it.
>> Almost all files are copied from x86/mm, there are only some
>> differences from hardware and architectures software.
>>
>> LINK: https://lore.kernel.org/lkml/20080127195536.50809974@daedalus.pq.iki.fi/
>>
>> Signed-off-by: Jialong Yang <jialong.yang@...ngroup.cn>
>> ---
>>   arch/powerpc/Kconfig.debug       |   3 +
>>   arch/powerpc/mm/Makefile         |   1 +
>>   arch/powerpc/mm/kmmio.c          | 649 +++++++++++++++++++++++++++++++
>>   arch/powerpc/mm/mmio-mod.c       | 414 ++++++++++++++++++++
>>   arch/powerpc/mm/mmiotrace_arch.c | 149 +++++++
>>   arch/powerpc/mm/mmiotrace_arch.h |  25 ++
>>   arch/powerpc/mm/pf_in.c          | 185 +++++++++
>>   arch/powerpc/mm/pf_in.h          |  33 ++
>>   8 files changed, 1459 insertions(+)
>    
> At a glance most of that code could be shared between arches. I don't
> think I can merge that as-is, without some attempt to split the generic
> parts out.


Right.

I just copy them from arch/x86/mm. There are many code not arch specific.


> cheers
>

Powered by blists - more mailing lists