linux-kernel - Re: [PATCH V2 00/10] perf script: Add API for filtering via dynamically loaded shared object

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 28 Jun 2021 07:57:03 -0700
From:   Andi Kleen <ak@...ux.intel.com>
To:     Adrian Hunter <adrian.hunter@...el.com>,
        Arnaldo Carvalho de Melo <acme@...nel.org>
Cc:     Jiri Olsa <jolsa@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Mark Rutland <mark.rutland@....com>,
        Namhyung Kim <namhyung@...nel.org>,
        Leo Yan <leo.yan@...aro.org>,
        Kan Liang <kan.liang@...ux.intel.com>,
        linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH V2 00/10] perf script: Add API for filtering via
 dynamically loaded shared object


On 6/28/2021 12:23 AM, Adrian Hunter wrote:
> On 27/06/21 7:13 pm, Andi Kleen wrote:
>> On 6/27/2021 6:18 AM, Adrian Hunter wrote:
>>> Hi In some cases, users want to filter very large amounts of data
>>> (e.g. from AUX area tracing like Intel PT) looking for something
>>> specific. While scripting such as Python can be used, Python is 10
>>> to 20 times slower than C. So define a C API so that custom filters
>>> can be written and loaded.
>> While I appreciate this for complex cases, in my experience filtering
>> is usually just a simple expression. It would be nice to also have a
>> way to do this reasonably fast without having to write a custom C
> I do not agree that writing C filters is a hassle e.g. a minimal do-nothing
> filter is only a few lines:

It still doesn't seem user friendly. Maybe it's obvious to you, but I 
suspect we left behind most of even the sophisticated perf users here.


>
>>                   Maybe we could have some kind of python fast path
>> just for filters?
> I expect there are ways to make it more efficient, but I doubt it would ever
> come close to C.

If it's within 2-3x I guess it would be ok. For any larger data files we 
should parallelize anyways, and that works fine with the --time x/y 
method (although it usually also needs some custom scripting, perhaps 
need to figure out how to make it more user friendly)


>
>> just for filters? Or maybe the alternative would be to have a
>> frontend in perf that can automatically generate/compile such a C
>> filter based on a simple expression, but I'm not sure if that would
>> be much simpler.
> If gcc is available, perf script could, in fact, build the .so on the fly
> since the compile time is very quick.
>
> Another point is that filters can be used for more than just filtering.
> Here is an example which sums cycles per-cpu and prints them, and the difference
> to the last print, at the beginning of each line.  I think this was something
> you were interested in doing?

Yes that's great and useful, but I would prefer to not maintain custom 
plugins for it. Often when I write a script it has to run in all kinds 
of weird environments that some random person installed, and it's not 
clear how portable building C will be there. And I doubt I can just copy 
the .so files around.

BTW I'm not arguing to not do the plugin (I can imagine extreme cases 
where such a plugin is the best option), but really for most of these 
things there should be easier and more portable alternatives, even if 
they are slightly slower.

-Andi