linux-kernel - Re: [RFC PATCH 0/3] Introduce a C extension module to allow libperf usage from python

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAP-5=fWjca+SMBdsyPe8Ggsqrqz=ZzWqxDYhaF+pcYk0MJ12zA@mail.gmail.com>
Date: Thu, 13 Mar 2025 09:01:10 -0700
From: Ian Rogers <irogers@...gle.com>
To: "John B. Wyatt IV" <jwyatt@...hat.com>
Cc: Gautam Menghani <gautam@...ux.ibm.com>, peterz@...radead.org, mingo@...hat.com, 
	acme@...nel.org, namhyung@...nel.org, mark.rutland@....com, 
	alexander.shishkin@...ux.intel.com, jolsa@...nel.org, adrian.hunter@...el.com, 
	kan.liang@...ux.intel.com, linux-perf-users@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Subject: Re: [RFC PATCH 0/3] Introduce a C extension module to allow libperf
 usage from python

On Thu, Mar 13, 2025 at 7:12 AM John B. Wyatt IV <jwyatt@...hat.com> wrote:
>
> Hello Gautam
>
> On Thu, Mar 13, 2025 at 01:21:21PM +0530, Gautam Menghani wrote:
> > In this RFC series, we are introducing a C extension module to allow
> > python programs to call the libperf API functions. Currently libperf can
> > be used by C programs, but expanding the support to python is beneficial
> > for python users.
> >
> > The structure of the patch series is as follows:
> > 1. Patch 1 : Create wrappers for the perf structs which are used by
> > examples/counting.c
> >
> > 2. Patch 2: Create the C extension module that maps and exposes the
> > libperf functions to python programs
>
> May I ask why you are not using SWIG? With libcpupower the kernel has
> already been using SWIG to generate Python bindings for a C user-space API.
>
> This has several advantages including a much smaller footprint (you only
> need to copy the header definitions into a .swg file), can generate for
> several languages (Perl, Ruby, Java and C#), and SWIG is an active,
> tested, and mature piece of software code that has been around for
> almost as long as the Linux kernel.
>
> Python bindings including the makefile as an example:
> https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python
>
> How to use the bindings in a script:
> https://elixir.bootlin.com/linux/v6.13.6/source/tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py
>
> Original discussion:
> https://lore.kernel.org/linux-pm/20240724221122.54601-1-jwyatt@redhat.com/
>
> SWIG has been pretty useful as it helped me find two issues in the
> libcpupower API that have been around for over 10 years:
> https://lore.kernel.org/linux-pm/20240905021916.15938-1-jwyatt@redhat.com/T/#mf04b4ba93f79fe68c20c1d88d8ed966164a1c7d7
> https://lore.kernel.org/linux-pm/20250305210901.24177-1-jwyatt@redhat.com/

So I think we should probably get rid of libperf and re-integrate it
back into the perf code. There are issues in the code, like removing a
list element:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=tmp.perf-tools-next#n58
just leaks the removed element. Deleting the element means that the
container evsel, rather than the libperf perf_evsel, leaks things like
the name variable. We can add yet more call backs and complexity but
I'm not sure what we're winning. Perhaps we can move things the other
way, perf code into libperf, like machine, session, pmus, .. I'd
prefer if we were to do that we refactored the code and adopted the
same license as libbpf as both libraries have similar packaging
issues. The viral GPLv2 on libperf is something of an issue. Perhaps
we can also migrate this code to Rust.

SWIG is fine, there is also CLIF, I'm way of dependencies as even a
python dependency in the perf build is optional.

We already have perf python bindings, and I've been working to extend
those for example in:
https://lore.kernel.org/lkml/20250228222308.626803-1-irogers@google.com/
and I've been working to expand those for things like hybrid CPUs. It
seems a shame to reinvent all of that logic again on top of libperf.

Thanks,
Ian

> >
> > 2. Patch 3: A python variant of counting.c - counting.py to demonstrate
> > the usage of libperf from python
> >
> > We have not added support for entire libperf, as we want to get
> > community feedback on the approach taken in this series.
> >
> > Gautam Menghani (3):
> >   libperf: Introduce wrappers for perf structs to be exposed to python
> >   libperf: Introduce a C extension module for python
> >   libperf: Add counting.py example to demonstrate libperf usage from
> >     python
> >
> >  tools/lib/perf/Build                          |   1 +
> >  .../perf/Documentation/examples/counting.py   |  74 +++
> >  tools/lib/perf/Makefile                       |  12 +-
> >  tools/lib/perf/include/perf/py_perf.h         | 431 ++++++++++++++++++
> >  tools/lib/perf/libperf.map                    |   1 +
> >  tools/lib/perf/py_perf.c                      | 262 +++++++++++
> >  6 files changed, 779 insertions(+), 2 deletions(-)
> >  create mode 100755 tools/lib/perf/Documentation/examples/counting.py
> >  create mode 100644 tools/lib/perf/include/perf/py_perf.h
> >  create mode 100644 tools/lib/perf/py_perf.c
> >
> > --
> > 2.47.0
> >
>
> --
> Sincerely,
> John Wyatt
> Software Engineer, Core Kernel
> Red Hat
>