Message-ID: <aLnAazxkczTFSMcL@x1>
Date: Thu, 4 Sep 2025 13:38:03 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Leo Yan <leo.yan@....com>
Cc: Adrian Hunter <adrian.hunter@...el.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...hat.com>, Namhyung Kim <namhyung@...nel.org>,
	Jiri Olsa <jolsa@...nel.org>, Ian Rogers <irogers@...gle.com>,
	Tomas Glozar <tglozar@...hat.com>, KP Singh <kpsingh@...nel.org>,
	Matt Bobrowski <mattbobrowski@...gle.com>,
	Song Liu <song@...nel.org>, Alexei Starovoitov <ast@...nel.org>,
	Daniel Borkmann <daniel@...earbox.net>,
	Andrii Nakryiko <andrii@...nel.org>,
	Martin KaFai Lau <martin.lau@...ux.dev>,
	Eduard Zingerman <eddyz87@...il.com>,
	Yonghong Song <yonghong.song@...ux.dev>,
	John Fastabend <john.fastabend@...il.com>,
	Stanislav Fomichev <sdf@...ichev.me>, Hao Luo <haoluo@...gle.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	Masami Hiramatsu <mhiramat@...nel.org>,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	James Clark <james.clark@...aro.org>,
	Suzuki K Poulose <suzuki.poulose@....com>,
	Mike Leach <mike.leach@...aro.org>,
	linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org,
	bpf@...r.kernel.org, linux-trace-kernel@...r.kernel.org
Subject: Re: [PATCH v3 0/6] perf auxtrace: Support AUX pause and resume with
 BPF

On Fri, Aug 08, 2025 at 12:47:34PM +0100, Leo Yan wrote:
> On Tue, Aug 05, 2025 at 10:16:29PM +0300, Adrian Hunter wrote:
> > On 30/07/2025 21:26, Leo Yan wrote:
> > > On Mon, Jul 28, 2025 at 08:02:51PM +0300, Adrian Hunter wrote:
> > >> On 25/07/2025 12:59, Leo Yan wrote:
> > >>> This series extends Perf for fine-grained tracing by using BPF program
> > >>> to pause and resume AUX tracing. The BPF program can be attached to
> > >>> tracepoints (including ftrace tracepoints and dynamic tracepoints, like
> > >>> kprobe, kretprobe, uprobe and uretprobe).

> > >> Using eBPF to pause/resume AUX tracing seems like a great idea.

> > >> AFAICT with this patch set, there is just support for pause/resume
> > >> much like what could be done directly without eBPF, so I wonder if you
> > >> could share a bit more on how you see this evolving, and what your
> > >> future plans are?

> > > IIUC, here you mean the tool can use `perf probe` to firstly create
> > > probes, then enable tracepoints as PMU event for AUX pause and resume.

> > Yes, like:

> > $ sudo perf probe 'do_sys_openat2 how->flags how->mode'
> > Added new event:
> >   probe:do_sys_openat2 (on do_sys_openat2 with flags=how->flags mode=how->mode)

> > You can now use it in all perf tools, such as:

> >         perf record -e probe:do_sys_openat2 -aR sleep 1

> > $ sudo perf probe do_sys_openat2%return
> > Added new event:
> >   probe:do_sys_openat2__return (on do_sys_openat2%return)

> > You can now use it in all perf tools, such as:

> >         perf record -e probe:do_sys_openat2__return -aR sleep 1

> > $ sudo perf record --kcore -e intel_pt/aux-action=start-paused/k -e probe:do_sys_openat2/aux-action=resume/ --filter='flags==0x98800' -e probe:do_sys_openat2__return/aux-action=pause/ -- ls

> Thanks a lot for sharing the commands. I was able to replicate them
> using CoreSight.

> Given that we can achieve the same result without using BPF, I am not
> sure how useful this series is. It may give us a base for exploring
> profiling that combines AUX trace and BPF, but I am fine with holding
> on until we have clear requirements for it.

> I would get suggestion from you and maintainers before proceeding
> further.

Maybe retrofit this for starting stopping profiling non HW tracing
sections?

We have now:

⬢ [acme@...lbx perf-tools-next]$ perf record -h switch

 Usage: perf record [<options>] [<command>]
    or: perf record [<options>] -- <command> [<options>]

        --switch-events   Record context switch events
        --switch-max-files <n>
                          Limit number of switch output generated files
        --switch-output[=<signal or size[BKMG] or time[smhd]>]
                          Switch output when receiving SIGUSR2 (signal) or cross a size or time threshold
        --switch-output-event <switch output event>
                          switch output event selector. use 'perf list' to list available events

⬢ [acme@...lbx perf-tools-next]$

That will dump a snapshot when some event takes place, but that is done
with a sideband thread, from 'man perf-record':

--switch-output-event::
Events that will cause the switch of the perf.data file, auto-selecting
--switch-output=signal, the results are similar as internally the side band
thread will also send a SIGUSR2 to the main one.

Uses the same syntax as --event, it will just not be recorded, serving only to
switch the perf.data file as soon as the --switch-output event is processed by
a separate sideband thread.

This sideband thread is also used to other purposes, like processing the
PERF_RECORD_BPF_EVENT records as they happen, asking the kernel for extra BPF
information, etc.

----------------------

And in perf-report we have:

----

--switch-on EVENT_NAME::
        Only consider events after this event is found.

        This may be interesting to measure a workload only after some initialization
        phase is over, i.e. insert a perf probe at that point and then using this
        option with that probe.

--switch-off EVENT_NAME::
        Stop considering events after this event is found.

--show-on-off-events::
        Show the --switch-on/off events too. This has no effect in 'perf report' now
        but probably we'll make the default not to show the switch-on/off events
        on the --group mode and if there is only one event besides the off/on ones,
        go straight to the histogram browser, just like 'perf report' with no events
        explicitly specified does.

----

If we had it in 'perf record' then we would have reduced perf.data
files.

I.e. we would have something like '-e {cycles,instructions}/action=start-paused/k' -e probe:do_sys_openat2/action=resume/ --filter='flags==0x98800' -e probe:do_sys_openat2__return/action=pause/

- Arnaldo

Powered by blists - more mailing lists