linux-kernel - Re: [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <aEmWyPqQw2Ly7Jlu@x1>
Date: Wed, 11 Jun 2025 11:46:32 -0300
From: Arnaldo Carvalho de Melo <acme@...nel.org>
To: Tomas Glozar <tglozar@...hat.com>
Cc: Steven Rostedt <rostedt@...dmis.org>,
	linux-trace-kernel@...r.kernel.org, linux-kernel@...r.kernel.org,
	John Kacur <jkacur@...hat.com>,
	Luis Goncalves <lgoncalv@...hat.com>, Chang Yin <cyin@...hat.com>,
	Costa Shulyupin <costa.shul@...hat.com>
Subject: Re: [PATCH 0/8] rtla/timerlat: Support actions on threshold and on
 end

On Wed, Jun 11, 2025 at 03:56:36PM +0200, Tomas Glozar wrote:
> This series adds a feature that allows to user to specify certain
> kinds of "actions" to be executed at one of two places in the rtla
> measurement: when tracing is stopped on latency threshold, and at the
> end of tracing.
 
> Two new options are added: -A/--on-threshold, and -N/--on-end, taking
> the action as an argument. For example:

I wouldn't add -A and -N, leaving just the long options, as it documents
scripts (and we should have autocomplete as well), leaving the one
letter options for things that are used super frequently, which could be
these new options, after a while, time will tell :-)

But I see that "A"ction connection, and since you show it is used
multiple times in a single command line, maybe its warranted the
one-letter option.
 
> $ rtla timerlat hist -T 10 -A shell,command="echo Threshold" \
> -N shell,command="echo Tracing stopped"
 
> will print "Threshold" if a thread latency higher than 10 microseconds
> is reached, and "Tracing stopped" always at the end.
 
> The list of possible actions is extensible and is covered in
> the commit messages. Later, a documentation patch series will be sent
> with clear explanation of every action and its syntax.

I think having the documentation together with the new options is
desirable.
 
> Notably, a special action "continue" resumes tracing. For example:
 
> $ rtla timerlat hist -T 100 -A shell,command="echo Threshold" \
> -A continue -d 10s

so --on-threshold ends up being a list of things to do when the
threshold is hit?
 
> will print "Threshold" as many times as tracing is stopped after
> thread latency reaches 100us.

> The feature was inspired by a case where collecting perf data on rtla
> latency overflow was required, which can be done by sending a signal
> to the perf process.
 
> Example of this with Intel PT aux buffer:
 
> $ perf record --cpu 0 -e intel_pt// -S -- rtla timerlat top -q -T 100 \
> -c 0 -A signal,pid=parent,num=12 -A continue
 
> In general, the feature is aiming to allow integration with external
> tooling. To implement even more flexibility, passing context to the
> shell through environmental variables, or even an entire scripting
> language with access to the rtla internals can be implemented if
> needed.

That is an interesting example of cross-tool integration using existing
mechanisms for detecting special events and asking for hardware tracing
snapshots, good stuff!

At some point we need to have this signalling to not involve userspace,
shortcircuiting the snapshot request closer to the event of interest,
inside the kernel.

- Arnaldo
 
> Tomas Glozar (8):
>   rtla/timerlat: Introduce enum timerlat_tracing_mode
>   rtla/timerlat: Add action on threshold feature
>   rtla/timerlat_bpf: Allow resuming tracing
>   rtla/timerlat: Add continue action
>   rtla/timerlat: Add action on end feature
>   rtla/tests: Check rtla output with grep
>   rtla/tests: Add tests for actions
>   rtla/tests: Limit duration to maximum of 10s
> 
>  tools/tracing/rtla/src/Build           |   1 +
>  tools/tracing/rtla/src/actions.c       | 260 +++++++++++++++++++++++++
>  tools/tracing/rtla/src/actions.h       |  52 +++++
>  tools/tracing/rtla/src/timerlat.bpf.c  |  13 +-
>  tools/tracing/rtla/src/timerlat.c      |  24 ++-
>  tools/tracing/rtla/src/timerlat.h      |  24 ++-
>  tools/tracing/rtla/src/timerlat_bpf.c  |  13 ++
>  tools/tracing/rtla/src/timerlat_bpf.h  |   3 +
>  tools/tracing/rtla/src/timerlat_hist.c | 145 ++++++++++----
>  tools/tracing/rtla/src/timerlat_top.c  | 167 ++++++++++------
>  tools/tracing/rtla/tests/engine.sh     |  21 +-
>  tools/tracing/rtla/tests/hwnoise.t     |   8 +-
>  tools/tracing/rtla/tests/osnoise.t     |   4 +-
>  tools/tracing/rtla/tests/timerlat.t    |  36 +++-
>  14 files changed, 652 insertions(+), 119 deletions(-)
>  create mode 100644 tools/tracing/rtla/src/actions.c
>  create mode 100644 tools/tracing/rtla/src/actions.h
> 
> -- 
> 2.49.0