Message-ID: <20250528142858.185017-1-tglozar@redhat.com>
Date: Wed, 28 May 2025 16:28:56 +0200
From: Tomas Glozar <tglozar@...hat.com>
To: Steven Rostedt <rostedt@...dmis.org>
Cc: linux-trace-kernel@...r.kernel.org,
	linux-kernel@...r.kernel.org,
	John Kacur <jkacur@...hat.com>,
	Luis Goncalves <lgoncalv@...hat.com>,
	Tomas Glozar <tglozar@...hat.com>
Subject: [RFC PATCH 0/2] rtla/timerlat: Implement flexible actions on latency threshold overflow

rtla-timerlat defines the -i and -T options to set a threshold for
latency, stopping tracing if a sample above the threshold is
encountered. After tracing is stopped, one of two optional actions can
be enabled: saving trace output and auto-analysis. Auto-analysis is
enabled by default, unless --aa-only is supplied, while trace output has
to be explicitly set with -t/--trace-output.

This is an RFC of a feature extending this to a potentionally arbitrary
action. It adds a new option, -A/--action-on-overflow, that takes
the action to be performed and optionally, arguments to that action. For
example:

$ rtla timerlat hist -T 10 -A command,exec=date

will print the date after stopping tracing. The argument also supports
trace output, via -A trace (equivalent to -t), and can be specified
multiple times. See the patch log for more information on the actions
implemented in the RFC.

The feature was inspired by a case where collecting perf data on rtla
latency overflow was required, which can be done by sending a signal
to the perf process. In general, the feature is aiming to allow
integration with external tooling.

The RFC contains two patches. The first one clearly distinguishes
between rtla-timerlat tracing modes with regards to collecting data:
BPF-only mode, tracefs-only mode, and mixed mode. It is not a hard
prerequisite of the other patch, but is expected to be useful in
the implementation of the next version of the patchset. The second one
contains the actions feature itself.

Missing functionality in this RFC expected to be in the final version:
- Allowing the tracing to continue after the action. This is expected
to be implemented via either explicitly stopping tracing with "-A stop"
if needed (and not stopping if any other action is specified), or vice
versa by explicitly continuing with "-A continue". The latter might be
better for more meaningful backwards compatibility.

An unresolved question is whether to stop tracing or not during
the execution of the actions, if the measurement is to continue.
- Specifying a trace output file, like -t does, e.g. "-A trace,file=a.txt".
- Converting auto-analysis into an action of this kind.
- Multiple actions of same kind in one run, e.g.:
"-A signal,pid=2,num=2 -A signal,pid=3,num=2"
- Properly handling errors when adding actions.
- Providing the latency to the exec action command via an environmental
variable.
- Plus there are a few minor checkpatch issues.

Suggestions (e.g. with respect to your own use cases) are most welcome.

Tomas Glozar (2):
  rtla/timerlat: Introduce enum timerlat_tracing_mode
  rtla/timerlat: Add action on threshold

 tools/tracing/rtla/src/Build           |   1 +
 tools/tracing/rtla/src/actions.c       | 155 +++++++++++++++++++++++++
 tools/tracing/rtla/src/actions.h       |  33 ++++++
 tools/tracing/rtla/src/timerlat.c      |  66 +++++++++--
 tools/tracing/rtla/src/timerlat.h      |  22 +++-
 tools/tracing/rtla/src/timerlat_hist.c |  83 ++++++++-----
 tools/tracing/rtla/src/timerlat_top.c  |  89 ++++++++------
 7 files changed, 375 insertions(+), 74 deletions(-)
 create mode 100644 tools/tracing/rtla/src/actions.c
 create mode 100644 tools/tracing/rtla/src/actions.h

-- 
2.49.0

Powered by blists - more mailing lists