Message-ID: <0628c50b-8ec6-4f84-a051-52674c2e9c1f@efficios.com>
Date: Fri, 19 Dec 2025 17:31:58 -0500
From: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To: lttng-dev <lttng-dev@...ts.lttng.org>
Cc: Diamon discuss <diamon-discuss@...ts.linuxfoundation.org>,
 linux-trace-users <linux-trace-users@...r.kernel.org>, lwn@....net,
 linux-kernel <linux-kernel@...r.kernel.org>
Subject: [RELEASE] LTTng 2.15.0-rc1 - Péché Mortel - Linux kernel and userspace tracer

Hi everyone,

We're excited to announce the availability of the first release
candidate of LTTng 2.15, codenamed “Péché Mortel”!

Only a few months after LTTng 2.14, this release delivers an impressive
number of features considering the short time frame, in particular:

• CTF 2 (Common Trace Format) as the default trace format.
• Reduced memory footprint for user space tracing.
• Improved resiliency of shared LTTng-UST buffers.
• Redesigned lttng-list(1) command with memory usage reporting.
• Extended automation capabilities for the lttng-add-trigger(1) command.

Read on for the details.

WHAT'S NEW IN LTTng 2.15?
━━━━━━━━━━━━━━━━━━━━━━━━━
New features and changes in LTTng 2.15:

• Choose the trace format for each recording session with the new
   `--trace-format` option of the lttng-create(1) command:

       $ lttng create my-session --trace-format=ctf-1.8

   Select between CTF 1.8 [1] and CTF 2 [2].

   CTF 2 now is the default trace format for all newly created recording
   sessions. Babeltrace 2 [3] supports CTF 2 since version 2.1 (January
   2025). To keep the default set to CTF 1.8 for your application, use
   the new `--default-trace-format` option of lttng-sessiond(8).

   The LTTng project adopts CTF 2 because it has many benefits over
   CTF 1.8:

   Easier decoding:
       CTF 1.8 requires parsing TSDL metadata, a CTF-specific language,
       which increases implementation time and limits tool integration.

       CTF 2 replaces this with JSON Text Sequences, readable by standard
       JSON parsers, significantly reducing new decoding code.

   Cleaner trace streaming:
       CTF 1.8 allows partial metadata items, forcing parsers to detect
       item boundaries manually, increasing complexity and bugs.

       CTF 2 separates metadata fragments with an RS byte, enabling
       simple chunk identification without decoding.

   Better trace readability (not in LTTng yet):
       CTF 1.8 lacks semantic annotations, so readers rely on external
       domain knowledge to interpret event record data.

       CTF 2 introduces user-defined attributes to describe semantics,
       enabling automated analysis (recognizing IP addresses,
       for example).

   Broader type support (not in LTTng yet):
       CTF 2 adds native field classes that can improve readability,
       performance, and trace size, unlike CTF 1.8's more limited types.

       Examples include a BLOB type for binary data, richer string
       encodings, bit maps, optionals, and LEB128 integers.

   This per-recording session configuration replaces the mechanism
   introduced in LTTng 2.14 where setting the
   `LTTNG_EXPERIMENTAL_FORCE_CTF_2` environment variable to `1` at
   `lttng-sessiond` startup forced _all_ recording sessions to produce
   CTF 2 traces.

• A new channel memory reclaim operation, exclusively available for user
   space channels, can free memory used by eligible sub-buffers by
   deallocating their backing memory when no longer needed.

   The two ways to reclaim memory are:

   Automatic reclaim:
       Configure a per-channel, automatic reclaim policy which
       periodically tries to reclaim memory for sub-buffers older than a
       given age, or as soon as they're consumed, with the new
       `--auto-reclaim-memory` option of the
       lttng-enable-channel(1) command.

       This new option is only available in discard mode (without the
       `--overwrite` option).

   Immediate reclaim:
       The new lttng-reclaim-memory(1) command reclaims memory
       immediately for eligible sub-buffers of one or more channels.

       Only reclaim the sub-buffers older than a given age with the
       `--older-than` option:

           $ lttng reclaim-memory --older-than=2m

   Both memory reclaim modes are possible because channel ring buffers
   are now backed by sparse files [4] when the file system supports them.

• Choose the buffer _preallocation_ policy of a user space channel with
   the new `--buffer-preallocation` option of the
   lttng-enable-channel(1) command:

       $ lttng enable-channel my-channel --userspace \
                                         --buffer-preallocation=on-demand

   LTTng can either preallocate all ring buffer memory up front (pre-2.15
   behaviour; still the default) or allocate as needed. Choose
   preallocation to ensure predictable memory usage and avoid run-time
   allocation latency; choose on demand to minimize the initial footprint
   for low traffic, for short-lived recording sessions, or in conjunction
   with memory reclaims.

   This feature is also enabled by sparse file-backed ring buffers.

• Before this release, an application terminated while writing to a user
   space ring-buffer could leave a sub-buffer stuck in a
   partially-written state, making it unusable.

   A new health check mechanism, the _watchdog timer_, now periodically
   monitors user space ring-buffers and makes stalled sub-buffers
   consumable again to keep trace recording running smoothly.

   This mechanism is enabled by default and can be controlled with the
   new `--watchdog-timer` option of the lttng-enable-channel(1) command.
   As of LTTng 2.15, it only applies to user space channels with a
   per-user buffer ownership model.

• We completely revamped the human-readable output of the
   lttng-list(1) command.

   The command now has a clean tree structure. Colors help improve the
   readability of objects and properties when the connected terminal
   supports it: the `LTTNG_TERM_COLOR` and `NO_COLOR` [5] environment
   variables provide further terminal color control.

       ✔ Channel `channel0` ❬42 event rules❭
         🞂 Auto. memory reclaim policy: None
         🞂 Loss mode:                   Discard newest event record
         🞂 Preallocation policy:        On demand
         🞂 Ring buffer configuration:   4 sub-buffers of 512.00 KiB per CPU,
                                        per Unix user

         🞂 Timer periods:
           🞂 Monitor timer:  1.00 s
           🞂 Read timer:     Inactive
           🞂 Switch timer:   Inactive
           🞂 Watchdog timer: 2.00 s

         🞂 Statistics:
           🞂 Discarded event records: 23,182

   By default, the command:

   ‣ Adds empty lines between blocks of related information. Remove those
     empty lines with the new `compact` setting of the new
     `--style` option.

   ‣ Truncates lines, adding an ellipsis, to fit the current terminal
     width. Avoid truncation with the new `--no-truncate` option.

   If you still prefer or need the previous (“legacy”) version, set the
   `LTTNG_LIST_LEGACY` environment variable to `1` before you run
   lttng-list(1). Please note, however, that the legacy output will never
   show anything related to features introduced after LTTng 2.14.

• The lttng-list(1) command now shows the total memory usage of
   each channel:

       🞂 Memory usage: ❲●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳   19.07 MiB /   40.12 MiB

   The new `--mem-usage` option controls the display mode of memory
   usage: `compact` shows the memory usage for each Unix user or process
   while `full` shows the memory usage for each CPU (if available):

       🞂 Memory usage: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳   13.30 MiB /   40.12 MiB
         ┆ For UID 1000 (64-bit): ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄❳   13.30 MiB /   40.12 MiB
         ┆   CPU   0: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  540.00 KiB /    2.51 MiB
         ┆   CPU   1: ❲●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  360.00 KiB /    2.51 MiB
         ┆   CPU   2: ❲●●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.30 MiB /    2.51 MiB
         ┆   CPU   3: ❲●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  256.00 KiB /    2.51 MiB
         ┆   CPU   4: ❲●●●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.20 MiB /    2.51 MiB
         ┆   CPU   5: ❲●●●●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳    1.07 MiB /    2.51 MiB
         ┆   CPU   6: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  768.00 KiB /    2.51 MiB
         ┆   CPU   7: ❲●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  524.00 KiB /    2.51 MiB
         ┆   CPU   8: ❲●●●●●●●●┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄❳  792.00 KiB /    2.51 MiB

• The lttng-status(1) command now accepts the same options
   as lttng-list(1):

       $ lttng status --mem-usage=compact --channel=my-channel

   lttng-status(1) forwards its options as is to lttng-list(1) for the
   current recording session.

• The lttng(1) commands won't emit multi-byte UTF-8 sequences when the
   `LTTNG_NO_UTF_8` environment variable is set to `1`, making a best
   effort to find alternatives.

• New trigger conditions are now available for the
   lttng-add-trigger(1) command:

   `channel-buffer-usage-ge` and `channel-buffer-usage-le`:
       Satisfied when the buffer usage of a given channel becomes
       greater/less than or equal to some threshold (size in bytes or
       ratio of the total channel buffer size):

           $ lttng add-trigger --condition=channel-buffer-usage-ge \
                               --session=my-session --channel=my-channel \
                               --domain=kernel --threshold-ratio=0.75 \
                               --action=notify

   `session-consumed-size-ge`:
       Satisfied when the total consumed size of the tracing data of all
       the channels of a recording session becomes greater than or equal
       to a threshold:

           $ lttng add-trigger --condition=session-consumed-size-ge \
                               --session=my-session --threshold-size=100M \
                               --action=snapshot-session other-session

   `session-rotation-starts`:
   `session-rotation-finishes`:
       Satisfied when the rotation operation of a recording session
       starts or finishes:

           $ lttng add-trigger --condition=session-rotation-finishes \
                               --session=my-session --action=notify

• The `liblttng-ctl` C API is updated to support the new features above:

   ‣ Set the trace format of a recording session descriptor with
     lttng_session_descriptor_set_trace_format() before creating the
     recording session.

   ‣ Immediately reclaim user space channel memory for eligible
     sub-buffers with lttng_reclaim_channel_memory().

     Get the results with lttng_reclaim_handle_get_reclaimed_subbuffer_count()
     and lttng_reclaim_handle_get_pending_subbuffer_count().

     Optionally wait for pending sub-buffers to be reclaimed with
     lttng_reclaim_handle_wait_for_completion().

     When done, destroy the memory reclaim operation handle
     with lttng_reclaim_handle_destroy().

   ‣ Set and get the automatic memory reclamation policy of a user space
     channel with lttng_channel_set_automatic_memory_reclamation_policy()
     and lttng_channel_get_automatic_memory_reclamation_policy().

   ‣ Set and get the buffer preallocation policy of a user space channel
     with lttng_channel_set_preallocation_policy()
     and lttng_channel_get_preallocation_policy().

   ‣ Set and get the watchdog timer period of a channel with
     lttng_channel_set_watchdog_timer_interval()
     and lttng_channel_get_watchdog_timer_interval().

   ‣ Enumerate and query the CPU ID and memory usage of the data streams
     of a channel with the data stream info API:

     ⁃ lttng_channel_get_data_stream_info_sets()
     ⁃ lttng_data_stream_info_sets_get_count()
     ⁃ lttng_data_stream_info_sets_get_at_index()
     ⁃ lttng_data_stream_info_set_get_count()
     ⁃ lttng_data_stream_info_set_get_uid()
     ⁃ lttng_data_stream_info_set_get_pid()
     ⁃ lttng_data_stream_info_set_get_app_bitness()
     ⁃ lttng_data_stream_info_set_get_at_index()
     ⁃ lttng_data_stream_info_get_cpu_id()
     ⁃ lttng_data_stream_info_get_memory_usage()
     ⁃ lttng_data_stream_info_sets_destroy()

• The session daemon and relay daemon now support systemd's sd_notify()
   protocol, enabling proper synchronization with dependent systemd units
   by signaling when the daemons are ready to receive commands and when
   they're stopping.

• The internal v2.15 communication protocol between LTTng-tools and
   LTTng-UST is incompatible with the LTTng 2.14 one: you cannot use
   LTTng-tools 2.15 with LTTng-UST 2.14 and vice versa, for example.

• The lttng(1) Zsh completion function now fully completes the
   `add-trigger` command.

   From now on, the Debian/Ubuntu packages of LTTng-tools will contain
   the Zsh completion functions.

VERSION NAME
━━━━━━━━━━━━
This release is named after Péché Mortel, the iconic Imperial Stout from
Montréal's Dieu du Ciel! and a long-time team favorite.

Deep black and unapologetically intense, Péché Mortel pours with a
dense, mocha-tinted head and an aroma that immediately announces freshly
roasted coffee. Decadent and dangerously drinkable, it is the ideal
companion for long, cold winter evenings.

IMPORTANT LINKS
━━━━━━━━━━━━━━━
LTTng tarball:
     • <https://lttng.org/files/lttng-tools/lttng-tools-2.15.0-rc1.tar.bz2>
     • <https://lttng.org/files/lttng-ust/lttng-ust-2.15.0-rc1.tar.bz2>
     • <https://lttng.org/files/lttng-modules/lttng-modules-2.15.0-rc1.tar.bz2>

LTTng website:
     <https://lttng.org/>

LTTng 2.15 documentation:
     <https://lttng.org/docs/v2.15>

Mailing list for support and development:
     <https://lists.lttng.org/>

IRC channel:
     `#lttng` on `irc.oftc.net`

Bug tracker:
     <https://bugs.lttng.org/projects/lttng/>

GitHub organization:
     <https://github.com/lttng>

Continuous integration:
     • <https://ci.lttng.org/view/LTTng-tools/>
     • <https://ci.lttng.org/view/LTTng-ust/>
     • <https://ci.lttng.org/view/LTTng-modules/>

Code review:
     • <https://review.lttng.org/q/project:lttng-tools>
     • <https://review.lttng.org/q/project:lttng-ust>
     • <https://review.lttng.org/q/project:lttng-modules>

REFERENCES
━━━━━━━━━━
[1]: https://diamon.org/ctf/v1.8.3/ (CTF v1.8.3 specification)
[2]: https://diamon.org/ctf/ (CTF 2 specification)
[3]: https://babeltrace.org/
[4]: https://en.wikipedia.org/wiki/Sparse_file
[5]: https://no-color.org/

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Powered by blists - more mailing lists