Message-ID: <b554bfa3-d710-4671-945b-5d6ec49e52cd@efficios.com>
Date: Mon, 14 Jul 2025 15:41:02 -0400
From: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To: Steven Rostedt <rostedt@...nel.org>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>,
 Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
 Christoph Hellwig <hch@...radead.org>,
 "Masami Hiramatsu (Google)" <mhiramat@...nel.org>,
 Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...nel.org>,
 Thomas Gleixner <tglx@...utronix.de>,
 linux-kernel <linux-kernel@...r.kernel.org>
Subject: [RFC] LTTng upstreaming next steps

Hi,

It was suggested that the LTTng kernel tracer be upstreamed and become
its own kernel subsystem in a recent LKML discussion [1]. This would
allow this 20 years old project to move its development from kernel
modules within an out-of-tree git repository into the Linux kernel tree.

The LTTng kernel tracer repository can be found here:

   https://git.lttng.org/lttng-modules.git

Considering that the most recent attempts to upstream it were not
successful due to the size of this project (~75k LOC) and lack of time
from tracing maintainers to review it, I'm sending this RFC to ask what
is the best way forward.

The LTTng project has been maintained for 20 years, being a kernel patch
series from 2005 to 2010, and an out-of-tree kernel module from 2010
until now.

Over those 20 years, we have tried to be good citizens by actively
contributing to Linux kernel within the following area:

- Tracepoints,
- Restartable Sequences (rseq) system call,
- Membarrier system call,
- RCU review,
- Tracing review,

LTTng is available in most Linux distributions [2]. We have also
contributed to help the tracing users and developers meet and discuss by
organizing 14 Tracing Summit events over the last 17 years [3]. For the
past two years, we have hosted monthly HPC tracing collaboration
meetings with the HPC community.

Past LTTng upstreaming attempts
===============================

Please keep in mind that this is only my personal recollection of the
LTTng upstreaming attempts.

- 2005 to 2010:
   - Several attempts at upstreaming LTTng 0.x were made between 2005 and
     2010. The original LTTng was a patch series, and it predates both
     ftrace and perf.
     In 2007 and 2008, I contributed the Kernel Markers and then the
     Tracepoint instrumentation APIs to the Linux kernel, so make static
     instrumentation available to all tracers. Ftrace and perf were quick
     to adopt it, then making it harder for LTTng to justify its
     integration upstream.
   - What made Tracepoint accepted by kernel maintainers was the ongoing
     work on asm goto (used for jump labels). I have been actively
     involved in its genesis through redaction of a desiderata document
     and various discussions with both kernel and compiler people. This
     led to the existence of asm goto in gcc.

- 2010:
   - The upstreaming plan was presented at the Collaboration Summit 2010,
     discussed with the community.
     - https://www.efficios.com/pub/lfcs2010/desnoyers-presentation-lfcs2010.pdf
   - I attempted to modify TRACE_EVENT macros to suit the needs of LTTng.
     My changes rejected because not immediately useful to in-kernel tracers.
   - After this first failure, I moved the focus on libringbuffer:
     - libringbuffer was discussed and planned with the community before
        beginning the effort, initially aimed at replacing ftrace ring buffer,
        creating a common ring buffer infrastructure for high-troughput tracers
        (ftrace and lttng). It was rejected upon submission upstream by the
        perf maintainers, moving the goal post to cover perf as well, which
        is significantly different because it focuses on sampling use-cases.
   - After those two unsuccessful attempts at upstreaming and running out
     of funding to work on kernel upstreaming, the focus of the
     lttng-modules project changed and aimed to become a standalone OOT
     kernel module. It was officially announced 2 years later:
     https://events.static.linuxfound.org/images/stories/pdf/eeus2012_desnoyers.pdf

- 2012:
   - I attempted to merge LTTng through the staging tree. Ingo Molnar
     opposed wanting LTTng to be integrated into Perf instead.
     https://lore.kernel.org/lkml/20111219104915.GA19861@elte.hu/
   - I just did not have the available resources to undertake the
     endeavor proposed by Ingo, which was to rebuild LTTng within
     ftrace and perf to make its code base common with tools which have
     only a partial requirement overlap.

- 2020:
   - Steven Rostedt showed interest in helping me upstream LTTng.
     After discussion with him, I prepared a "v5.7-rc1-lttng-upstreaming"
     branch. The initial plan was that he would provide a few rounds of
     feedback privately before I would publish it publically.
     He never had time to review it and provide feedback, only mentioned
     that it was more code than he initially expected. This effort did
     not lead anywhere.
     https://github.com/compudj/linux-dev/tree/v5.7-rc1-lttng-upstreaming

Open questions on how to proceed with upstreaming
=================================================

My goal is to move the development of the "lttng-modules" project into
the upstream kernel.

There are a few possible approaches we can envision, just to list a few
choices we have:

A) Bulk upstreaming

    Upsides:
    - The upstreamed LTTng kernel tracer becomes a drop-in replacement
      for the lttng-modules project. Allows to quickly phase out the OOT
      development model.
    - Preserves the existing LTTng-modules userspace ABI.
    - This moves the status-quo from "out of tree" to "in tree".

    Downsides:
    - Not developed from the ground up with the help of the kernel
      community. This rather becomes an after-the-fact review of a large
      code base.
    - Brings in backward compatibility macros and ifdefs to support
      older kernels which are pointless upstream. This can be easily
      mitigated by doing a clean up removing this compatibility code
      as part of the upstreaming exercise.

B) Piecewise upstreaming, basically creating a "new" LTTng kernel tracer
    by reimplementing the current tracer differently based on feedback.

    Upsides:
    - Allows the kernel community to be involved in the
      "re-implementation" of LTTng within the kernel tree.

    Downsides:
    - Large commitment. Not sure if funding this effort is feasible.
    - Uncertainty about the result: after all this effort, there is no
      guarantee that any upstream maintainer will be able to justify the
      time needed for review.
    - Requires to still maintain lttng-modules OOT for the forseeable
      future, as the upstream work is not a drop-in replacement ABI-wise.
    - This preserves the status-quo of LTTng-modules as "out of tree".

C) If we go for the bulk upstreaming, we can either:

   - Try to keep the development history of LTTng-modules. This may be
     tricky because the lttng-modules git tree is separate from the Linux
     kernel. AFAIR this has been done in the past when integrating Btrfs
     into the kernel tree:
       commit 4b48d9d44ebe ("Merge git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable")

   - Import LTTng-modules as a set of new kernel commits, e.g. one commit
     grouping a set of files which implement a given facet of the tracer.

D) Should we integrate LTTng through the staging tree ?

    LTTng could end up into kernel/lttng/, or could go through the staging
    tree while we remove the code which is not relevant in the context of
    the upstream kernel (e.g. older kernel support).

E) Keep existing ioctl ABIs or move to new set of ABIs ?

    If we choose to go for a piecewise approach where we want to redefine
    the userspace ABIs, we would have to use new ioctl numbers to make
    sure we do not break the existing LTTng-modules ABI guarantees.

F) Integrate the LTTng-modules "core", initially leaving the tracepoint
    and system call instrumentation out-of-tree.

    Considering that the LTTng-modules instrumentation code for
    tracepoints and system calls represents a significant portion of
    the SLOC, we could keep those out of the initial upstreaming and
    focus on the LTTng core first.

LTTng Kernel Tracer Overview
============================

SLOC    path
----------------------------------------
11231   include/instrumentation/events
25667   include/instrumentation/syscalls

5090    include/lttng
1767    include/ringbuffer
1246    include/wrapper
396     include/counter
11      include/blacklist

23334   src/*.[ch]
4645    src/lib
865     src/probes
273     src/wrapper
207     src/tests

The largest piece of code is include/instrumentation/syscalls
(25667 SLOC), which contains the generated instrumentation code for system
calls which allow fetching the system call ABI fields from userspace for
the following architectures: arm 32/64, mips 32/64, powerpc 32/64, x86
32/64.

Those are included from src/lttng-syscalls* (1687 SLOC) to generate the
syscall tables for instrumentation.

A large chunk is also the tracepoint instrumentation
include/instrumentation/events (11231 SLOC), which reimplements a
portion of the upstream kernel include/trace/events in ways that are
compatible with LTTng filtering. Those are included from "probe" files
in src/probes/* to generate probe modules (865 SLOC).  Note that a large
portion of those instrumentation headers consists of backward
compatibility for older kernels with kernel version #ifdefs, and very
fortunately be removed as part of the upstreaming process.

The "wrapper" include subdir is mostly for backward compatibility with
the supported older kernels, which can go away as a cleanup as part of
the upstreaming benefits.

The LTTng ring buffer sits in include/ringbuffer and src/lib/ringbuffer,
for a total of 3589 SLOC. This is a generic ring buffer which was
originally created to cover LTTng-UST, LTTng modules and Ftrace. The
LTTng ring buffer "clients" (templated C) wiring up the ring buffer
library for LTTng can be found in src/lttng-ring-buffer-* (1455 SLOC).

The other significant portion is src/*.[ch], which contains the ABI that
exposes LTTng-modules to userspace:

lttng-abi.c (4190 SLOC) implements the userspace ABI (LTTng ioctls).

Note that the other relevant portion of the ABI is in the ringbuffer
code:

src/lib/ringbuffer/ring_buffer_vfs.c (381 SLOC)

All the src/lttng-context*.c implement the LTTng contexts, stuff like
pid, uid, perf counters, which can be optionally enabled by users.
(2947 SLOC)

The LTTng dynamic filter is implemented under src/lttng-bytecode*
(5028 SLOC).

The LTTng support for CTF 1.8 and CTF 2 is implemented under
src/metadata-ctf* (2082 SLOC).

We also have src/lttng-events.c (3742 SLOC) which handles all the
session/channel/event object model of LTTng.

Also relevant is src/lttng-statedump-impl.c (601 SLOC) which implement
the LTTng state dump, dumping kernel state at trace start and when
requested by the end user (e.g. list processes, interrupts, FDs,
namespaces, network interfaces, cpu topology), block devices.

Note that src/tests (207 SLOC) is fairly small. Our testing is mostly
done from the lttng-tools project [4].

Your feedback would be most appreciated,

Thanks,

Mathieu

[1] https://lore.kernel.org/lkml/20250711142834.73b2864c@batman.local.home/
[2] https://lttng.org/download/
[3] https://tracingsummit.org/
[4] https://ci.lttng.org/

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Powered by blists - more mailing lists