Message-ID: <1ee4cd59-39ca-46b3-bf60-38d2042e67e2@efficios.com>
Date: Wed, 23 Jul 2025 10:32:37 -0400
From: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To: Indu Bhagat <indu.bhagat@...cle.com>, rostedt <rostedt@...dmis.org>
Cc: linux-kernel@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 bpf@...r.kernel.org, x86@...nel.org, Masami Hiramatsu <mhiramat@...nel.org>,
 Josh Poimboeuf <jpoimboe@...nel.org>, Peter Zijlstra <peterz@...radead.org>,
 Ingo Molnar <mingo@...nel.org>, Jiri Olsa <jolsa@...nel.org>,
 Namhyung Kim <namhyung@...nel.org>, Thomas Gleixner <tglx@...utronix.de>,
 Andrii Nakryiko <andrii@...nel.org>, "Jose E. Marchesi" <jemarch@....org>,
 Beau Belgrave <beaub@...ux.microsoft.com>, Jens Remus
 <jremus@...ux.ibm.com>, Linus Torvalds <torvalds@...ux-foundation.org>,
 Andrew Morton <akpm@...ux-foundation.org>, Jens Axboe <axboe@...nel.dk>,
 Florian Weimer <fweimer@...hat.com>, Sam James <sam@...too.org>,
 Brian Robbins <brianrob@...rosoft.com>,
 Elena Zannoni <elena.zannoni@...cle.com>
Subject: Re: [RFC] New codectl(2) system call for sframe registration

On 2025-07-23 04:16, Indu Bhagat wrote:
> On 7/22/25 11:49 AM, Mathieu Desnoyers wrote:
>> On 2025-07-22 14:21, Indu Bhagat wrote:
>>> On 7/21/25 8:20 AM, Mathieu Desnoyers wrote:
>>>> Hi!
>>>>
>>>> I've written up an RFC for a new system call to handle sframe 
>>>> registration
>>>> for shared libraries. There has been interest to cover both sframe in
>>>> the short term, but also JIT use-cases in the long term, so I'm
>>>> covering both here in this RFC to provide the full context. 
>>>> Implementation
>>>> wise we could start by only covering the sframe use-case.
>>>>
>>>> I've called it "codectl(2)" for now, but I'm of course open to 
>>>> feedback.
>>>>
>>>> For ELF, I'm including the optional pathname, build id, and debug link
>>>> information which are really useful to translate from instruction 
>>>> pointers
>>>> to executable/library name, symbol, offset, source file, line number.
>>>> This is what we are using in LTTng-UST and Babeltrace debug-info filter
>>>> plugin [1], and I think this would be relevant for kernel tracers as 
>>>> well
>>>> so they can make the resulting stack traces meaningful to users.
>>>>
>>>> sys_codectl(2)
>>>> =================
>>>>
>>>> * arg0: unsigned int @option:
>>>>
>>>> /* Additional labels can be added to enum code_opt, for 
>>>> extensibility. */
>>>>
>>>> enum code_opt {
>>>>      CODE_REGISTER_ELF,
>>>>      CODE_REGISTER_JIT,
>>>>      CODE_UNREGISTER,
>>>> };
>>>>
>>>> * arg1: void * @info
>>>>
>>>> /* if (@option == CODE_REGISTER_ELF) */
>>>>
>>>> /*
>>>>   * text_start, text_end, sframe_start, sframe_end allow unwinding 
>>>> of the
>>>>   * call stack.
>>>>   *
>>>>   * elf_start, elf_end, pathname, and either build_id or debug_link 
>>>> allows
>>>>   * mapping instruction pointers to file, symbol, offset, and source 
>>>> file
>>>>   * location.
>>>>   */
>>>> struct code_elf_info {
>>>> :   __u64 elf_start;
>>>>      __u64 elf_end;
>>>
>>> What are the elf_start , elf_end intended for ?
>>
>> The intent is to know at which address the first loadable segment of
>> the shared object is mapped (elf_start), and the size of the shared
>> object mapping, which is the sum of the size of its PT_LOAD segments.
>>
>> This allows tooling to easily lookup which addresses belong to that
>> shared object, for any loaded segment, whether it's code or data.
>>
>>>
>>>>      __u64 text_start;
>>>>      __u64 text_end;
>>>>      __u64 sframe_start;
>>>>      __u64 sframe_end;
>>>>      __u64 pathname;              /* char *, NULL if unavailable. */
>>>>
>>>>      __u64 build_id;              /* char *, NULL if unavailable. */
>>>>      __u64 debug_link_pathname;   /* char *, NULL if unavailable. */
>>>>      __u32 build_id_len;
>>>>      __u32 debug_link_crc;
>>>> };
>>>>
>>>>
>>>> /* if (@option == CODE_REGISTER_JIT) */
>>>>
>>>> /*
>>>>   * Registration of sorted JIT unwind table: The reserved memory 
>>>> area is
>>>>   * of size reserved_len. Userspace increases used_len as new code is
>>>>   * populated between text_start and text_end. This area is 
>>>> populated in
>>>>   * increasing address order, and its ABI requires to have no 
>>>> overlapping
>>>>   * fre. This fits the common use-case where JITs populate code into
>>>>   * a given memory area by increasing address order. The sorted unwind
>>>>   * tables can be chained with a singly-linked list as they become 
>>>> full.
>>>>   * Consecutive chained tables are also in sorted text address order.
>>>>   *
>>>>   * Note: if there is an eventual use-case for unsorted jit unwind 
>>>> table,
>>>>   * this would be introduced as a new "code option".
>>>>   */
>>>>
>>>> struct code_jit_info {
>>>>      __u64 text_start;      /* text_start >= addr */
>>>>      __u64 text_end;        /* addr < text_end */
>>>>      __u64 unwind_head;     /* struct code_jit_unwind_table * */
>>>> };
>>>>
>>>
>>> I see the discussion has evolved here with the general sentiment that 
>>> the JIT part needs to be kept in mind for a rough sketch but cannot 
>>> be designed at this time. But two comments (if we keep JIT part in 
>>> the discussion):
>>>    - I think we need to keep __u64 unwind_head not a pointer to a 
>>> defined structure (struct code_jit_unwind_table * above), but some 
>>> opaque type like we have for SFrame case.
>>
>> What is the reason for making this an opaque type for sframe ?
>>
> 
> So that the system call only does the work of registering the memory of 
> specific size as stack trace data for addr range (text_start, text_end). 
>   IIUC, in the current proposal, the format of the stack trace 
> information is exposed in the arg. So when the format evolves, this will 
> mean additional management via some flags?

There are various way to handle extensions here. The simplest
would be to add a new label to enum code_opt and register the extended
JIT abi as a new option. But this would likely involve a lot of
duplication if the goal is just to add one more field to struct
code_jit_unwind_fre.

I suspect that the unwind table and linked list of unwind tables is
something we won't want to change for this ABI. What I see could be
a relevant extension point is struct code_jit_unwind_fre, but given
that it will be placed into an array, making it extensible requires
some care: we'd need to keep track of its stride. We could do it like
this:

struct code_jit_unwind_table {
     __u64 reserved_len;
     __u64 used_len; /*
                      * Incremented by userspace (store-release), read by
                      * the kernel (load-acquire).
                      */
     __u64 next;     /* Chain with next struct code_jit_unwind_table. */
     __u32 fre_stride; /* Stride of fre array (includes padding). */
     __u32 fre_size;   /* Offset at end of last used field. */
     char fre[];
};

So extending struct code_jit_unwind_fre could be done by adding fields
at the end, thus potentially increasing its size or turning padding into
used fields. fre_size would keep track of the "used" fields.

I'm open to extend by size (with fre_size) or using flags, whatever fits
best.

Thanks,

Mathieu

> 
>>
>>>    - The reserved_len should ideally be a part of code_jit_info, so 
>>> the length can be known without parsing the contents.
>>
>> I've placed reserved_len within the unwind table because I planned to
>> have the jit information for a given range of text be a linked list of
>> tables. Therefore, if one table fills up, then another table can be
>> chained at the tail. Having the reserved_len part of each table makes
>> things easier to combine into a linked list.
>>
>> Thanks for your feedback !
>>
>> Mathieu
>>
>>>
>>>> struct code_jit_unwind_fre {
>>>>      /*
>>>>       * Contains info similar to sframe, allowing unwind for a given
>>>>       * code address range.
>>>>       */
>>>>      __u32 size;
>>>>      __u32 ip_off;  /* offset from text_start */
>>>>      __s32 cfa_off;
>>>>      __s32 ra_off;
>>>>      __s32 fp_off;
>>>>      __u8 info;
>>>> };
>>>>
>>>> struct code_jit_unwind_table {
>>>>      __u64 reserved_len;
>>>>      __u64 used_len; /*
>>>>                       * Incremented by userspace (store-release), 
>>>> read by
>>>>                       * the kernel (load-acquire).
>>>>                       */
>>>>      __u64 next;     /* Chain with next struct 
>>>> code_jit_unwind_table. */
>>>>      struct code_jit_unwind_fre fre[];
>>>> };
>>>>
>>>> /* if (@option == CODE_UNREGISTER) */
>>>>
>>>> void *info
>>>>
>>>> * arg2: size_t info_size
>>>>
>>>> /*
>>>>   * Size of @info structure, allowing extensibility. See
>>>>   * copy_struct_from_user().
>>>>   */
>>>>
>>>> * arg3: unsigned int flags (0)
>>>>
>>>> /* Flags for extensibility. */
>>>>
>>>> Your feedback is welcome,
>>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>> [1] https://babeltrace.org/docs/v2.0/man7/babeltrace2-filter.lttng- 
>>>> utils.debug-info.7/
>>>>
>>>
>>
>>
> 


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Powered by blists - more mailing lists