Message-ID: <9c11b765-66df-46f3-b4ea-a0c7f52dac35@oracle.com>
Date: Wed, 5 Nov 2025 16:44:24 -0800
From: Indu Bhagat <indu.bhagat@...cle.com>
To: Fangrui Song <maskray@...rceware.org>
Cc: linux-toolchains@...r.kernel.org, linux-perf-users@...r.kernel.org,
        linux-kernel@...r.kernel.org
Subject: Re: Concerns about SFrame viability for userspace stack walking

On 11/5/25 12:21 AM, Fangrui Song wrote:
>> On Tue, Nov 4, 2025 at 1:21 AM Indu <indu.bhagat@...cle.com> wrote:
>> On 2025-10-29 11:53 p.m., Fangrui Song wrote:
>>> I've been following the SFrame discussion and wanted to share some
>>> concerns about its viability for userspace adoption, based on concrete
>>> measurements and comparison with existing compact unwind implementations
>>> in LLVM.
>>>
>>> **Size overhead concerns**
>>>
>>> Measurements on a x86-64 clang binary show that .sframe (8.87 MiB) is
>>> approximately 10% larger than the combined size of .eh_frame
>>> and .eh_frame_hdr (8.06 MiB total).
>>> This is problematic because .eh_frame cannot be eliminated - it contains
>>> essential information for restoring callee-saved registers, LSDA, and
>>> personality information needed for debugging (e.g. reading local
>>> variables in a coredump) and C++ exception handling.
>>>
>>> This means adopting SFrame would result in carrying both formats, with a
>>> large net size increase.
>>>
>>> **Learning from existing compact unwind implementations**
>>>
>>> It's worth noting that LLVM has had a battle-tested compact unwind
>>> format in production use since 2009 with OS X 10.6, which transitioned
>>> to using CFI directives in 2013 [1]. The efficiency gains are dramatic:
>>>
>>>     __text section: 0x4a55470 bytes
>>>     __unwind_info section: 0x79060 bytes (0.6% of __text)
>>>     __eh_frame section: 0x58 bytes
>>>
>>
>> I believe this is only synchronous? If yes, do you think this is a fair
>> measurement to compare against ?
>>
>> Does the compact unwind info scheme work well for cases of
>> shrink-wrapping ? How about the case of AArch64, where the ABI does not
>> mandate if and where frame record is created ?
>>
>> For the numbers above, does it ensure precise stack traces ?
>>
>>   From the The Apple Compact Unwinding Format document
>> (https://faultlore.com/blah/compact-unwinding/),
>> "One consequence of only having one opcode for a whole function is that
>> functions will generally have incorrect instructions for the function’s
>> prologue (where callee-saved registers are individually PUSHed onto the
>> stack before the rest of the stack space is allocated)."
>>
>> "Presumably this isn’t a very big deal, since there’s very few
>> situations where unwinding would involve a function still executing its
>> prologue/epilogue."
>>
>> Well, getting precise stack traces is a big deal and the users want them.
> 
> **Shrink-wrapping and precise stack traces**: Yes, compact unwind
> handles these through an extension proposed by OpenVMS (not yet
> upstreamed to LLVM):
> https://lists.llvm.org/pipermail/llvm-dev/2018-January/120741.html
> 

Thanks for the link.

The above questions were strictly in the context of the battle-tested 
"The Apple Compact Unwinding Format" in production in the lld/MachO 
implementation, not for the proposed OpenVMS extensions.

Is it possible to get answers to those questions with that context in place?

If shrink-wrapping and precise stack traces isnt supported without the 
OpenVMS extension (that is not yet implemented), arent we comparing 
apples vs pears here ?

> Technical details of the extension:
> 
> - A single unwind group describes a (prologue_part1, prologue_part2,
> body, epilogue) tuple.
> - The prologue is conceptually split into two parts: the first part
> extends up to and including the instruction that decreases RSP; the
> second part extends to a point after the last preserved register is
> saved but before any preserved register is modified (this location is
> not unique, providing flexibility).
>    + When unwinding in the prologue, the RSP register value can be
> inferred from the PC and the set of saved registers.
> - Since register restoration is idempotent (restoring preserved
> registers multiple times during unwinding causes no harm), there is no
> need to describe `pop $reg` sequences. The unwind group needs just one
> bit to describe whether the 1-byte `ret` instruction is present.

Is this true for the case of asynchronous stack tracing too ?

> - The `length` field in the compact unwind group descriptor is
> repurposed to describe the prologue's two parts.
> - By composing multiple unwind groups, potentially with zero-sized
> prologues or omitting `ret` instructions in epilogues, it can describe
> functions with shrink wrapping or tail duplication optimization.
> - Null frame groups (with no prologue or epilogue) are the default and
> can describe trampolines and PLT stubs.

PLT stubs may use stack (push to stack). As per the document "A null 
frame (MODE = 8) is the simplest possible frame, with no allocated stack 
of either kind (hence no saved registers)".  So null frame can be used 
for PLT only if the functions invoking the PLT stub were using an 
RBP-based frame.  Isnt it ?
BTW, but both EH Frame and SFrame have specific, condensed 
representation for metadata for PLT entries.

> 

Anyway, thanks for the summary.

I see that OpenVMS extension for asynchronous compact unwind descriptors 
is an RFC state ATM.  But few important observations and questions:

  - As noted in the recently revived discussion, 
https://discourse.llvm.org/t/rfc-improving-compact-x86-64-compact-unwind-descriptors/47471, 
there is going to be a *non-negligible* size overhead as soon as you 
move towards a specification for asynchronous (vs the current 
specification that caters to synchronous only).  Now add to it, the 
quirks of each architecture/ABI :). Any comments ?

  - From the document: "Use of any preserved register must be delayed 
until all of the preserved registers have been saved."
    Q: Does this work well with optimizing compilers ? Is this an ABI 
change being asked for multiple architectures ?

  - From the document: "It appears technically feasible for a null frame 
function to have a personality routine. However, the utility of such a 
capability seems too meager to justify allowing this. We propose to not
support this." and "If the first attempt to lookup an unwind group for 
an exception address fails, then it is (tentatively) assumed to have 
occurred within a null frame function or in a part of a function
that is adequately described by a null frame. The presumed return 
address is (virtually or actually) popped from the top of stack and 
looked up. This second attempted lookup must succeed, in which case 
processing continues normally. A failure is a fatal error."
   Q: Is this a problem, especially because the goal is to evolve the 
OpenVMS RFC proposal is subsume .eh_frame ?

Are there people actively working towards bringing this to fruition?

> Now, to compare this against SFrame's space efficiency for synchronous
> unwinding, I've built llvm-mc, opt, and clang with
> -fno-asynchronous-unwind-tables -funwind-tables across multiple build
> configurations (clang vs gcc, frame pointer vs sframe). The resulting
> .sframe section sizes are significant:
> 
> % cat ~/tmp/test-unwind.sh
> #!/bin/zsh
> conf() {
>    configure-llvm $@ -DCMAKE_EXE_LINKER_FLAGS='-pie
> -Wl,-z,pack-relative-relocs' -DLLVM_ENABLE_UNWIND_TABLES=on \
>      -DCMAKE_{EXE,SHARED}_LINKER_FLAGS=-fuse-ld=bfd -DLLVM_ENABLE_LLD=off
> }
> 
> clang=-fno-integrated-as
> gcc=("-DCMAKE_C_COMPILER=$HOME/opt/gcc-15/bin/gcc"
> "-DCMAKE_CXX_COMPILER=$HOME/opt/gcc-15/bin/g++")
> 
> fp="-fno-omit-frame-pointer -momit-leaf-frame-pointer
> -B$HOME/opt/binutils/bin -Wa,--gsframe=no
> -fno-asynchronous-unwind-tables -funwind-tables"
> sframe="-fomit-frame-pointer -momit-leaf-frame-pointer
> -B$HOME/opt/binutils/bin -Wa,--gsframe -fno-asynchronous-unwind-tables
> -funwind-tables"
> 
> conf custom-fp-sync -DCMAKE_{C,CXX}_FLAGS="$clang $fp"
> conf custom-sframe-sync -DCMAKE_{C,CXX}_FLAGS="$clang $sframe"
> conf custom-fp-gcc-sync -DCMAKE_{C,CXX}_FLAGS="$fp" ${gcc[@]}
> conf custom-sframe-gcc-sync -DCMAKE_{C,CXX}_FLAGS="$sframe" ${gcc[@]}
> 
> for i in fp-sync sframe-sync fp-gcc-sync sframe-gcc-sync; do ninja -C
> /tmp/out/custom-$i llvm-mc opt clang; done
> 
> % ~/Dev/unwind-info-size-analyzer/section_size.rb
> /tmp/out/custom-{fp,sframe}-{,gcc-}sync/bin/{llvm-mc,opt,clang}
> Filename                                    |       .text size |
>   EH size |   .sframe size |   VM size | VM increase
> --------------------------------------------+------------------+----------------+----------------+-----------+------------
> /tmp/out/custom-fp-sync/bin/llvm-mc         |  2124031 (23.5%) |
> 301136 (3.3%) |       0 (0.0%) |   9050149 |           -
> /tmp/out/custom-sframe-sync/bin/llvm-mc     |  2114383 (22.3%) |
> 367452 (3.9%) |  348235 (3.7%) |   9483621 |       +4.8%
> /tmp/out/custom-fp-gcc-sync/bin/llvm-mc     |  2744214 (29.2%) |
> 301836 (3.2%) |       0 (0.0%) |   9389677 |       +3.8%
> /tmp/out/custom-sframe-gcc-sync/bin/llvm-mc |  2705860 (27.7%) |
> 354292 (3.6%) |  356073 (3.6%) |   9780985 |       +8.1%
> /tmp/out/custom-fp-sync/bin/opt             | 38873081 (69.9%) |
> 3538408 (6.4%) |       0 (0.0%) |  55598521 |           -
> /tmp/out/custom-sframe-sync/bin/opt         | 39011423 (62.4%) |
> 4557012 (7.3%) | 4452908 (7.1%) |  62494765 |      +12.4%
> /tmp/out/custom-fp-gcc-sync/bin/opt         | 54654535 (78.1%) |
> 3631076 (5.2%) |       0 (0.0%) |  70001573 |      +25.9%
> /tmp/out/custom-sframe-gcc-sync/bin/opt     | 53644831 (70.4%) |
> 4857220 (6.4%) | 5263530 (6.9%) |  76205733 |      +37.1%
> /tmp/out/custom-fp-sync/bin/clang           | 68345753 (73.8%) |
> 6643384 (7.2%) |       0 (0.0%) |  92638305 |           -
> /tmp/out/custom-sframe-sync/bin/clang       | 68500319 (64.9%) |
> 8684540 (8.2%) | 8521760 (8.1%) | 105572021 |      +14.0%
> /tmp/out/custom-fp-gcc-sync/bin/clang       | 96515079 (82.8%) |
> 6556756 (5.6%) |       0 (0.0%) | 116524565 |      +25.8%
> /tmp/out/custom-sframe-gcc-sync/bin/clang   | 94583903 (74.0%) |
> 8817628 (6.9%) | 9696263 (7.6%) | 127839309 |      +38.0%
> 
> Note: in GCC FP builds, .text is larger due to missing optimization
> for RBP-based frames (e.g.
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108386). Once this
> optimization is implemented, GCC FP builds should actually have
> smaller .text than RSP-based builds, because RBP-relative addressing
> produces more compact encodings than RSP-relative addressing (which
> requires an extra SIB byte).
> 
> .sframe for sync is not noticeably smaller than that for async. This
> is probably because
> there are still many DW_CFA_advance_loc ops even in
> -fno-asynchronous-unwind-tables -funwind-tables builds.
> 

Possible that its because in the Apple Compact Unwind Format, the linker 
optimizes compact unwind descriptors into the three-level paged 
structure, effectively de-duplicating some content.

>>>     (On macOS you can check the section size with objdump --arch x86_64 -
>>> h clang and dump the unwind info with  objdump --arch x86_64 --unwind-
>>> info clang)
>>>
>>> OpenVMS's x86-64 port, which is ELF-based, also adopted this format as
>>> documented in their "VSI OpenVMS Calling Standard" and their 2018 post:
>>> https://discourse.llvm.org/t/rfc-asynchronous-unwind-tables-attribute/59282
>>>
>>> The compact unwind format achieves this efficiency through a two-level
>>> page table structure. It describes common frame layouts compactly and
>>> falls back to DWARF only when necessary, allowing most DWARF CFI entries
>>> to be eliminated while maintaining full functionality. For more details,
>>> see: https://faultlore.com/blah/compact-unwinding/ and the lld/MachO
>>> implemention https://github.com/llvm/llvm-project/blob/main/lld/MachO/
>>> UnwindInfoSection.cpp
>>>
>>
>> How does your vision of "linker-friendly" stack tracing/stack unwinding
>> format reconcile with these suggested approaches ? As far as I can tell,
>> these formats also require linker created indexes and are
>> non-concatenable (custom handling in every linker).  Something you've
>> had "significant concerns" about.
>>

This question is unanswered: What do you think about 
"linker-friendliness" of the current implementation of the lld/MachO 
implementation of the compact unwind format in LLVM ?

> 
> We can distinguish between linking-time and execution-time
> representations by using different section names.
> The OpenVMS specification says:
> 
>      It is useful to note that the run-time representation of unwind
> information can vary from little more than a simple concatenation of
> the compile-time information to a substantial rewriting of unwind
> information by the linker. The proposal favors simple concatenation
> while maintaining the same ordering of groups as their associated
> code.
> 
> The runtime library can build this index at runtime and cache it to disk.
> 

This will include the dynamic linker and the stack tracer in the Linux 
kernel (the latter when stack tracing user space stacks).  Do you think 
this is feasible ?

> Once the design becomes sufficiently stable, we can introduce an
> opt-in linker option --xxxxframe-index that builds an index from
> recognized format versions while reporting warnings for unrecognized
> ones.> We need to carefully design this mechanism to be stable and robust,
> avoiding frequent format updates.
> >>   From
>> https://docs.vmssoftware.com/vsi-openvms-calling-standard/#STACK_UNWIND_EXCEPTION_X86_64:
>> "The unwind dispatch table (see Section B.3.1, ''Unwind Dispatch
>> Table'') is created by the linker using information in the unwind
>> descriptors (see Section B.3.2, ''DWARF Unwind Descriptors'' and Section
>> B.3.3, ''Compact Unwind Description'') provided by compilers. The linker
>> may use the provided unwind descriptors directly or replace them with
>> equivalent optimized forms based on its optimization strategies."
>>
>> Above all, do users want a solution which requires falling back on
>> DWARF-based processing for precise stack tracing ?
> 
> The key distinction is that compact unwind handles the vast majority
> of functions without DWARF—the macOS measurements show __unwind_info
> at 0.6% of __text size with __eh_frame reduced to negligible size
> (0x58 bytes). While SFrame also cannot handle all frames, compact
> unwind achieves dramatic size reductions by making DWARF the exception
> rather than requiring it alongside a supplementary format.
> 

As we have tried to reason, this is a misleading comparison. The compact 
unwind tables format:
   - needs to be extended for asynchronous stack unwinding
   - needs to be extended for other ABI/architectures
   - Making it concatenable / linker-friendly will also likely impose 
some negative effects on size.

> The DWARF fallback provides flexibility for additional coverage when
> needed, but nothing is lost (at least for the clang binary on macOS)
> if DWARF fallback were disabled in a hypothetical future linux-perf
> implementation.
> 

Fair enough, thats something for linux-perf/kernel to decide.  Once the 
OpenVMS RFC is sufficiently shaped to become a viable replacement for 
.eh_frame, this question will be for the stakeholders to decide.

>>> **The AArch64 case: size matters even more**
>>>
>>> The size consideration becomes even more critical for AArch64, which is
>>> heavily deployed on mobile phones.
>>> There's an active feature request for compact unwind support in the
>>> AArch64 ABI: https://github.com/ARM-software/abi-aa/issues/344
>>> This underscores the broader industry need for efficient unwind
>>> information that doesn't duplicate data or significantly increase binary
>>> size.
>>>
>>
>> Our measurements with a dataset of about 1400 userspace artifacts
>> (binaries and shared libraries) show that the SFrame/(EH Frame + EH
>> Frame HDR) ratio is:
>>     - Average of 0.70 on AArch64.
>>     - Average of 1.00 on x86_64.
>>
>> Projecting the size of what you observe for clang binary on x86_64 to
>> conclude the size ratio on AArch64 is not very wise to do.
>>
>> Whether the size impact is worth the benefit: its a choice for users to
>> make.  SFrame offers the users fast, precise stack traces with simple
>> stack tracers.
> 
> Thank you for providing the AArch64 measurements. Even with a 0.70x ratio on
> AArch64, this represents substantial memory overhead when considering:
> 
> .eh_frame is already large and being complained about.
> Being unable to eliminate it (needed for debugging and C++ exceptions)
> and adding 0.70x more means significant additional overhead for users.
> 
>>> There are at least two formats the ELF one can learn from: LLVM's
>>> compact unwind format (aarch64) and Windows ARM64 Frame Unwind Code.
>>>
>>
>> Please, if you have any concrete suggestions (keeping the above goals in
>> mind), you already know how/where to engage.
> 
> I've provided concrete suggestions throughout this discussion.
> 

Apologies, I should have been more precise.  And I ask because you know 
the details about both SFrame and the variants of Compact Unwind 
Descriptor formats at this point :). If you have concrete suggestions to 
improve the SFrame format for size, please let us know.

>>> **Path forward**
>>>
>>> Unless SFrame can actually replace .eh_frame (rather than supplementing
>>> it as an accelerator for linux-perf) and demonstrate sizes smaller
>>> than .eh_frame - matching the efficiency of existing compact unwind
>>> approaches — I question its practical viability for userspace.
>>> The current design appears to add overhead rather than reduce it.
>>> This isn't to suggest we should simply adopt the existing compact unwind
>>> format wholesale.
>>> The x86-64 design dates back to 2009 or earlier, and there are likely
>>> improvements we can make. However, we should aim for similar or better
>>> efficiency gains.
>>>
>>> For additional context, I've documented my detailed analysis at:
>>>
>>> - https://maskray.me/blog/2025-09-28-remarks-on-sframe (covering
>>> mandatory index building problems, section group compliance and garbage
>>> collection issues, and version compatibility challenges)
>>
>> GC issue is a bug currently tracked and with a target milestone of 2.46.
>>
>>> - https://maskray.me/blog/2025-10-26-stack-walking-space-and-time-trade-
>>> offs (size analysis)
>>>

Powered by blists - more mailing lists