Message-ID: <358c63d6-5a25-462d-af04-1703bb7840e8@redhat.com>
Date: Thu, 11 Dec 2025 08:46:08 -0500
From: Tony Camuso <tcamuso@...hat.com>
To: Bagas Sanjaya <bagasdotme@...il.com>, ning.sun@...el.com,
 tboot-devel@...ts.sourceforge.net
Cc: linux-kernel@...r.kernel.org, rppt@...nel.org, tglx@...utronix.de,
 mingo@...nel.org, bp@...en8.de, michal.camacho.romero@...ux.intel.com
Subject: Re: [tboot-devel] [PATCH 1/1] Disable CET when calling tboot shutdown
 procedure.


On 12/10/2025 1:10 PM, Tony Camuso wrote:
> On 11/24/2025 7:34 PM, Bagas Sanjaya wrote:
>> On Thu, Nov 13, 2025 at 09:37:14AM -0500, Tony Camuso wrote:
>>> The tboot->shutdown_entry is effectively bios code and CET needs to be
>>> disabled before calling it.
>>>
>>> It resolves TBOOT shutdown failure bug, reported on the SLES (SUSE Linux
>>> Enterprise Server) 16.0 OS. OS power off, called by the "init 0" command,
>>> was failing, due to activated Intel Control-Flow Enforcement Technology
>>> (CET).
>>> Disabling CET has allowed to execute OS and TBOOT shutdown properly.
>>
>> Are ``systemctl poweroff`` and ``shutdown -P`` are also affected?
>>
>> Confused...
>>
> 
> Yes, all shutdown methods on kernels launched with tboot, on systems that
> expose the CPU ibt flag to kernels v6.12+ will cause the stack trace appended
> below.
> 
> The stack trace demonstrates that CET enforcement collides with legacy
> BIOS shutdown code that lacks ENDBR markers. The kernel BUG at
> cet.c:102 is a direct result of CET being active when jumping into
> tboot->shutdown_entry.
> 
> Legacy BIOS/tboot code without ENDBR now traps, requiring CET to be disabled
> around that call.
> 
> The patch:
>      Prevents CET from falsely trapping on non-CET BIOS code.

Need to clarify further:

The patch prevents CET from trapping when tboot invokes the BIOS-provided
shutdown_entry routine, which lacks ENDBR instructions.

tboot side:
   In tboot_shutdown(), the kernel switches to the tboot page tables and
   then calls:
	shutdown = (void(*)(void))(unsigned long)tboot->shutdown_entry;
	shutdown();

   That shutdown_entry pointer comes from the tboot structure, populated
   at boot.
     In the tboot project directory, see include/tboot.h
     In the kernel, see include/linux/tboot.h

BIOS side:
   The actual routine behind shutdown_entry is implemented in BIOS/firmware.
   It’s not compiled with CET/IBT support, so it lacks the required ENDBR64
   instruction at its entry point

   When CET is still enabled, the CPU enforces IBT. Jumping into that BIOS
   routine without ENDBR triggers a #CP (control protection exception),
   which is what the stack trace shows.

   So it is the BIOS shutdown_entry function itself that causes the trap,
   but only because tboot is handing control to it while CET is active.

What happens:
   From the stack trace
     Missing ENDBR: 0x8041d0
     kernel BUG at arch/x86/kernel/cet.c:102!
     RIP: 0010:0x8041d0
   
   This shows the CPU trapping on entry into the shutdown routine at address
   0x8041d0, which is the pointer stored in tboot->shutdown_entry

   The shutdown_entry field is explicitly documented as the physical address
   of the BIOS shutdown routine. This structure is populated by tboot at boot.

> 
>      Maintains system stability during shutdown.
> 
>      Preserves CET protection elsewhere, only disabling it for the
>      narrow window where legacy firmware must run.
> 
> 
> [  169.420078] reboot: Power down
> [  169.427516] Missing ENDBR: 0x8041d0
> [  169.431128] ------------[ cut here ]------------
> [  169.435805] kernel BUG at arch/x86/kernel/cet.c:102!
> [  169.440840] Oops: invalid opcode: 0000 [#1] SMP NOPTI
> [  169.445966] CPU: 0 UID: 0 PID: 3354 Comm: poweroff Kdump: loaded Not tainted 6.12.0-124.8.1.el10_1.x86_64 #1 PREEMPT(voluntary)
> [  169.457580] Hardware name: Dell Inc. PowerEdge R570/03TJR3, BIOS 1.2.1 01/23/2025
> [  169.465113] RIP: 0010:exc_control_protection+0x18c/0x190
> [  169.470490] Code: 1c ff 45 31 c9 49 89 d8 b9 09 00 00 00 48 8b 93 80 00 00 00 be 63 00 00 00 48 c7 c7 a4 85 e5 a4 e8 79 92 30 ff e9 02 ff ff ff <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f
> [  169.489292] RSP: 0018:ff55b3cba167fa88 EFLAGS: 00010002
> [  169.494581] RAX: 0000000000000017 RBX: ff55b3cba167faa8 RCX: 00000000ffff7fff
> [  169.501765] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
> [  169.508949] RBP: 0000000000000003 R08: 0000000000000000 R09: ffffffffa59e2b08
> [  169.516132] R10: ffffffffa5922ac8 R11: 0000000000000003 R12: 0000000000000000
> [  169.523316] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> [  169.530514] FS:  00007f5f9a122140(0000) GS:ff39175c2de00000(0000) knlGS:0000000000000000
> [  169.538659] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  169.544454] CR2: 0000559386dc5320 CR3: 000000010fbe2000 CR4: 0000000000f71ef0
> [  169.551651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  169.558835] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
> [  169.566019] PKRU: 55555554
> [  169.568784] Call Trace:
> [  169.571295]  <TASK>
> [  169.573458]  ? show_trace_log_lvl+0x1b0/0x2f0
> [  169.577880]  ? show_trace_log_lvl+0x1b0/0x2f0
> [  169.582301]  ? asm_exc_control_protection+0x26/0x30
> [  169.587244]  ? exc_control_protection+0x18c/0x190
> [  169.592011]  ? __die_body.cold+0x8/0x12
> [  169.595910]  ? die+0x2e/0x50
> [  169.598863]  ? do_trap+0xca/0x110
> [  169.602243]  ? do_error_trap+0x65/0x80
> [  169.606049]  ? exc_control_protection+0x18c/0x190
> [  169.610816]  ? exc_invalid_op+0x50/0x70
> [  169.614715]  ? exc_control_protection+0x18c/0x190
> [  169.619482]  ? asm_exc_invalid_op+0x1a/0x20
> [  169.623728]  ? exc_control_protection+0x18c/0x190
> [  169.628496]  ? exc_control_protection+0x14f/0x190
> [  169.633263]  asm_exc_control_protection+0x26/0x30
> [  169.638030] RIP: 0010:0x8041d0
> [  169.641142] Code: Unable to access opcode bytes at 0x8041a6.
> [  169.646857] RSP: 0018:ff55b3cba167fb50 EFLAGS: 00010007
> [  169.652144] RAX: 00000000008041d0 RBX: 0000000000000000 RCX: 0000000000000005
> [  169.659341] RDX: 00c6e8a7c0000000 RSI: 0000000000000001 RDI: ffffffffff1ff000
> [  169.666525] RBP: 0000000000000005 R08: 0000000000000000 R09: 000000000000ffff
> [  169.673709] R10: 0000000000000000 R11: ffffffffffff0000 R12: 0000000000002001
> [  169.680906] R13: ffffffffa5ae02c8 R14: 00000000ffffffff R15: 0000000000000000
> [  169.688091]  ? tboot_shutdown+0x5b/0x140
> [  169.692084]  ? tboot_sleep+0x12c/0x140
> [  169.695890]  ? acpi_os_enter_sleep+0x2b/0x60
> [  169.700221]  ? acpi_hw_legacy_sleep+0x140/0x1c0
> [  169.704816]  ? acpi_power_off+0x16/0x40
> [  169.708715]  ? sys_off_notify+0x48/0x70
> [  169.712615]  ? notifier_call_chain+0x5a/0xd0
> [  169.716943]  ? atomic_notifier_call_chain+0x32/0x50
> [  169.721885]  ? do_kernel_power_off+0x3e/0x50
> [  169.726213]  ? native_machine_power_off+0x21/0x40
> [  169.730983]  ? __do_sys_reboot+0x1d2/0x240
> [  169.735151]  ? do_syscall_64+0x7d/0x160
> [  169.739053]  ? syscall_exit_work+0xf3/0x120
> [  169.743302]  ? syscall_exit_to_user_mode+0x32/0x190
> [  169.748243]  ? do_syscall_64+0x89/0x160
> [  169.752143]  ? __count_memcg_events+0xdf/0x170
> [  169.756645]  ? handle_mm_fault+0x256/0x370
> [  169.760813]  ? do_user_addr_fault+0x347/0x640
> [  169.765235]  ? exc_page_fault+0x73/0x160
> [  169.769228]  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
> 

Powered by blists - more mailing lists