| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20250616163659.d372cf60ee0c476168cd448e@kernel.org>
Date: Mon, 16 Jun 2025 16:36:59 +0900
From: Masami Hiramatsu (Google) <mhiramat@...nel.org>
To: Masami Hiramatsu (Google) <mhiramat@...nel.org>
Cc: Naresh Kamboju <naresh.kamboju@...aro.org>, Steven Rostedt
<rostedt@...dmis.org>, open list <linux-kernel@...r.kernel.org>, Linux
trace kernel <linux-trace-kernel@...r.kernel.org>,
lkft-triage@...ts.linaro.org, Stephen Rothwell <sfr@...b.auug.org.au>, Arnd
Bergmann <arnd@...db.de>, Dan Carpenter <dan.carpenter@...aro.org>, Anders
Roxell <anders.roxell@...aro.org>
Subject: Re: next-20250605: Test regression: qemu-x86_64-compat mode ltp
tracing Oops int3 kernel panic
On Fri, 13 Jun 2025 17:27:53 +0900
Masami Hiramatsu (Google) <mhiramat@...nel.org> wrote:
> > Run 3:
> > - https://lkft.validation.linaro.org/scheduler/job/8311139#L1703
>
> This is the same as Run 2, and clearer.
>
> In do_int3(), if we hit a disappeared int3, it is evacuated after
> all. This means kprobe_int3_handler() is hit, and call get_kprobe()
> to find the corresponding kprobes. But,
>
> ffffffff8150a040 <get_kprobe>:
> ffffffff8150a040: f3 0f 1e fa endbr64
> ffffffff8150a044: e8 07 b0 e2 ff call ffffffff81335050 <__fentry__>
> ffffffff8150a049: 48 b8 eb 83 b5 80 46 movabs $0x61c8864680b583eb,%rax
> ffffffff8150a050: 86 c8 61
>
> It hits the ftrace and hooked by fgraph, and eventually returns
> via ftrace_return_to_handler()
>
> [ 137.338572] RIP: 0010:ftrace_return_to_handler+0xd5/0x1f0
> [ 137.338577] Code: 00 89 55 c8 48 85 ff 74 07 4c 89 b7 80 00 00 00 49 8b 94 24 38 0b 00 00 48 98 48 8b 04 c2 48 c1 e8 0c 0f b7 c0 48 89 45 b8 cc <90> 48 8b 05 e3 ac c2 01 48 63 80 f8 00 00 00 48 0f a3 45 b8 72 39
>
> This address is;
>
> $ eu-addr2line -fi -e vmlinux ftrace_return_to_handler+0xd5
> arch_static_branch inlined at /builds/linux/kernel/trace/fgraph.c:839:6 in ftrace_return_to_handler
> /builds/linux/arch/x86/include/asm/jump_label.h:36:2
> __ftrace_return_to_handler
> /builds/linux/kernel/trace/fgraph.c:839:6
> ftrace_return_to_handler
> /builds/linux/kernel/trace/fgraph.c:874:9
>
> It is for static_branch, which also uses a text_poke.
>
> -----
> #ifdef CONFIG_HAVE_STATIC_CALL
> if (static_branch_likely(&fgraph_do_direct)) { <======
> if (test_bit(fgraph_direct_gops->idx, &bitmap))
> static_call(fgraph_retfunc)(&trace, fgraph_direct_gops, fregs);
> -----
>
> But actually, this static_branch modifies the kernel code with
> smp_text_poke_single() (note, this is a wrapper of smp_text_poke_batch).
>
> And this is MISSED by the smp_text_poke_int3_handler() again and
> go through the kprobes, and hit ftrace (fgraph) and caused this
> loop.
>
> So the fundamental issue is that smp_text_poke_batch missed
> handling INT3.
>
> I guess some text_poke user do not get text_mutex?
Hmm, I've checked the smp_text_poke_* users, but it seems no problem.
Basically, those smp_text_poke* user locks text_mutex, and another
suspicious ftrace_start_up is also set under ftrace_lock.
ftrace_arch_code_modify_post_process() is also paired with
ftrace_arch_code_modify_prepare() and under ftrace_lock.
smp_text_poke_single()
ftrace_mod_jmp()
ftrace_enable_ftrace_graph_caller()
ftrace_modify_all_code() -> see [*1]
ftrace_disable_ftrace_graph_caller()
ftrace_modify_all_code() -> see [*1]
ftrace_update_ftrace_func()
update_ftrace_func()
ftrace_modify_all_code() -> see [*1]
smp_text_poke_batch_add()
arch_jump_label_transform_queue() -> lock text_mutex
ftrace_replace_code()
ftrace_modify_all_code() <------[*1]
arch_ftrace_update_code()
ftrace_run_update_code() -> lock text_mutex
ftrace_modify_code_direct() (only if ftrace_poke_late != 0)
ftrace_make_nop()
__ftrace_replace_code() <----[*3]
ftrace_replace_code(weak) --> Not used on x86 (overridden)
ftrace_modify_all_code() <--- [*1]
arch_ftrace_update_code() <---- [*4]
ftrace_run_update_code()-> lock text_mutex
__ftrace_modify_code()
ftrace_run_stop_machine()
arch_ftrace_update_code(weak) -> overridden on x86 see [*4]
ftrace_module_enable() -> lock text_mutex (see below)
ftrace_init_nop()
ftrace_nop_initialize()
ftrace_update_code()
ftrace_module_enable() -> lock text_mutex
prepare_coming_module()
load_modole()
ftrace_process_locs() -> lock ftrace_lock.
ftrace_init() -> OK (ftrace_poke_late == 0 because its early)
ftrace_module_init() -> OK (ftrace_poke_late == 0 because module is not live)
load_module()
ftrace_make_call()
__ftrace_replace_code() -> see [*3]
smp_text_poke_batch_finish()
arch_jump_label_transform_apply() -> lock text_mutex
ftrace_arch_code_modify_post_process() -> must be OK because this unlock text_mutex
ftrace_run_update_code()-> paired with ftrace_arch_code_modify_prepare()
ftrace_module_enable()-> paired with ftrace_arch_code_modify_prepare() (depends on ftrace_lock && ftrace_start_up)
ftrace_replace_code()
ftrace_modify_all_code() -> see [*1]
ftrace_start_up <does variable set under ftrace_lock ?>
ftrace_startup()
ftrace_startup_subops()
register_ftrace_graph() -> lock ftrace_lock
register_ftrace_function_probe() -> lock ftrace_lock
register_ftrace_function_nolock() -> lock ftrace_lock
ftrace_shutdown()
unregister_ftrace_function() -> lock ftrace_lock
ftrace_arch_code_modify_prepare() < this set ftrace_poke_late = 1>
ftrace_module_enable() -> lock ftrace_lock.
ftrace_run_update_code()
ftrace_run_modify_code()
ftrace_ops_update_code()
__ftrace_hash_move_and_update_ops()
ftrace_update_ops()
ftrace_startup_subops()
register_ftrace_graph() -> lock ftrace_lock
ftrace_shutdown_subops()
unregister_ftrace_graph() -> lock ftrace_lock
ftrace_hash_move_and_update_subops()
ftrace_hash_move_and_update_ops() -> [*2]
ftrace_hash_move_and_update_ops() <-- [*2]
process_mod_list() -> lock ftrace_lock
register_ftrace_function_probe() -> lock ftrace_lock
unregister_ftrace_function_probe_func() -> lock ftrace_lock
ftrace_set_hash() -> lock ftrace_lock
ftrace_regex_release() -> lock ftrace_lock
unregister_ftrace_function_probe_func() -> lock ftrace_lock
ftrace_startup_enable()
ftrace_startup_all()
ftrace_pid_reset() -> lock ftrace_lock
pid_write() -> lock ftrace_lock
ftrace_startup()
ftrace_startup_subops()
register_ftrace_graph() -> lock ftrace_lock
register_ftrace_function_probe() -> lock ftrace_lock
register_ftrace_function_nolock() -> lock ftrace_lock
ftrace_startup_sysctl()
ftrace_enable_sysctl() -> lock ftrace_lock
ftrace_shutdown()
ftrace_shutdown_subops()
unregister_ftrace_graph() -> lock ftrace_lock
unregister_ftrace_function_probe_func() -> lock ftrace_lock
ftrace_destroy_filter_files() -> lock ftrace_lock
unregister_ftrace_function() -> lock ftrace_lock
ftrace_shutdown_sysctl()
ftrace_enable_sysctl() -> lock ftrace_lock
Thanks,
--
Masami Hiramatsu (Google) <mhiramat@...nel.org>
Powered by blists - more mailing lists