Message-ID: <5044e453-4fd8-4530-a793-292adc2888ac@kylinos.cn>
Date: Wed, 28 Jan 2026 16:39:05 +0800
From: Yaxiong Tian <tianyaxiong@...inos.cn>
To: "Masami Hiramatsu (Google)" <mhiramat@...nel.org>
Cc: rostedt@...dmis.org, axboe@...nel.dk, mathieu.desnoyers@...icios.com,
 linux-block@...r.kernel.org, linux-kernel@...r.kernel.org,
 linux-trace-kernel@...r.kernel.org
Subject: Re: [PATCH v3 2/3] tracing/kprobes: Make setup_boot_kprobe_events()
 asynchronous


在 2026/1/28 15:53, Masami Hiramatsu (Google) 写道:
> On Wed, 28 Jan 2026 15:24:15 +0800
> Yaxiong Tian <tianyaxiong@...inos.cn> wrote:
>
>> 在 2026/1/28 12:49, Masami Hiramatsu (Google) 写道:
>>> On Tue, 27 Jan 2026 09:23:12 +0800
>>> Yaxiong Tian <tianyaxiong@...inos.cn> wrote:
>>>
>>>> During kernel boot, the setup_boot_kprobe_events() function causes
>>>> significant delays, increasing overall startup time.
>>>>
>>>> The root cause is a lock contention chain: its child function
>>>> enable_boot_kprobe_events() requires the event_mutex, which is
>>>> already held by early_event_add_tracer(). early_event_add_tracer()
>>>> itself is blocked waiting for the trace_event_sem  read-write lock,
>>>> which is held for an extended period by trace_event_update_all().
>>>>
>>>> To resolve this, we have moved the execution of
>>>> setup_boot_kprobe_events() to the trace_init_wq  workqueue, allowing
>>>> it to run asynchronously.
>>>>
>>>> Signed-off-by: Yaxiong Tian <tianyaxiong@...inos.cn>
>>>> ---
>>>>    kernel/trace/trace_kprobe.c | 14 +++++++++++++-
>>>>    1 file changed, 13 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
>>>> index 9953506370a5..4c6621f02696 100644
>>>> --- a/kernel/trace/trace_kprobe.c
>>>> +++ b/kernel/trace/trace_kprobe.c
>>>> @@ -2031,6 +2031,13 @@ static __init int init_kprobe_trace_early(void)
>>>>    }
>>>>    core_initcall(init_kprobe_trace_early);
>>>>    
>>>> +static struct work_struct kprobe_trace_work __initdata;
>>>> +
>>>> +static void __init kprobe_trace_works_func(struct work_struct *work)
>>>> +{
>>>> +	setup_boot_kprobe_events();
>>>> +}
>>>> +
>>>>    /* Make a tracefs interface for controlling probe points */
>>>>    static __init int init_kprobe_trace(void)
>>>>    {
>>>> @@ -2048,7 +2055,12 @@ static __init int init_kprobe_trace(void)
>>>>    	trace_create_file("kprobe_profile", TRACE_MODE_READ,
>>>>    			  NULL, NULL, &kprobe_profile_ops);
>>>>    
>>>> -	setup_boot_kprobe_events();
>>>> +	if (trace_init_wq) {
>>>> +		INIT_WORK(&kprobe_trace_work, kprobe_trace_works_func);
>>>> +		queue_work(trace_init_wq, &kprobe_trace_work);
>>> Hmm, this queue_work is not required if kprobe_boot_events_buf[] is
>>> empty. We should check it because most of the time we don't need it.
>> Yes, I will improve it in the next version.
>>> Also, deferring initialization makes it indeterminate when this
>>> tracing will begin.
>> Indeed, While most scenarios don't need boot-time tracing, and users
>> prioritize boot speed, we must balance the need for deterministic traces
>> with faster startup.
> I just wonder why don't you define kprobe events after boot (e.g.
> from init script) instead of kernel cmdline. Using cmdline means
> it will be used for tracing kernel boot.
>
> - tracing kernel boot -> use kernel cmdline (synchronous)
> - tracing user boot -> use tracefs (asynchronous)
I was actually working on a boot optimization task when I found that my 
machine was getting blocked on trace-related operations.
>
>>> For kprobe event use case, I think setup_boot_kprobe_events() should
>>> check the kprobe_boot_events_buf is empty at first. But I think Yaxiong
>>> use case happens when you are using kprobe events via cmdline, is that
>>> correct?
>> The issue was identified without enabling kprobe events via the cmdline.
> Interesting. So is it fixed by another patch [1]?
>
> [1] https://lore.kernel.org/all/20260127053848.108473-1-sunliming@linux.dev/
>
Yes, specifically looking at trace_kprobe in isolation, that issue is 
indeed resolved. However, the process still ends up blocking the 
initialization of blktrace later on.
>> The core finding is that asynchronous initialization can drastically cut
>> boot time under different workloads. As blocking occurs elsewhere in the
>> tracing infrastructure beyond just kprobe events, adopting async is a
>> broadly applicable strategy for boot time optimization.
> Yes, but it is also possible to set it up from user space, because that
> user process can work asynchronously.
> We can make the ftrace initialization async to accelerate boot time, but
> that means it is not useful for tracing kernel boot.
>
>>> I think introducing "async" cmdline option is more preferable.
>> Agreed, this works. Users focused on boot speed over early-boot tracing
>> can opt for this parameter to gain a startup performance boost.
> Yeah, that is an option. Anyway, basically, users have another option to
> setup ftrace after boot user space asynchronously. That is my
> recommendation for such use case.
Yes, that's what I normally do.
>
>>> BTW, I found that the kprobe events from kernel cmdline will be made
>>> after boot-time tracing from bootconfig. Maybe it should be run this
>>> earlier timing too.
>> Yes. Additionally, this optimization does not conflict with the current
>> patch series at all.
>>
>> I'll submit the updated patch for the next version promptly.
> OK.
>
> Thank you,
>
>>> Thank you,
>>>
>>>
>>>> +	} else {
>>>> +		setup_boot_kprobe_events();
>>>> +	}
>>>>    
>>>>    	return 0;
>>>>    }
>>>> -- 
>>>> 2.25.1
>>>>
>

Powered by blists - more mailing lists