Message-ID:
 <MEYP282MB231286A088AE3BB50B468713C626A@MEYP282MB2312.AUSP282.PROD.OUTLOOK.COM>
Date: Fri, 1 Aug 2025 22:42:03 +0800
From: Levi Zim <rsworktech@...look.com>
To: Lucas Tanure <tanure@...ux.com>,
 Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
 kernelnewbies@...nelnewbies.org
Subject: Re: Linux Device Performance Analyses - Where are my cylces?

Hi Lucas,

On 8/1/25 5:34 PM, Lucas Tanure wrote:
> Hey there,
>
> I'm a kernel developer working on an Android device with a 5.10 
> kernel. I'm trying to improve performance, and to do that, I need to 
> figure out exactly where the CPU is spending its time.
>
> Our vendor provides a generic Android BSP, so the system is likely 
> running many unnecessary processes for things like GPS and phone calls 
> that my device doesn't even need. Since I don't have a good handle on 
> all these processes, I want to create a detailed breakdown of all the 
> tasks, workqueues, IRQs, kernel threads, softIRQs, and tasklets 
> running and measure the CPU time each one is using.
>
> To do this, I've been using kernel traces with the following events:
>
> task:task_newtask sched:sched_process_fork
> sched:sched_process_exec sched:sched_process_exit sched:sched_switch 
> irq:irq_handler_entry irq:irq_handler_exit irq:softirq_entry 
> irq:softirq_exit workqueue:workqueue_execute_start 
> workqueue:workqueue_execute_end syscalls:sys_enter_execve 
> syscalls:sys_exit_execve syscalls:sys_enter_execveat 
> syscalls:sys_exit_execveat
>
> However, the trace logs don't provide the full picture. For example, 
> when a new process is executed, the logs don't specify the script 
> being run. The sched_switch event tells me which PID is being 
> executed, but multiple processes can originate from the same binary, 
> so it's hard to distinguish them.
>
> To solve this, I've developed a patch for the do_execveat_common 
> function to log all new processes along with their full command lines. 
> I plan to use this patch with a Python script to parse the logs and 
> generate a report on CPU usage.
>
> My main questions are:
>
> - Is my patch correct? When I log the new process, do current->pid and 
> argv refer to the same new process?
Although I am not an experienced kernel developer, the patch LGTM, but 
it comes with a caveat that joining the args with whitespace makes it 
impossible to tell whether a whitespace comes from the arg itself or 
serves as a delimiter for the argument. It is fine for identifying the 
process but can cause problems for debugging.
> - It feels like logging new processes with their command lines is a 
> fundamental requirement for this kind of analysis. Am I missing a 
> better or more standard way of doing this?

I think the standard way to do this is using

- strace(strace -Y -f -qqq -s99999 -e trace=execve,execveat <CMD>), 
which is implemented via ptrace and might be hard to use if you want to 
trace all exec events as that would require you to strace all processes.
- execsnoop, which is implemented via eBPF and thus naturally traces all 
exec events. There are multiple implementations of execsnoop, from BCC 
and bpftrace. (And there are other implementations, one from 
brendangregg/perf-tools implemented via ftrace for example)
- tracexec [1] (disclaimer: I am the author),  which has both ptrace and 
eBPF backends united within three consistent user interface(log, TUI and 
JSON export). I think this is so far the most debugging friendly 
solution as it can also capture the (diff of) environment variables, 
inherited file descriptors and output the cmdline as copy'n'paste-able 
shell command. Most eBPF tracing solutions like execsnoop suffer from an 
issue that it silently ignores exec events happening in non-native 
syscall interface(like 32bit exec on x64 systems), which is addressed in 
tracexec. But as the eBPF backend of tracexec is still experimental and 
requires 6.x kernel, it might not suit your need well.

> - I don't want to reinvent the wheel, so if there's a better way to 
> analyze kernel performance on a device, I'd love to hear about it.
>
> After looking into other methods like eBPF and kprobes, I found that 
> this patch is the most straightforward way to get the information I need.
I think eBPF is the most convenient way to go and there are already a 
lot of tools available but as you are already a kernel developer you may 
find patching the kernel more convenient. Hope you find the solution 
that best suit you.

[1]: https://github.com/kxxt/tracexec

Best,
Levi

Powered by blists - more mailing lists