linux-kernel - Re: Query regarding work scheduling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250814134550.4b64b4ec@gandalf.local.home>
Date: Thu, 14 Aug 2025 13:45:50 -0400
From: Steven Rostedt <rostedt@...dmis.org>
To: Subbaraya Sundeep <sbhatta@...vell.com>
Cc: Tejun Heo <tj@...nel.org>, <mingo@...hat.com>, <peterz@...radead.org>,
 <juri.lelli@...hat.com>, <vincent.guittot@...aro.org>,
 <dietmar.eggemann@....com>, <bsegall@...gle.com>, <mgorman@...e.de>,
 <vschneid@...hat.com>, <jiangshanlai@...il.com>,
 <linux-kernel@...r.kernel.org>
Subject: Re: Query regarding work scheduling

On Thu, 14 Aug 2025 03:54:58 +0000
Subbaraya Sundeep <sbhatta@...vell.com> wrote:

> > Difficult to tell where the latencies are coming from. Maybe you can use
> > something like https://github.com/josefbacik/systing to look further into
> > it? All the scheduling events are tracked by default and you should be able
> > to add tracepoints and other events relatively easily. You can also set  

> Thanks for the reply. I am using simple busybox to avoid overhead of any other apps
> or deamons running in background and taking CPU time in between.
> I will try building systing and running it. 6.16 histogram shows that it
> is not one high latency event causing overall latency but bunch of small
> latencies are adding up and causing big latency.
> I suspect this has something to do with EEVDF scheduling since this behavior is
> seen from 6.6 (please note I may be wrong completly).
> Are there any methods or options with which I can bring back CFS scheduling behavior
> maybe with the knobs in /sys/kernel/debug/sched/features as a quick check? 

You could also use tracefs as that works on busybox:

 # echo 0 > /sys/kernel/tracing/tracing_on
 # echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable
[ and perhaps even more events ]
 # echo 1 > /sys/kernel/tracing/tracing_on
 # <run test>; echo 0 > /sys/kernel/tracing/tracing_on
 # cat /sys/kernel/tracing/trace

You could even make it a trace.dat file:

 # mkdir /tmp/tracing
 # cp -r /sys/kernel/tracing/events /tmp/tracing/
 # cp -r /proc/kallsyms /tmp/tracing/
[ have bs be PAGE_SIZE for your architecture ]
 # dd bs=4096 if=/sys/kernel/tracing/per_cpu/cpu0/trace_pipe_raw of=/tmp/tracing/trace0.raw
 # cd /tmp
 # tar cvf trace.tar tracing

Copy trace.tar to a desktop and extract it.

 $ cd /tmp
 $ tar xvf trace.tar
[ Make sure you have the latest trace-cmd installed ]
 $ trace-cmd restore -t /tmp/tracing/ -k /tmp/tracing/kallsyms -o /tmp/trace.dat /tmp/tracing/trace0.raw 
 $ trace-cmp report /tmp/trace.dat

Now you can send us the trace.dat file and we could analyze it more.

You could also enable more events than just sched_switch, like sched_waking
and such.

-- Steve