linux-kernel - Re: Query regarding work scheduling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <aKRhpN8ES2kJYtSF@opensource>
Date: Tue, 19 Aug 2025 11:36:04 +0000
From: Subbaraya Sundeep <sbhatta@...vell.com>
To: Steven Rostedt <rostedt@...dmis.org>
CC: Tejun Heo <tj@...nel.org>, <mingo@...hat.com>, <peterz@...radead.org>,
        <juri.lelli@...hat.com>, <vincent.guittot@...aro.org>,
        <dietmar.eggemann@....com>, <bsegall@...gle.com>, <mgorman@...e.de>,
        <vschneid@...hat.com>, <jiangshanlai@...il.com>,
        <linux-kernel@...r.kernel.org>
Subject: Re: Query regarding work scheduling

Hi Steven,

On 2025-08-14 at 17:45:50, Steven Rostedt (rostedt@...dmis.org) wrote:
> On Thu, 14 Aug 2025 03:54:58 +0000
> Subbaraya Sundeep <sbhatta@...vell.com> wrote:
> 
> > > Difficult to tell where the latencies are coming from. Maybe you can use
> > > something like https://github.com/josefbacik/systing to look further into
> > > it? All the scheduling events are tracked by default and you should be able
> > > to add tracepoints and other events relatively easily. You can also set  
> 
> > Thanks for the reply. I am using simple busybox to avoid overhead of any other apps
> > or deamons running in background and taking CPU time in between.
> > I will try building systing and running it. 6.16 histogram shows that it
> > is not one high latency event causing overall latency but bunch of small
> > latencies are adding up and causing big latency.
> > I suspect this has something to do with EEVDF scheduling since this behavior is
> > seen from 6.6 (please note I may be wrong completly).
> > Are there any methods or options with which I can bring back CFS scheduling behavior
> > maybe with the knobs in /sys/kernel/debug/sched/features as a quick check? 
> 
> You could also use tracefs as that works on busybox:
> 
>  # echo 0 > /sys/kernel/tracing/tracing_on
>  # echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable
> [ and perhaps even more events ]
>  # echo 1 > /sys/kernel/tracing/tracing_on
>  # <run test>; echo 0 > /sys/kernel/tracing/tracing_on
>  # cat /sys/kernel/tracing/trace

Thank you for tracefs. I use it extensively and using timestamps in trace
data calculated the latencies.
> 
> You could even make it a trace.dat file:
> 
>  # mkdir /tmp/tracing
>  # cp -r /sys/kernel/tracing/events /tmp/tracing/
>  # cp -r /proc/kallsyms /tmp/tracing/
> [ have bs be PAGE_SIZE for your architecture ]
>  # dd bs=4096 if=/sys/kernel/tracing/per_cpu/cpu0/trace_pipe_raw of=/tmp/tracing/trace0.raw
>  # cd /tmp
>  # tar cvf trace.tar tracing
> 
> Copy trace.tar to a desktop and extract it.
> 
>  $ cd /tmp
>  $ tar xvf trace.tar
> [ Make sure you have the latest trace-cmd installed ]
>  $ trace-cmd restore -t /tmp/tracing/ -k /tmp/tracing/kallsyms -o /tmp/trace.dat /tmp/tracing/trace0.raw 
>  $ trace-cmp report /tmp/trace.dat
> 
> Now you can send us the trace.dat file and we could analyze it more.
> 
> You could also enable more events than just sched_switch, like sched_waking
> and such.
> 
I am on it. Thanks for the commands and your time.

Sundeep
> -- Steve