linux-kernel - Re: sched_wakeup_new and sched_kthread

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1269524133.2957.4067.camel@gandalf.stny.rr.com>
Date:	Thu, 25 Mar 2010 09:35:33 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Xiao Guangrong <xiaoguangrong@...fujitsu.com>
Cc:	Ingo Molnar <mingo@...e.hu>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Hitoshi Mitake <mitake@....info.waseda.ac.jp>,
	Peter Zijlstra <peterz@...radead.org>,
	LKML <linux-kernel@...r.kernel.org>
Subject: Re: sched_wakeup_new and sched_kthread_stop events cause great
 overload

On Thu, 2010-03-25 at 17:36 +0800, Xiao Guangrong wrote:
> We have done sysbench test for ftrace's performance and it looks sched_wakeup_new
> and sched_kthread_stop events can cause great overload.
> 
> When we only enable sched_wakeup_new and sched_kthread_stop events, sysbench.threads
> shows the overload is 10%, sysbench.mutex shows the overload is 7.5%.
> 
> The more weird thing is that we found the sched_kthread_stop event is never called
> in this test, the test steps as follow:
> 
> echo > debugfs/tracing/set_event
> echo 1 > debugfs/tracing/events/sched/sched_wakeup_new/enable
> echo 1 > debugfs/tracing/events/sched/sched_kthread_stop/enable
> 
> com_opt="--num-threads=5000 --max-requests=50000"
> echo > debugfs/tracing/trace
> sysbench $com_opt --test=threads --thread-yields=1000 --thread-locks=10000 run >& log
> [or sysbench $com_opt --test=mutex --mutex-num=100 --mutex-locks=50000 --mutex-loops=10000 run for mutex]
> echo > debugfs/tracing/set_event
> 
> For sysbench.threads:
> cat debugfs/tracing/trace | grep "sched_wakeup_new" | wc -l
> 5001
> cat debugfs/tracing/trace | grep "sched_kthread_stop" | wc -l
> 0

Strange? So if you did:

cat debugfs/tracing/trace | wc -l

you should get 5005?

> 
> For sysbench.mutex:
> cat debugfs/tracing/trace | grep "sched_wakeup_new" | wc -l
> 5001
> cat debugfs/tracing/trace | grep "sched_kthread_stop" | wc -l
> 0
> 
> And, if only enable sched_kthread_stop event, the sysbench.threads's
> overload is 5.90%, the sysbench.mutex's overload is 3.36%.

Could be something that messes with the cache lines?

> 
> It hardly explain why sched_kthread_stop is never called but cause great overload.

Would be worth running perf against this test, to see where the issues
are occurring.

-- Steve


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/