[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <1220527806.12161.14.camel@minggr>
Date: Thu, 04 Sep 2008 19:30:06 +0800
From: Lin Ming <ming.m.lin@...el.com>
To: Ingo Molnar <mingo@...e.hu>
Cc: Peter Zijlstra <a.p.zijlstra@...llo.nl>,
linux-kernel <linux-kernel@...r.kernel.org>,
"Zhang, Yanmin" <yanmin_zhang@...ux.intel.com>
Subject: Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
On Thu, 2008-09-04 at 13:09 +0200, Ingo Molnar wrote:
> * Lin Ming <ming.m.lin@...el.com> wrote:
>
> > I observed schedstats of sysbench, there's more
> > "nr_failed_migrations_hot"
> >
> > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> >
> > task migration failed because of task_hot, the system is un-balanced?
>
> would be nice to get a representative (==steady state) scheduler trace
> from the critical portions of that workload. See:
>
> http://people.redhat.com/mingo/sched-devel.git/readme-tracer.txt
>
> Ingo
A huge trace result file, see the attachment for more.
# tracer: sched_switch
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
mysqld-3799 [07] 151.411009: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411015: 3796:120:R + 3804:120:S
mysqld-3796 [00] 151.411018: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411020: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411020: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411028: 3799:120:R + 3807:120:S
mysqld-3796 [00] 151.411029: 3796:120:R + 3804:120:S
mysqld-3799 [07] 151.411030: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411031: 3796:120:R ==> 3804:120:R
sysbench-3807 [07] 151.411032: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411049: 3804:120:S ==> 3796:120:R
mysqld-3799 [07] 151.411052: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411053: 3799:120:R ==> 3807:120:R
mysqld-3796 [00] 151.411055: 3796:120:R + 3804:120:S
sysbench-3807 [07] 151.411063: 3807:120:S ==> 3799:120:R
mysqld-3796 [00] 151.411069: 3796:120:S ==> 3804:120:R
mysqld-3799 [07] 151.411070: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411071: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411073: 3807:120:S ==> 3799:120:R
sysbench-3804 [00] 151.411078: 3804:120:S ==> 3798:120:R
mysqld-3798 [00] 151.411093: 3798:120:R + 3806:120:S
mysqld-3799 [07] 151.411093: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411095: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411104: 3807:120:S ==> 3799:120:R
mysqld-3799 [07] 151.411114: 3799:120:R + 3807:120:S
mysqld-3799 [07] 151.411115: 3799:120:R ==> 3807:120:R
sysbench-3807 [07] 151.411116: 3807:120:S ==> 3799:120:R
mysqld-3798 [00] 151.411131: 3798:120:S ==> 3806:120:R
mysqld-3799 [07] 151.411140: 3799:120:R + 3807:120:S
sysbench-3806 [00] 151.411149: 3806:120:S ==> 3801:120:R
mysqld-3799 [07] 151.411149: 3799:120:S ==> 3807:120:R
mysqld-3801 [00] 151.411153: 3801:120:R + 3787:120:S
mysqld-3801 [00] 151.411154: 3801:120:R ==> 3787:120:R
tracer-3811 [02] 151.411195: 3811:120:R + 3812:120:R
tracer-3811 [02] 151.411198: 3811:120:R ==> 3812:120:R
sysbench-3807 [07] 151.411201: 3807:120:S ==> 0:140:R
mysqld-3787 [00] 151.411204: 0:140:R + 3797:120:S
mysqld-3787 [00] 151.411208: 3787:120:S ==> 3801:120:R
<idle>-0 [01] 151.411210: 0:140:R ==> 3797:120:R
mysqld-3801 [00] 151.411217: 3801:120:R + 3809:120:S
mysqld-3797 [01] 151.411225: 0:140:R + 3800:120:S
<idle>-0 [03] 151.411228: 0:140:R ==> 3800:120:R
mysqld-3797 [01] 151.411231: 3797:120:R + 3805:120:S
mysqld-3800 [03] 151.411257: 0:140:R + 3802:120:S
<idle>-0 [05] 151.411259: 0:140:R ==> 3802:120:R
mysqld-3800 [03] 151.411266: 3800:120:R + 3808:120:S
mysqld-3801 [00] 151.411267: 3801:120:S ==> 3809:120:R
mysqld-3802 [05] 151.411268: 0:140:R + 3791:120:S
<idle>-0 [07] 151.411270: 0:140:R ==> 3791:120:R
mysqld-3797 [01] 151.411271: 3797:120:S ==> 3805:120:R
sysbench-3809 [00] 151.411271: 3809:120:R + 3801:120:S
sysbench-3805 [01] 151.411274: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411275: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411277: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411277: 3809:120:R + 3796:120:S
sysbench-3810 [05] 151.411279: 3810:120:S ==> 3802:120:R
mysqld-3791 [07] 151.411282: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411283: 3791:120:R ==> 3803:120:R
sysbench-3803 [07] 151.411285: 3803:120:S ==> 3791:120:R
sysbench-3805 [01] 151.411289: 3805:120:S ==> 3797:120:R
sysbench-3809 [00] 151.411292: 3809:120:S ==> 3796:120:R
mysqld-3797 [01] 151.411292: 3797:120:R + 3805:120:S
mysqld-3796 [00] 151.411297: 3796:120:R + 3798:120:S
mysqld-3796 [00] 151.411300: 3796:120:R + 3804:120:S
mysqld-3802 [05] 151.411308: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411309: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411312: 3791:120:R + 3803:120:S
mysqld-3791 [07] 151.411313: 3791:120:R ==> 3803:120:R
sysbench-3810 [05] 151.411323: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411326: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411332: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411333: 3802:120:R ==> 3810:120:R
mysqld-3791 [07] 151.411333: 3791:120:R + 3803:120:S
sysbench-3810 [05] 151.411334: 3810:120:S ==> 3802:120:R
mysqld-3796 [00] 151.411341: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411344: 3804:120:R + 3796:120:S
sysbench-3804 [00] 151.411356: 3804:120:S ==> 3796:120:R
mysqld-3802 [05] 151.411357: 3802:120:R + 3810:120:S
mysqld-3802 [05] 151.411358: 3802:120:R ==> 3810:120:R
mysqld-3796 [00] 151.411359: 3796:120:R + 3804:120:S
mysqld-3791 [07] 151.411363: 3791:120:S ==> 3803:120:R
sysbench-3803 [07] 151.411366: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411370: 3810:120:S ==> 3802:120:R
sysbench-3803 [07] 151.411374: 3803:120:S ==> 3791:120:R
mysqld-3802 [05] 151.411377: 3802:120:R + 3810:120:S
mysqld-3791 [07] 151.411378: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411394: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411396: 3804:120:R + 3796:120:S
mysqld-3797 [01] 151.411402: 3797:120:S ==> 3805:120:R
sysbench-3805 [01] 151.411405: 3805:120:R + 3797:120:S
mysqld-3802 [05] 151.411406: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411407: 3804:120:S ==> 3796:120:R
sysbench-3810 [05] 151.411409: 3810:120:R + 3802:120:S
mysqld-3791 [07] 151.411410: 3791:120:S ==> 3803:120:R
mysqld-3796 [00] 151.411410: 3796:120:R + 3804:120:S
sysbench-3803 [07] 151.411412: 3803:120:R + 3791:120:S
sysbench-3810 [05] 151.411420: 3810:120:S ==> 3802:120:R
sysbench-3805 [01] 151.411422: 3805:120:S ==> 3797:120:R
mysqld-3802 [05] 151.411423: 3802:120:R + 3810:120:S
sysbench-3803 [07] 151.411424: 3803:120:S ==> 3791:120:R
mysqld-3797 [01] 151.411426: 3797:120:R + 3805:120:S
mysqld-3791 [07] 151.411427: 3791:120:R + 3803:120:S
mysqld-3796 [00] 151.411443: 3796:120:S ==> 3804:120:R
sysbench-3804 [00] 151.411445: 3804:120:R + 3796:120:S
mysqld-3802 [05] 151.411454: 3802:120:S ==> 3810:120:R
sysbench-3804 [00] 151.411456: 3804:120:S ==> 3796:120:R
View attachment "trace.txt" of type "text/plain" (121037 bytes)
Powered by blists - more mailing lists