Message-ID: <D108F32273D3044F8C7B24328CB7713701A4D6@york.rivers.zai.com>
Date:	Tue, 25 Mar 2008 12:44:11 -0400
From:	"Sanders, Rob M." <sanders-rob@....com>
To:	"Ray Lee" <ray-lk@...rabbit.org>
Cc:	<linux-kernel@...r.kernel.org>, <bart.vanassche@...il.com>
Subject: RE: Performance changes between 2.6.13 and 2.6.23




-----Original Message-----
From: madrabbit@...il.com on behalf of Ray Lee
Sent: Tue 3/25/2008 12:41 PM
To: Sanders, Rob M.
Cc: linux-kernel@...r.kernel.org; bart.vanassche@...il.com
Subject: Re: Performance changes between 2.6.13 and 2.6.23
 
On Tue, Mar 25, 2008 at 5:34 AM, Sanders, Rob M. <sanders-rob@....com> wrote:
> Hello all,
>   I've been lurking on the digest for some time (don't want to receive full lkml traffic at work) and saw the
>  posts about Wine performance regressions in 2.6.24.  Some of what I saw there, particularly Andi Kleen's
>  responses, mirror something that I see on my box at home.  I had emailed Andi directly since I only read the
>  digest, and I'm posting this here at his suggestion.  Please CC: rms@....com with any replies, as I'm only
>  getting the lkml digest.
>   I'm running on a dual 2GHZ G5 Powermac w/2GB ram.  I recently upgraded from YDL4.0.91 (2.6.13 kernel) to
>  YDL6 (2.6.23 kernel) and noticed that the overall performance of the box seems more sluggish after the upgrade.
>  Of more particular concern, the main application that I build has seen a 4-5X slowdown in performance.  Under
>  YDL4.0.91 I could process roughly 1e8 data points in ~2 seconds, and under YDL6.0 I now process 1e8 data points

So, two processors, and multiple processes passing data back and
forth. Key point seems to be:

>  in ~8 seconds.  Total CPU loading (from top) is about 5%, under both systems.  The application spawns multiple
>  processes and uses semaphores and shared memory to move data between the processes.  When I use 'vmstat 3' the
>  single biggest difference I see between YDL4.0.91 and YDL6 is that the YDL4.0.91 system is the YDL4.0.91 system is
>  context switching about 7000 per interval, whereas the YDl6 system is context switching about 1200 times per interval.

Many more context switches per second, so a lot less work is getting
done each time.

>   I've been talking somewhat with Owen Stampflee (works for the distro maker, TSS) and have rebuilt the kernel on
>  the 2.6 box removing some of the things that I don't need on my box (cell support, etc).  There did seem (subjectively)
>  to be a slight improvement after that.  When I get some more time to play at home I'm going to take the 2.6.13 .config
>  I use and build a 2.6.23 kernal using 'make oldconfig' and try to figure if it is a config issues.
>   I realize that going from 2.6.13 to 2.6.23 is a *huge* change, and that the problems may not be tied to the kernel
>  but to other things, but are there any other suggestions folks have for finding the cause of the performance slowdow

A lot has changed between 2.6.23 and current mainline as well.
Particularly in the scheduler, which I suspect is the issue for your
test. If possible, could you try a 2.6.25-rc-latest kernel, both
before and after an "echo 5 > /proc/sys/kernel/sched_features" and see
if that makes any difference?

===
Ray, 
  I'll add that to the list of things to try.  I do want to clarify that the YDL4 system with 7000 context switches every
3 seconds was processing 4-5 times as much data as the YLD6 system that had 1200 context switches.  The other test
I want to do is (re)install YDL5 and pull my app over to it.  I hadn't used YDL5 due to my compilers not working
on it.  I've upgraded those (at least to eval versions) now for YDL6.

Rob
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists