Date:	Mon, 30 Apr 2012 10:29:36 -0700
From:	Dave Johansen <davejohansen@...il.com>
To:	Yong Zhang <yong.zhang0@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: High CPU usage of scheduler?

On Fri, Apr 27, 2012 at 8:23 AM, Dave Johansen <davejohansen@...il.com> wrote:
> On Thu, Apr 26, 2012 at 8:10 PM, Yong Zhang <yong.zhang0@...il.com> wrote:
>>
>> On Thu, Apr 26, 2012 at 03:08:51PM -0700, Dave Johansen wrote:
>> > I am looking into moving an application from RHEL 5 to RHEL 6 and I
>> > noticed an unexpected increase in CPU usage. A little digging has led
>> > me to believe that the scheduler may be the culprit.
>> >
>> > I created the attached test_select_work.c file to test this out. I
>> > compiled it with the following command on RHEL 5:
>> >
>> > cc test_select_work.c -O2 -DSLEEP_TYPE=0 -Wall -Wextra -lm -lpthread
>> > -o test_select_work
>>
>> Hmm...Do both RHEL 5 and RHEL 6 have high resolution timer enabled?
>>
>> If not, could you please try to boot the one which enable high resolution
>> timer with 'highres=off' to see if things change?
>
> Yes, RHEL 6 has CONFIG_HIGH_RES_TIMERS=y. I rebooted and used the
> 'highres=off' in grub and got the following results:
>
>   ./test_select_work 1000 10000 300 4
>   time_per_iteration: min: 3130.1 us avg: 3152.2 us max: 3162.2 us
> stddev: 15.0 us
>   ./test_select_work 1000 10000 300 8
>   time_per_iteration: min: 4314.6 us avg: 4407.9 us max: 4496.3 us
> stddev: 60.6 us
>   ./test_select_work 1000 10000 300 40
>   time_per_iteration: min: 8901.7 us avg: 9056.5 us max: 9121.3 us
> stddev: 57.5 us
>
> Any other info that might be helpful?
>
> Thanks,
> Dave

I made some improvements to the program to make comparisons a bit
easier and the standard deviation a bit more meaningful. I have
attached the updated program and script I used to run it. I also made
a git repo to track the code and it's available at
git://github.com/daveisfera/test_sleep.git

I also ran the tests on a Dell Vostro 200 (dual core CPU) with several
different OS versions. I realize that several of these will have
different patches applied and won't be "pure kernel code", but this
was the simplest way that I could run the tests on different versions
of the kernel and get comparisons. I've attached the .svg files output
by gnuplot and the data that was used to generate the plots. I
apologize for having so many attachments, but I figured that this was
the best way to convey the information.

All of these tests were run with the following command with the
executable that was built on Ubuntu 11.10 (except for CentOS 6.2 which
had gcc available on it so I re-compiled it):
./run_test 1000 10 1000 250 8 4

The first interesting results is with Ubuntu 7.10. usleep seems to
behave as expected, but select and poll took significantly longer than
expected. Then with CentOS 5.6, the opposite is true, so I'm not sure
what result can be drawn from those conclusions.

I tried running the Ubuntu 8.04-4 LiveCD (because that's when the CFS
was introduced), but it wouldn't boot on the Dell Vostro 200.

CentOS 6.2 is where things start to get interesting, because a 10-15%
increase in the execution time is seen with select and poll. With
usleep, that penalty doesn't seem to be present for low numbers of
threads but seems to reach similar levels once the number of threads
is 2x the number of cores.

Ubuntu 11.10, Ubuntu 12.04, and Fedora 16 all show results basically
in line with CentOS 6.2 but with usleep always being a comparable
penalty the other sleep methods. The one exception to this is that on
Fedora 16, poll is lower than the other sleep methods with 3 threads.

One final test that I did was to install the PPA on Ubuntu 11.10 that
has BFS instead of CFS. Info about it can be found at:
https://launchpad.net/~chogydan/+archive/ppa
It doesn't seem to be suffering from this sleep issue (or at least in
a different way) because the results for 1-3 threads are at 2 ms as
expected (basically no penalty), but then at 4 threads it's about 20%
higher than the no sleep methods and it seems to maintain more of a
penalty at higher numbers of threads. I also didn't see the system CPU
usage when running the previous test with BFS (but this might just be
an accounting thing since CFS and BFS account for CPU usage
differently).

I know that these results aren't 100% conclusive, but they seem to
indicate to me that something about the sleeping mechanism in the
kernel is using more CPU than I expect it to and that this is the
cause of the increased CPU usage. It doesn't appear to be the
scheduler itself because then a similar sort of penalty would have
been seen with sched_yield. It appears to me that it is something to
do with the sleep mechanism's interaction with the scheduler
(particularly the CFS more than the BFS).

Is there any more data I can gather or tests that I can run that can
help diagnose this problem?

Thanks in advance for any help,
Dave

View attachment "test_sleep.c" of type "text/x-csrc" (7372 bytes)

Download attachment "centos_5.6.svg" of type "image/svg+xml" (16791 bytes)

Download attachment "centos_6.2.svg" of type "image/svg+xml" (16776 bytes)

Download attachment "fedora_16.svg" of type "image/svg+xml" (16536 bytes)

Download attachment "results.tgz" of type "application/x-gzip" (9128 bytes)

Download attachment "ubuntu_7.10.svg" of type "image/svg+xml" (16776 bytes)

Download attachment "ubuntu_11.10.svg" of type "image/svg+xml" (16556 bytes)

Download attachment "ubuntu_11.10_bfs.svg" of type "image/svg+xml" (16564 bytes)

Download attachment "ubuntu_12.04.svg" of type "image/svg+xml" (16539 bytes)

Download attachment "run_test" of type "application/octet-stream" (746 bytes)

Powered by blists - more mailing lists