linux-kernel - Re: High CPU usage of scheduler?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAAcYxUe-26DNZHqC8t4D7RgET6c_SYA=2AFa3DgPHR8cKYzxDA@mail.gmail.com>
Date:	Mon, 7 May 2012 11:12:38 -0700
From:	Dave Johansen <davejohansen@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: Re: High CPU usage of scheduler?

On Tue, May 1, 2012 at 10:28 AM, Dave Johansen <davejohansen@...il.com> wrote:
>
> On Mon, Apr 30, 2012 at 10:11 PM, Dave Johansen <davejohansen@...il.com> wrote:
> > On Fri, Apr 27, 2012 at 8:23 AM, Dave Johansen <davejohansen@...il.com> wrote:
> >> On Thu, Apr 26, 2012 at 8:10 PM, Yong Zhang <yong.zhang0@...il.com> wrote:
> >>>
> >>> On Thu, Apr 26, 2012 at 03:08:51PM -0700, Dave Johansen wrote:
> >>> > I am looking into moving an application from RHEL 5 to RHEL 6 and I
> >>> > noticed an unexpected increase in CPU usage. A little digging has led
> >>> > me to believe that the scheduler may be the culprit.
> >>> >
> >>> > I created the attached test_select_work.c file to test this out. I
> >>> > compiled it with the following command on RHEL 5:
> >>> >
> >>> > cc test_select_work.c -O2 -DSLEEP_TYPE=0 -Wall -Wextra -lm -lpthread
> >>> > -o test_select_work
> >>>
> >>> Hmm...Do both RHEL 5 and RHEL 6 have high resolution timer enabled?
> >>>
> >>> If not, could you please try to boot the one which enable high resolution
> >>> timer with 'highres=off' to see if things change?
> >>
> >> Yes, RHEL 6 has CONFIG_HIGH_RES_TIMERS=y. I rebooted and used the
> >> 'highres=off' in grub and got the following results:
> >>
> >>   ./test_select_work 1000 10000 300 4
> >>   time_per_iteration: min: 3130.1 us avg: 3152.2 us max: 3162.2 us
> >> stddev: 15.0 us
> >>   ./test_select_work 1000 10000 300 8
> >>   time_per_iteration: min: 4314.6 us avg: 4407.9 us max: 4496.3 us
> >> stddev: 60.6 us
> >>   ./test_select_work 1000 10000 300 40
> >>   time_per_iteration: min: 8901.7 us avg: 9056.5 us max: 9121.3 us
> >> stddev: 57.5 us
> >>
> >> Any other info that might be helpful?
> >>
> >> Thanks,
> >> Dave
> >
> > I made some improvements to the program to make comparisons a bit
> > easier and the standard deviation a bit more meaningful. It is
> > available in a git repo at git://github.com/daveisfera/test_sleep.git
> >
> > I tried sending results from running this updated program on several
> > OS versions, but it must have been rejected by the filters. The info
> > can be found in the bugzilla:
> > https://bugzilla.redhat.com/show_bug.cgi?id=812148
> >
> > Is there any more data I can gather or tests that I can run that can
> > help diagnose this problem?
> >
> > Thanks in advance for any help,
> > Dave
>
> I also added the images and descriptions to the stackexchange
> question, so they can be more easily viewed there:
> http://unix.stackexchange.com/questions/37391/high-cpu-usage-with-cfs

Here's an updated version that measures the user and system time of
each thread and also outputs those results. It also has a correction
for how the mean and standard deviation are updated inside of the
outer_iterations loop.

The updated results are available here:
http://unix.stackexchange.com/questions/37391/high-cpu-usage-with-cfs
and here:
https://bugzilla.redhat.com/show_bug.cgi?id=812148

Any other tests I can run to help diagnose the problem?

Thanks,
Dave

View attachment "test_sleep.c" of type "text/x-csrc" (11284 bytes)