linux-kernel - Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160616172256.GJ30921@twins.programming.kicks-ass.net>
Date:	Thu, 16 Jun 2016 19:22:56 +0200
From:	Peter Zijlstra <peterz@...radead.org>
To:	Jirka Hladky <jhladky@...hat.com>
Cc:	linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...hat.com>,
	Kamil Kolakowski <kkolakow@...hat.com>
Subject: Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and
 SPECjvm2008 benchmarks against 4.6 kernel

On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote:
> Hello,
> 
> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008

Blergh, of course I don't have those.. :/

> benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.
> 
> We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
> well affected.
> 
> We have observed the drop on variety of different x86_64 servers with
> different configuration (different CPU models, RAM sizes, both with
> Hyper Threading ON and OFF, different NUMA configurations (2 and 4
> NUMA nodes)

What kind of config and userspace setup? Do you run this cruft in a
cgroup of sorts?

If so, does it change anything if you run it in the root cgroup?

> Linpack and Stream benchmarks do not show any performance drop.
> 
> The performance drop increases with higher number of threads. The
> maximum number of threads in each benchmark is the same as number of
> CPUs.
> 
> We have opened a BZ to track the progress:
> https://bugzilla.kernel.org/show_bug.cgi?id=120481
> 
> You can find more details along with graphs and tables there.
> 
> Do you have any hints which commit should we try to reverse?

There were only 66 commits or so, and I think we can rule out the
hotplug changes, which should reduce it even further.

You could see what the parent of this one does:

  2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels

If not that, maybe the parent of:

  c58d25f371f5 sched/fair: Move record_wakee()

After that I suppose you'll have to go bisect.