linux-kernel - Inconsistent load average on tickless kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK=uwuy80LaNdQ4P49i+bGUXFvxf1zB=oKqNBTdAyvFc+mOTyw@mail.gmail.com>
Date:	Sun, 5 Feb 2012 22:51:07 -0800
From:	Aman Gupta <aman@...1.net>
To:	linux-kernel@...r.kernel.org,
	Peter Zijlstra <peterz@...radead.org>,
	Chase Douglas <chase.douglas@...onical.com>,
	Damien Wyart <damien.wyart@...e.fr>,
	Kyle McMartin <kyle@...hat.com>,
	Venkatesh Pallipadi <venki@...gle.com>,
	Jonathan Nieder <jrnieder@...il.com>,
	Lesław Kopeć <leslaw.kopec@...za-klasa.pl>
Subject: Inconsistent load average on tickless kernels

I have an LVS/DR cluster of 10 machines that receive similar traffic
via a round-robin strategy. These machines run Debian Lenny with
2.6.26, and consistently have a 15-minute load average between 4-12
depending on the time of day.

Upgrading any one of these machines to a newer kernel compiled with
NO_HZ=y causes the reported load average to drop significantly. Here,
fe3 and fe12 are running 3.2.4:

        fe3:    0.48    0.53    0.55
        fe4:    6.73    5.59    5.11
        fe5:    5.93    5.29    5.60
        fe6:    6.20    5.79    6.08
        fe7:    8.32    5.65    5.05
        fe8:    6.34    5.85    5.93
        fe9:    5.80    5.46    5.53
       fe10:    5.49    4.91    5.03
       fe11:    6.60    6.11    6.10
       fe12:    0.39    0.54    0.46

The newly reported load average is much lower than the other machines
performing equivalent work, and does not match cpu usage numbers
reported by vmstat.

Originally, I attempted to upgrade to 2.6.32 (used by lenny-backports
and squeeze). In 2.6.32, load averages are misreported even when using
NO_HZ=n. This bug was fixed in 2.6.34-rc4 (74f5187ac8: sched: Cure
load average vs NO_HZ woes).

The NO_HZ=y case was supposed to be fixed in 2.6.37-rc5 (0f004f5a69:
sched: Cure more NO_HZ load average woes), with the commit stating
"behaviour between CONFIG_NO_HZ=[yn] should be equivalent". In my
environment, however, kernels after this patch was introduced still
misreport load averages when compiled with NO_HZ=y.

Here's a list of the kernel versions I've tried (more details in
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620297):

  Correct Load Average
    2.6.26.25
    2.6.32.55-620297patch (CONFIG_NO_HZ=n)

  Incorrect Load Average
    2.6.32-bpo.5-amd64
    2.6.32.55
    2.6.32.55-620297patch
    2.6.32.55-620297patch (nohz=off)
    2.6.37-rc5-cure-more
    2.6.39.4
    3.2.2
    3.2.4

Also worth nothing is that fe12 is much newer than fe3, which should
help rule out hardware as a cause of this bug.

    fe3: Dell PowerEdge 2950 w/ Xeon(R) CPU           L5420  @ 2.50GHz
  fe12: Dell PowerEdge R710 w/ Xeon(R) CPU           E5645  @ 2.40GHz

I've attached dmesg output from fe12 booting up on 3.2.4. I am happy
to provide any other information that would be useful, and would
appreciate any advice on patches to try or ways to narrow the bug down
further.

  Aman

View attachment "dmesg-fe12-3.2.4.txt" of type "text/plain" (59632 bytes)