linux-kernel - [BISECTED][REGRESSION] INFO: rcu_sched

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <87mxne6z4o.fsf@puppet.chdir.org>
Date:	Thu, 06 Jan 2011 12:38:15 +0100
From:	Nicolas Bareil <nico@...ir.org>
To:	cl@...ux.com, mel@....ul.ie
Cc:	akpm@...ux-foundation.org, torvalds@...ux-foundation.org,
	linux-kernel@...r.kernel.org
Subject: [BISECTED][REGRESSION] INFO: rcu_sched_state detected stall on CPU


Hello

On my two HP Proliant DL160 G6, the system locks up for tens of seconds
when I copy a regular file into a LVM volume with the following command
line:

$ sudo dd if=5gigabytesfile of=/dev/hosts/myvol bs=4096

Logs are filled with call traces and theses messages:

  kernel: INFO: rcu_sched_state detected stall on CPU 5 (t=6000 jiffies)
  kernel: Uhhuh. NMI received for unknown reason 00 on CPU 7.
  kernel: Do you have a strange power saving mode enabled?
  kernel: Dazed and confused, but trying to continue


 My .config is available here : http://chdir.org/~nbareil/aa45484031/config-2.6.37.gz
 The (big!) kern.log is here  : http://chdir.org/~nbareil/aa45484031/kern.log.gz
 My System.map                : http://chdir.org/~nbareil/aa45484031/System.map-2.6.37.gz

After bisection, the culprit is aa45484031, to be 100% sure, I compiled
a 2.6.37 with this commit reverted and it works.

As a reminder, here is the commit:

commit aa45484031ddee09b06350ab8528bfe5b2c76d1c
Author: Christoph Lameter <cl@...ux.com>
Date:   Thu Sep 9 16:38:17 2010 -0700

    mm: page allocator: calculate a better estimate of NR_FREE_PAGES
    when memory is low and kswapd is awake
    
    Ordinarily watermark checks are based on the vmstat NR_FREE_PAGES as
    it is cheaper than scanning a number of lists.  To avoid
    synchronization overhead, counter deltas are maintained on a per-cpu
    basis and drained both periodically and when the delta is above a
    threshold.  On large CPU systems, the difference between the
    estimated and real value of NR_FREE_PAGES can be very high.  If
    NR_FREE_PAGES is much higher than number of real free page in buddy,
    the VM can allocate pages below min watermark, at worst reducing the
    real number of pages to zero.  Even if the OOM killer kills some
    victim for freeing memory, it may not free memory if the exit path
    requires a new page resulting in livelock.
    
    This patch introduces a zone_page_state_snapshot() function
    (courtesy of Christoph) that takes a slightly more accurate view of
    an arbitrary vmstat counter.  It is used to read NR_FREE_PAGES while
    kswapd is awake to avoid the watermark being accidentally broken.
    The estimate is not perfect and may result in cache line bounces but
    is expected to be lighter than the IPI calls necessary to
    continually drain the per-cpu counters while kswapd is awake.
    
    Signed-off-by: Christoph Lameter <cl@...ux.com>
    Signed-off-by: Mel Gorman <mel@....ul.ie>
    Signed-off-by: Andrew Morton <akpm@...ux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@...ux-foundation.org>


Let me know if you need anything.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/