linux-kernel - Re: 3.9-rc5: Encountedred INFO: rcu_sched self-detected stall on CPU due to 09a9f1d27

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130415175929.GB30583@redhat.com>
Date:	Mon, 15 Apr 2013 13:59:29 -0400
From:	Vivek Goyal <vgoyal@...hat.com>
To:	Michel Lespinasse <walken@...gle.com>
Cc:	linux kernel mailing list <linux-kernel@...r.kernel.org>,
	Hugh Dickins <hughd@...gle.com>,
	Rik van Riel <riel@...hat.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: 3.9-rc5: Encountedred INFO: rcu_sched self-detected stall on CPU
 due to 09a9f1d27

CCing akpm.

Vivek

On Mon, Apr 15, 2013 at 01:34:24PM -0400, Vivek Goyal wrote:
> On Mon, Apr 15, 2013 at 12:35:52PM -0400, Vivek Goyal wrote:
> 
> [..]
> > > My first guess would be that mmap_sem is held during exec, so you
> > > can't have __mm_populate() try holding it recursively.
> > 
> > I think it is not mmap_sem as even with VM_LOCKED, we take mmap_sem
> > and things are fine. 
> > 
> > So things work till 3.8 and break in 3.8-rc1 (with both VM_LOCKED and
> > VM_POPULATE specifed). I will do git bisect and try to figure out which
> > is first commit which has the issue.
> 
> Ok, following seems to be first bad commit.
> 
> commit bebeb3d68b24bb4132d452c5707fe321208bcbcd
> Author: Michel Lespinasse <walken@...gle.com>
> Date:   Fri Feb 22 16:32:37 2013 -0800
> 
>     mm: introduce mm_populate() for populating new vmas
> 
> I have locked down /sbin/kexec. And I get following traceback after a while.
> 
> Thanks
> Vivek
> 
> [   93.130001] INFO: rcu_sched self-detected stall on CPU[   93.131007]
> INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 3,
> t=60002 jiffies, g=2786, c=2785, q=10480)
> [   93.131016] Task dump for CPU 2:
> [   93.131018] kexec           R  running task        0  4861   2982
> 0x00000088
> [   93.131022]  ffff88013252bc68 ffffffff8109ab12 00000000ffffffff
> 0000000000000000
> [   93.131026]  ffff88013252bc58 ffff88013a43c180 ffffffff81881edc
> 00007feed583221a
> [   93.131028]  00007feed5832000 0000000000000000 ffff88013a43c828
> ffff88013a43c180
> [   93.131032] Call Trace:
> [   93.131032]  [<ffffffff8109ab12>] ? mark_held_locks+0x82/0x130
> [   93.131040]  [<ffffffff81881edc>] ? retint_restore_args+0xe/0xe
> [   93.131045]  [<ffffffff8109ab12>] ? mark_held_locks+0x82/0x130
> [   93.131047]  [<ffffffff81881edc>] ? retint_restore_args+0xe/0xe
> [   93.131049]  [<ffffffff8109acc5>] ?
> trace_hardirqs_on_caller+0x105/0x190
> [   93.131051]  [<ffffffff8132913e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [   93.131055]  [<ffffffff81881edc>] ? retint_restore_args+0xe/0xe
> [   93.131058]  [<ffffffff81127898>] ? __get_user_pages+0x1d8/0x540
> [   93.131061]  [<ffffffff81129fb7>] ? __mlock_vma_pages_range+0x67/0x70
> [   93.131064]  [<ffffffff8112a721>] ? __mm_populate+0x71/0x150
> [   93.131066]  [<ffffffff8111d01f>] ? vm_mmap_pgoff+0xdf/0xf0
> [   93.131071]  [<ffffffff8112c839>] ? sys_mmap_pgoff+0xa9/0x180
> [   93.131073]  [<ffffffff8109acc5>] ?
> trace_hardirqs_on_caller+0x105/0x190
> [   93.131076]  [<ffffffff8132913e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [   93.131078]  [<ffffffff81006972>] ? sys_mmap+0x22/0x30
> [   93.131081]  [<ffffffff8188a342>] ? system_call_fastpath+0x16/0x1b
> 
> [   93.130001]  { 2}  (t=60153 jiffies g=2786 c=2785 q=10480)
> [   93.130001] Pid: 4861, comm: kexec Not tainted 3.8.0+ #270
> [   93.130001] Call Trace:
> [   93.130001]  <IRQ>  [<ffffffff810ce69a>]
> rcu_check_callbacks+0x21a/0x760
> [   93.130001]  [<ffffffff810d22ec>] ? acct_account_cputime+0x1c/0x20
> [   93.130001]  [<ffffffff81077570>] ? account_system_time+0x160/0x230
> [   93.130001]  [<ffffffff8104f388>] update_process_times+0x48/0x80
> [   93.130001]  [<ffffffff810934ad>] tick_sched_handle+0x3d/0x50
> [   93.130001]  [<ffffffff810936b5>] tick_sched_timer+0x45/0x70
> [   93.130001]  [<ffffffff81067661>] __run_hrtimer+0x81/0x220
> [   93.130001]  [<ffffffff81093670>] ? tick_nohz_handler+0xa0/0xa0
> [   93.130001]  [<ffffffff8108ce7c>] ? ktime_get_update_offsets+0x4c/0xd0
> [   93.130001]  [<ffffffff81067fb7>] hrtimer_interrupt+0xf7/0x250
> [   93.130001]  [<ffffffff8188bdb9>] smp_apic_timer_interrupt+0x69/0x99
> [   93.130001]  [<ffffffff8188af6f>] apic_timer_interrupt+0x6f/0x80
> [   93.130001]  <EOI>  [<ffffffff81881edc>] ? retint_restore_args+0xe/0xe
> [   93.130001]  [<ffffffff81127898>] ? __get_user_pages+0x1d8/0x540
> [   93.130001]  [<ffffffff81129fb7>] __mlock_vma_pages_range+0x67/0x70
> [   93.130001]  [<ffffffff8112a721>] __mm_populate+0x71/0x150
> [   93.130001]  [<ffffffff8111d01f>] vm_mmap_pgoff+0xdf/0xf0
> [   93.130001]  [<ffffffff8112c839>] sys_mmap_pgoff+0xa9/0x180
> [   93.130001]  [<ffffffff8109acc5>] ?
> trace_hardirqs_on_caller+0x105/0x190
> [   93.130001]  [<ffffffff8132913e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [   93.130001]  [<ffffffff81006972>] sys_mmap+0x22/0x30
> [   93.130001]  [<ffffffff8188a342>] system_call_fastpath+0x16/0x1b
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/