linux-kernel - Re: Linux 3.0-rc5 doesnt boot and hangs at rcu_sched

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAM=x7Sa1Tb2Vn-RL=AFsJZKFYbnHxOp2Qpn9O3D7p6FVFtuf6w@mail.gmail.com>
Date:	Mon, 11 Jul 2011 19:12:25 +0530
From:	RKK <kulkarni.ravi4@...il.com>
To:	paulmck@...ux.vnet.ibm.com, maciej.rutecki@...il.com
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Linux 3.0-rc5 doesnt boot and hangs at rcu_sched_state ()

Hi Paul

On Mon, Jul 11, 2011 at 3:48 PM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Mon, Jul 11, 2011 at 10:46:30AM +0530, RKK wrote:
>> Hi Paul,
>>
>> On Mon, Jul 11, 2011 at 9:21 AM, Paul E. McKenney
>> <paulmck@...ux.vnet.ibm.com> wrote:
>> > On Sat, Jul 09, 2011 at 09:01:31AM -0700, Paul E. McKenney wrote:
>> >> On Wed, Jun 29, 2011 at 06:56:35PM +0530, RKK wrote:
>> >> > Hello,
>> >> > I tried booting Linux3.0.rc5 on my machine today but everytime it
>> >> > hangs after this message
>> >> >
>> >> > a)starting configure read only root support
>> >> >
>> >> > after this waiting for sometime then this message appears
>> >> >
>> >> > b)INFO rcu_sched_state: RCU stalls CPU/disks
>> >> >
>> >> > i tried to read the Documentation/RCU and enable CONFIG_RCU_TRACE but
>> >> > dint know how to proceed further  .
>> >> >
>> >> > i tried repeating this 4-5 times , one thing i observed that is
>> >> > appearance of rcu_sched_state is intermittent but everytime the boot
>> >> > stops/hangs at a) message .
>> >>
>> >> Can you set up the SysRq key as described in Documentation/sysrq.txt?
>> >> This might help you get some information about what the system is doing
>> >> during the wait time.
>> >>
>> >> My guess is that your kernel is spinning with interrupts disabled, and
>> >> that RCU eventually tries to complain about this.  The possible causes
>> >> of this are listed in Documentation/RCU/stallwarn.txt.
>> >
>> > Could you please try out this patch and see if it helps?
>> >
>> >                                                        Thanx, Paul
>
> [ . . . ]
>
>> Please give me some time as im away. i will test the patch and  get
>> back to you by today evening .
>> Warm Regards
>> Ravi Kulkarni.
>
> Just as well -- I fat-fingered the patch creation.  :-/
>
> Please see below for the real patch.
>
>                                                        Thanx, Paul
>
> ------------------------------------------------------------------------
>
> rcu: Prevent RCU callbacks from executing during early boot
>
> Under some rare but real combinations of configuration parameters, RCU
> callbacks are posted during early boot that use kernel facilities that
> are not yet initialized.  Therefore, when these callbacks are invoked,
> hard hangs and crashes ensue.  This commit therefore prevents RCU
> callbacks from being invoked until after the scheduler is up and running.
>
> It might well turn out that a better approach is to identify the specific
> RCU callbacks that are causing this problem, but that discussion will
> wait until such time as someone really needs an RCU callback to be
> invoked during early boot.
>
> Reported-by: julie Sullivan <kernelmail.jms@...il.com>
> Tested-by: julie Sullivan <kernelmail.jms@...il.com>
> Signed-off-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
>
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 7e59ffb..4c0210f 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1467,7 +1467,7 @@ static void rcu_process_callbacks(struct softirq_action *unused)
>  */
>  static void invoke_rcu_callbacks(struct rcu_state *rsp, struct rcu_data *rdp)
>  {
> -       if (likely(!rsp->boost)) {
> +       if (likely(rcu_scheduler_active && !rsp->boost)) {
>                rcu_do_batch(rsp, rdp);
>                return;
>        }
> diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> index 14dc7dd..ca3c6dc 100644
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -1703,7 +1703,7 @@ static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
>
>  static void invoke_rcu_callbacks_kthread(void)
>  {
> -       WARN_ON_ONCE(1);
> +       WARN_ON_ONCE(rcu_scheduler_active);
>  }
>
>  static void rcu_preempt_boost_start_gp(struct rcu_node *rnp)
>

The above patch fixes the bug and now 3.0.rc5 is bootable :). thanks.

maciej rutecki,

can we close the the below bugzilla entry ?
https://bugzilla.kernel.org/show_bug.cgi?id=38732

Warm regards,
Ravi Kulkarni.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/