linux-kernel - RE: kdump always hangs in rcu_barrier() -> wait_for

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <MW2PR2101MB1787FF5912C90FC4FFEF3993BFBA9@MW2PR2101MB1787.namprd21.prod.outlook.com>
Date:   Thu, 28 Jan 2021 07:28:20 +0000
From:   Dexuan Cui <decui@...rosoft.com>
To:     "paulmck@...nel.org" <paulmck@...nel.org>,
        Neeraj Upadhyay <neeraju@...eaurora.org>
CC:     "boqun.feng@...il.com" <boqun.feng@...il.com>,
        Ingo Molnar <mingo@...hat.com>,
        "rcu@...r.kernel.org" <rcu@...r.kernel.org>,
        vkuznets <vkuznets@...hat.com>,
        Michael Kelley <mikelley@...rosoft.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: kdump always hangs in rcu_barrier() -> wait_for_completion()

> From: Paul E. McKenney <paulmck@...nel.org>
> Sent: Thursday, November 26, 2020 3:55 PM
> To: Dexuan Cui <decui@...rosoft.com>
> Cc: boqun.feng@...il.com; Ingo Molnar <mingo@...hat.com>;
> rcu@...r.kernel.org; vkuznets <vkuznets@...hat.com>; Michael Kelley
> <mikelley@...rosoft.com>; linux-kernel@...r.kernel.org
> Subject: Re: kdump always hangs in rcu_barrier() -> wait_for_completion()
> 
> On Thu, Nov 26, 2020 at 10:59:19PM +0000, Dexuan Cui wrote:
> > > From: Paul E. McKenney <paulmck@...nel.org>
> > > Sent: Thursday, November 26, 2020 1:42 PM
> > >
> > > > > Another possibility is that rcu_state.gp_kthread is non-NULL, but that
> > > > > something else is preventing RCU grace periods from completing, but in
> > > >
> > > > It looks like somehow the scheduling is not working here: in rcu_barrier()
> > > > , if I replace the wait_for_completion() with
> > > > wait_for_completion_timeout(&rcu_state.barrier_completion, 30*HZ),
> the
> > > > issue persists.
> > >
> > > Have you tried using sysreq-t to see what the various tasks are doing?
> >
> > Will try it.
> >
> > BTW, this is a "Generation 2" VM on Hyper-V, meaning sysrq only starts to
> > work after the Hyper-V para-virtualized keyboard driver loads... So, at this
> > early point, sysrq is not working. :-( I'll have to hack the code and use a
> > virtual NMI interrupt to force the sysrq handler to be called.
> 
> Whatever works!
> 
> > > Having interrupts disabled on all CPUs would have the effect of disabling
> > > the RCU CPU stall warnings.
> > > 							Thanx, Paul
> >
> > I'm sure the interrupts are not disabled. Here the VM only has 1 virtual CPU,
> > and when the hang issue happens the virtual serial console is still responding
> > when I press Enter (it prints a new line) or Ctrl+C (it prints ^C).
> >
> > Here the VM does not use the "legacy timers" (PIT, Local APIC timer, etc.) at
> all.
> > Instead, the VM uses the Hyper-V para-virtualized timers. It looks the
> Hyper-V
> > timer never fires in the kdump kernel when the hang issue happens. I'm
> > looking into this... I suspect this hang issue may only be specific to Hyper-V.
> 
> Fair enough, given that timers not working can also suppress RCU CPU
> stall warnings.  ;-)
> 
> 							Thanx, Paul

FYI: the issue has been fixed by this fix:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fff7b5e6ee63c5d20406a131b260c619cdd24fd1

Thanks,
-- Dexuan