linux-kernel - Re: rcu: endless stalls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1339566974.7472.45.camel@marge.simpson.net>
Date:	Wed, 13 Jun 2012 07:56:14 +0200
From:	Mike Galbraith <efault@....de>
To:	Hugh Dickins <hughd@...gle.com>
Cc:	paulmck@...ux.vnet.ibm.com, LKML <linux-kernel@...r.kernel.org>,
	Miklos Szeredi <mszeredi@...e.cz>
Subject: Re: rcu: endless stalls

On Tue, 2012-06-12 at 21:31 -0700, Hugh Dickins wrote: 
> On Wed, 13 Jun 2012, Mike Galbraith wrote:
> > On Mon, 2012-06-11 at 20:10 +0200, Mike Galbraith wrote: 
> > > On Mon, 2012-06-11 at 11:01 -0700, Paul E. McKenney wrote: 
> > 
> > > > > 2aa15890 - mm: prevent concurrent unmap_mapping_range() on the same inode
> > > > 
> > > > I confess, you lost me on this one.  You believe that this commit is
> > > > the cause of the RCU CPU stall warnings?
> > > 
> > > 4096 tasks on 4096 CPUs exit (well, try to) simultaneously.
> > > 
> > > Call Trace:
> > >                 __mutex_lock_slowpath+0x94/0x150
> > >                 mutex_lock+0x1a/0x40
> > >                 unlink_file_vma+0x3f/0xf0
> > >                 free_pgtables+0x40/0x100
> > >                 exit_mmap+0xb0/0x120
> > >                 mmput+0x49/0x120
> > >                 exit_mm+0x122/0x160
> > >                 do_exit+0x179/0x8d0
> > >                 do_group_exit+0x3d/0xb0
> > >                 sys_exit_group+0x12/0x20
> > > 
> > > Monster box dies screaming.
> > 
> > That commit landed in stable, box with way too many cores (NR_CPUS=0!!)
> > chokes instantly with loads of spinners.  Ok, so zillion CPUs grabbing a
> > mutex in lockstep is a bad idea (_having_ zillion?), but is there pilot
> > error involved in a logjam like this?
> 
> Surely some mistake...

Yup.

> I can't find any mention of which kernel release you're talking about.
> 
> But Miklos's 2aa15890 unmap_mutex was introduced in 2.6.38 and removed
> in 3.0, when PeterZ converted i_mmap_lock to i_mmap_mutex, and removed
> the need for the additional unmap_mutex.
> 
> The unmap_mutex would never have been taken in unlink_file_vma(),
> shown in your stacktrace above: it was for truncation and invalidation.
> 
> The likely mutex in unlink_file_vma() would be the i_mmap_mutex.
> So I expect you're talking about a 3.0 or later kernel.

Yeah.

> But then why would someone "backport" Miklos's patch to stable for it?
> 
> You lost me too!

Sorry, /me is hopping around vigorously between too many darn trees,
it's already there, not a stable addition.

Question remains though.  Maybe the box hit some other problem that led
to death by RCU gripage, but the info I received indicated the box was
in the midst of a major spin-fest.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/