linux-kernel - Re: [PATCH] x86: Avoid intermixing cpu dump_stack output on multi-processor systems

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20120604142353.GI32472@redhat.com>
Date:	Mon, 4 Jun 2012 10:23:53 -0400
From:	Don Zickus <dzickus@...hat.com>
To:	Russ Anderson <rja@....com>
Cc:	linux-kernel@...r.kernel.org, x86@...nel.org,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>,
	"H. Peter Anvin" <hpa@...or.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
	rja@...ricas.sgi.com
Subject: Re: [PATCH] x86: Avoid intermixing cpu dump_stack output on
 multi-processor systems

On Fri, Jun 01, 2012 at 05:56:03PM -0500, Russ Anderson wrote:
> > I am curious, your NMI handler has locking wrapped around dump_stack,
> > shouldn't that serialize the output the way you want it?  Why isn't that
> > working?
> 
> Yes, you're right, it does.  It is working.  I'd forgotten that 
> the community kernel has uv_nmi_lock in uv_handle_nmi.  Must 
> be working too much with distro kernels.  :-)  But that doesn't
> help for all the other code paths than call dump_stack.

Sure, I agree.  But not every caller of dump_stack needs to dump for all
cpus.  I was just trying to avoid the ugly complicated code of
cmp_xchange you were requesting.

> 
> 
> > > FWIW, "Wait for up to 10 seconds for all CPUs to do the backtrace" on
> > > a 4096 cpu system isn't long enough.  :-)
> > 
> > Good point. :-)
> > 
> > > 
> > > > Whereas the lock you are proposing can be called in a mixture of NMI and
> > > > IRQ which could cause deadlocks I believe.
> > > 
> > > Since this is a lock just around the dump_stack printk, would 
> > > checking for forward progress and a timeout to catch any possible
> > > deadlock be sufficient?  In the unlikely case of a deadlock the
> > > lock gets broken and some of the cpu backtraces get intermixed.
> > > That is still a huge improvement over the current case where 
> > > all of the backtraces get intermixed.
> > 
> > I saw your new patch based on Frederick's input.  It seems to take care of
> > deadlock situations though you run into the starving lock problem that
> > ticketed spinlocks solved.  Which is why I am curious why moving the
> > locking one layer up to the NMI handler (which is where it is currently),
> > didn't fix your problem.
> 
> Locking in dump_stack would remove the need for uv_nmi_lock.

I agree.  I was just wondering if the added complexities were worth it for
the normal case.

If the uv_nmi_lock works, then I feel comfortable that your second patch
based on Frederic's suggestion will work too.  I just feel uncomfortable
with locking on a stack dump that should be reliable.  It is one thing to
do the locking in only NMI space or only IRQ space, but now we are
traversing both.  I don't think there will be any deadlocks (based on the
else path).

It could just be an overhyped paranoia of mine.  But that was my biggest
hestitation.

Cheers,
Don
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/