linux-kernel - Re: frequent lockups in 3.18rc4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+55aFxOKsK-E438QEOd5Tq2dUZjO8sOnQtpTtPmpBKngmgpkg@mail.gmail.com>
Date:	Mon, 1 Dec 2014 15:25:01 -0800
From:	Linus Torvalds <torvalds@...ux-foundation.org>
To:	Chris Mason <clm@...com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Dâniel Fraga <fragabr@...il.com>,
	Dave Jones <davej@...hat.com>,
	Sasha Levin <sasha.levin@...cle.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: frequent lockups in 3.18rc4

On Mon, Dec 1, 2014 at 3:08 PM, Chris Mason <clm@...com> wrote:
> I'm not sure if this is related, but running trinity here, I noticed it
> was stuck at 100% system time on every CPU.  perf report tells me we are
> spending all of our time in spin_lock under the sync system call.
>
> I think it's coming from contention in the bdi_queue_work() call from
> inside sync_inodes_sb, which is spin_lock_bh().

Please do a perf run with -g to get the call chain to make sure..

> I wonder if we're just spinning so hard on this one bh lock that we're
> starving the watchdog?

If it was that simple, we should see it in the actual soft-lockup stack trace.

That said, looking at the bdi_queue_work() function, I don't think you
should see any real contention there, although:

 - spin-lock debugging can make any bad situation about 10x worse by
making the spinlocks just that much more horrible from a performance
standpoint

 - the whole "complete(work->done)" thing seems to be pointlessly done
inside the spinlock, and that just seems horrible. Do you have a ton
of BDI's that might fail that BDI_registered thing?

 - even the "mod_delayed_work()" is dubious wrt the wb_lock. From what
I can tell, the spinlock is supposed to just protect the list.

So I think that bdi_queue_work() quite possibly is horribly broken
crap and *if* it really is contention on wb_lock, we could rewrite it
to not be so bad locking-wise.

That said, contention that happens with spinlock debugging enabled
really tends to fall under the heading of "that's your own fault".

                     Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/