linux-kernel - Re: [PATCH -next] locking/rwsem: don't spin in heavy contention

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1425680663.19505.65.camel@stgolabs.net>
Date:	Fri, 06 Mar 2015 14:24:23 -0800
From:	Davidlohr Bueso <dave@...olabs.net>
To:	Dave Chinner <david@...morbit.com>
Cc:	Ming Lei <ming.lei@...onical.com>, linux-kernel@...r.kernel.org,
	"Peter Zijlstra (Intel)" <peterz@...radead.org>,
	Jason Low <jason.low2@...com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Michel Lespinasse <walken@...gle.com>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Tim Chen <tim.c.chen@...ux.intel.com>,
	Ingo Molnar <mingo@...nel.org>, Theodore Ts'o <tytso@....edu>
Subject: Re: [PATCH -next] locking/rwsem: don't spin in heavy contention

On Sat, 2015-03-07 at 08:54 +1100, Dave Chinner wrote:
> On Fri, Mar 06, 2015 at 11:13:10PM +0800, Ming Lei wrote:
> > Before commit b3fd4f03ca0b995(locking/rwsem: Avoid deceiving lock
> > spinners), rwsem_spin_on_owner() returns false if the owner is changed.
> > This commit just returns true under the situation, then kernel
> > softlock can be triggered easily in xfstest.
> > 
> > So this patch recovers to previous behaviour, and it should be
> > reasonable to stop spining in case of heavy contention.
> > 
> > The soft lockup can be reproduced easily in xfstests(generic/299)
> > over ext4:
> > 
> > [  236.417011] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/5:80:3288]
> > [  236.417011] Modules linked in: nbd ipv6 kvm_intel kvm serio_raw
> > [  236.417011] CPU: 5 PID: 3288 Comm: kworker/5:80 Not tainted 4.0.0-rc1-next-20150303+ #69
> > [  236.417011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> > [  236.417011] Workqueue: dio/sda dio_aio_complete_work
> > [  236.417011] task: ffff8800b87c0000 ti: ffff8800b703c000 task.ti: ffff8800b703c000
> > [  236.417011] RIP: 0010:[<ffffffff81083c20>]  [<ffffffff81083c20>] __rcu_read_unlock+0x47/0x55
> > [  236.417011] RSP: 0018:ffff8800b703fb98  EFLAGS: 00000246
> > [  236.417011] RAX: 0000000000000000 RBX: ffff8800b703fb48 RCX: 000000000003b080
> > [  236.417011] RDX: fffffffe00000001 RSI: ffff880231f03a20 RDI: ffff8800bb755568
> > [  236.417011] RBP: ffff8800b703fba8 R08: ffff880227908078 R09: ffff8800b87c0000
> > [  236.417011] R10: 0000000000000001 R11: 0000000000000020 R12: ffff8800b703c000
> > [  236.417011] R13: ffff8800b87c0000 R14: 000000000000000f R15: 0000000000000101
> > [  236.417011] FS:  0000000000000000(0000) GS:ffff88023eca0000(0000) knlGS:0000000000000000
> > [  236.417011] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [  236.417011] CR2: 00007f549369f948 CR3: 00000000ba891000 CR4: 00000000000007e0
> > [  236.417011] Stack:
> > [  236.417011]  fffffffe00000001 ffff8800bb755568 ffff8800b703fbc8 ffffffff81073917
> > [  236.417011]  ffff8800bb755568 ffff8800bb755584 ffff8800b703fc48 ffffffff814d1ba4
> > [  236.417011]  ffff8800b703fbe8 ffff8800b87c0000 ffff8800b703fc78 ffffffff811cf51b
> > [  236.417011] Call Trace:
> > [  236.417011]  [<ffffffff81073917>] rwsem_spin_on_owner+0x2b/0x79
> > [  236.417011]  [<ffffffff814d1ba4>] rwsem_down_write_failed+0xc0/0x2f1
> > [  236.417011]  [<ffffffff811cf51b>] ? start_this_handle+0x494/0x4bd
> > [  236.417011]  [<ffffffff810d1149>] ? trace_preempt_on+0x12/0x2f
> > [  236.417011]  [<ffffffff812ae8f3>] call_rwsem_down_write_failed+0x13/0x20
> > [  236.417011]  [<ffffffff814d1623>] ? down_write+0x24/0x33
> > [  236.417011]  [<ffffffff81199404>] ext4_map_blocks+0x236/0x3cb
> > [  236.417011]  [<ffffffff811bb407>] ? ext4_convert_unwritten_extents+0xd2/0x19c
> > [  236.417011]  [<ffffffff811bcae4>] ? __ext4_journal_start_sb+0x77/0xb8
> > [  236.417011]  [<ffffffff811bb42e>] ext4_convert_unwritten_extents+0xf9/0x19c
> > [  236.417011]  [<ffffffff8119e214>] ext4_put_io_end+0x3a/0x5d
> > [  236.417011]  [<ffffffff81197268>] ext4_end_io_dio+0x2a/0x2c
> > [  236.417011]  [<ffffffff8116418c>] dio_complete+0x97/0x12d
> > [  236.417011]  [<ffffffff81164333>] dio_aio_complete_work+0x21/0x23
> 
> If you're getting stuff there, I'd be looking for a bug in ext4, not
> the rwsem code. There's no way there should be enough unwritten
> extent conversion pending to lock up the system for that length of
> time. Especially considering the test has concurrent truncates
> running which should drain the entire IO queue every couple of
> seconds at worst....

FYI, this issue is being handled here:
https://lkml.org/lkml/2015/3/6/811

Thanks,
Davidlohr

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/