linux-kernel - Re: [PATCH 05/10] block: remove per-queue plugging

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Thu, 17 Mar 2011 14:46:15 -0400
From:	Mike Snitzer <snitzer@...hat.com>
To:	Jens Axboe <jaxboe@...ionio.com>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"hch@...radead.org" <hch@...radead.org>
Subject: Re: [PATCH 05/10] block: remove per-queue plugging

On Thu, Mar 17 2011 at  2:31pm -0400,
Jens Axboe <jaxboe@...ionio.com> wrote:

> On 2011-03-17 16:51, Mike Snitzer wrote:
> > On Tue, Mar 08 2011 at  5:05pm -0500,
> > Mike Snitzer <snitzer@...hat.com> wrote:
> > 
> >> On Tue, Mar 08 2011 at  3:27pm -0500,
> >> Jens Axboe <jaxboe@...ionio.com> wrote:
> >>
> >>> On 2011-03-08 21:21, Mike Snitzer wrote:
> >>>> On Tue, Mar 08 2011 at  7:16am -0500,
> >>>> Jens Axboe <jaxboe@...ionio.com> wrote:
> >>>>
> >>>>> On 2011-03-03 23:13, Mike Snitzer wrote:
> >>>>>> I'm now hitting a lockdep issue, while running a 'for-2.6.39/stack-plug'
> >>>>>> kernel, when I try an fsync heavy workload to a request-based mpath
> >>>>>> device (the kernel ultimately goes down in flames, I've yet to look at
> >>>>>> the crashdump I took)
> >>>>>
> >>>>> Mike, can you re-run with the current stack-plug branch? I've fixed the
> >>>>> !CONFIG_BLOCK and rebase issues, and also added a change for this flush
> >>>>> on schedule event. It's run outside of the runqueue lock now, so
> >>>>> hopefully that should solve this one.
> >>>>
> >>>> Works for me, thanks.
> >>>
> >>> Super, thanks! Out of curiousity, did you use dm/md?
> >>
> >> Yes, I've been using a request-based DM multipath device.
> > 
> > 
> > Against latest 'for-2.6.39/core', I just ran that same fsync heavy
> > workload against XFS (ontop of a DM multipath volume).  ffsb induced the
> > following hangs (ripple effect causing NetworkManager to get hung up on
> > this data-only XFS volume, etc):
> 
> Ugh. Care to send the recipee for how to reproduce this? Essentially
> just looks like IO got stuck.

Here is the sequence to reproduce with the attached fsync-happy.ffsb
(I've been running the following in a KVM guest):

<create multipath device>
mkfs.xfs /dev/mapper/mpathb
mount /dev/mapper/mpathb /mnt/test
./ffsb fsync-happy.ffsb

And I just verified that the deadlock does _not_ seem to occur without
DM multipath -- by directly using an underlying SCSI device instead.

So multipath is exposing this somehow (could just be changing timing?).

Mike

p.s. though I did get this lockdep warning when unmounting the xfs
filesystem:

=================================
[ INFO: inconsistent lock state ]
2.6.38-rc6-snitm+ #8
---------------------------------
inconsistent {IN-RECLAIM_FS-R} -> {RECLAIM_FS-ON-W} usage.
umount/1524 [HC0[0]:SC0[0]:HE1:SE1] takes:
 (iprune_sem){+++++-}, at: [<ffffffff81114a22>] evict_inodes+0x2f/0x107
{IN-RECLAIM_FS-R} state was registered at:
  [<ffffffff810727c2>] __lock_acquire+0x3a4/0xd26
  [<ffffffff81073227>] lock_acquire+0xe3/0x110
  [<ffffffff81372fa2>] down_read+0x51/0x96
  [<ffffffff81114d57>] shrink_icache_memory+0x4a/0x215
  [<ffffffff810d1e48>] shrink_slab+0xe0/0x164
  [<ffffffff810d3e8f>] kswapd+0x5e7/0x9dc
  [<ffffffff8105fb7c>] kthread+0xa0/0xa8
  [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
irq event stamp: 73433
hardirqs last  enabled at (73433): [<ffffffff81070ffe>] debug_check_no_locks_freed+0x12e/0x145
hardirqs last disabled at (73432): [<ffffffff81070f13>] debug_check_no_locks_freed+0x43/0x145
softirqs last  enabled at (72996): [<ffffffff8104a1f1>] __do_softirq+0x1b4/0x1d3
softirqs last disabled at (72991): [<ffffffff81003b1c>] call_softirq+0x1c/0x28

other info that might help us debug this:
2 locks held by umount/1524:
 #0:  (&type->s_umount_key#24){++++++}, at: [<ffffffff81102a27>] deactivate_super+0x3d/0x4a
 #1:  (iprune_sem){+++++-}, at: [<ffffffff81114a22>] evict_inodes+0x2f/0x107

stack backtrace:
Pid: 1524, comm: umount Not tainted 2.6.38-rc6-snitm+ #8
Call Trace:
 [<ffffffff8107097f>] ? valid_state+0x17e/0x191
 [<ffffffff810712e8>] ? check_usage_backwards+0x0/0x81
 [<ffffffff81070ae4>] ? mark_lock+0x152/0x22d
 [<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
 [<ffffffff81070cc8>] ? lockdep_trace_alloc+0x99/0xbb
 [<ffffffff810fa98a>] ? kmem_cache_alloc+0x30/0x145
 [<ffffffffa014dcdd>] ? kmem_zone_alloc+0x69/0xb1 [xfs]
 [<ffffffffa014dd39>] ? kmem_zone_zalloc+0x14/0x35 [xfs]
 [<ffffffffa0147ed9>] ? _xfs_trans_alloc+0x27/0x64 [xfs]
 [<ffffffffa0148c97>] ? xfs_trans_alloc+0x9f/0xac [xfs]
 [<ffffffff810643b7>] ? up_read+0x23/0x3c
 [<ffffffffa0133000>] ? xfs_iunlock+0x7e/0xbc [xfs]
 [<ffffffffa014c140>] ? xfs_free_eofblocks+0xea/0x1f1 [xfs]
 [<ffffffffa014c707>] ? xfs_inactive+0x108/0x3a6 [xfs]
 [<ffffffff8106ff27>] ? lockdep_init_map+0xa6/0x11b
 [<ffffffffa015787f>] ? xfs_fs_evict_inode+0xf6/0xfe [xfs]
 [<ffffffff811147c6>] ? evict+0x24/0x8c
 [<ffffffff8111485f>] ? dispose_list+0x31/0xaf
 [<ffffffff81114ae3>] ? evict_inodes+0xf0/0x107
 [<ffffffff81101660>] ? generic_shutdown_super+0x5c/0xdf
 [<ffffffff8110170a>] ? kill_block_super+0x27/0x69
 [<ffffffff81101d89>] ? deactivate_locked_super+0x26/0x4b
 [<ffffffff81102a2f>] ? deactivate_super+0x45/0x4a
 [<ffffffff81118b87>] ? mntput_no_expire+0x105/0x10e
 [<ffffffff81119db6>] ? sys_umount+0x2d9/0x304
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81002b82>] ? system_call_fastpath+0x16/0x1b

View attachment "fsync-happy.ffsb" of type "text/plain" (1643 bytes)