Date:	Thu, 17 Mar 2011 11:51:21 -0400
From:	Mike Snitzer <snitzer@...hat.com>
To:	Jens Axboe <jaxboe@...ionio.com>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"hch@...radead.org" <hch@...radead.org>
Subject: Re: [PATCH 05/10] block: remove per-queue plugging

On Tue, Mar 08 2011 at  5:05pm -0500,
Mike Snitzer <snitzer@...hat.com> wrote:

> On Tue, Mar 08 2011 at  3:27pm -0500,
> Jens Axboe <jaxboe@...ionio.com> wrote:
> 
> > On 2011-03-08 21:21, Mike Snitzer wrote:
> > > On Tue, Mar 08 2011 at  7:16am -0500,
> > > Jens Axboe <jaxboe@...ionio.com> wrote:
> > > 
> > >> On 2011-03-03 23:13, Mike Snitzer wrote:
> > >>> I'm now hitting a lockdep issue, while running a 'for-2.6.39/stack-plug'
> > >>> kernel, when I try an fsync heavy workload to a request-based mpath
> > >>> device (the kernel ultimately goes down in flames, I've yet to look at
> > >>> the crashdump I took)
> > >>
> > >> Mike, can you re-run with the current stack-plug branch? I've fixed the
> > >> !CONFIG_BLOCK and rebase issues, and also added a change for this flush
> > >> on schedule event. It's run outside of the runqueue lock now, so
> > >> hopefully that should solve this one.
> > > 
> > > Works for me, thanks.
> > 
> > Super, thanks! Out of curiousity, did you use dm/md?
> 
> Yes, I've been using a request-based DM multipath device.


Against latest 'for-2.6.39/core', I just ran that same fsync heavy
workload against XFS (ontop of a DM multipath volume).  ffsb induced the
following hangs (ripple effect causing NetworkManager to get hung up on
this data-only XFS volume, etc):

XFS mounting filesystem dm-0
Ending clean XFS mount for filesystem: dm-0
mount used greatest stack depth: 3296 bytes left
ffsb used greatest stack depth: 2592 bytes left
INFO: task kswapd0:23 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kswapd0         D ffff880037b8f6e0  3656    23      2 0x00000000
 ffff880037b8f6d0 0000000000000046 ffff880037b8f630 ffffffff8107012f
 ffff880037b8e010 ffff880037b8ffd8 00000000001d21c0 ffff880037b90600
 ffff880037b90998 ffff880037b90990 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffff8107012f>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffffa0150df7>] ? kmem_alloc+0x69/0xb1 [xfs]
 [<ffffffff810fad18>] ? __kmalloc+0x14e/0x160
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa0138a18>] xfs_itruncate_finish+0x1fd/0x2bd [xfs]
 [<ffffffffa014f202>] xfs_free_eofblocks+0x1ac/0x1f1 [xfs]
 [<ffffffffa014f707>] xfs_inactive+0x108/0x3a6 [xfs]
 [<ffffffff8106ff27>] ? lockdep_init_map+0xa6/0x11b
 [<ffffffffa015a87f>] xfs_fs_evict_inode+0xf6/0xfe [xfs]
 [<ffffffff81114766>] evict+0x24/0x8c
 [<ffffffff811147ff>] dispose_list+0x31/0xaf
 [<ffffffff81114e92>] shrink_icache_memory+0x1e5/0x215
 [<ffffffff810d1e14>] shrink_slab+0xe0/0x164
 [<ffffffff810d3e5b>] kswapd+0x5e7/0x9dc
 [<ffffffff810d3874>] ? kswapd+0x0/0x9dc
 [<ffffffff8105fb7c>] kthread+0xa0/0xa8
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
 [<ffffffff813749d4>] ? restore_args+0x0/0x30
 [<ffffffff8105fadc>] ? kthread+0x0/0xa8
 [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10
4 locks held by kswapd0/23:
 #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810d1d71>] shrink_slab+0x3d/0x164
 #1:  (iprune_sem){++++.-}, at: [<ffffffff81114cf7>] shrink_icache_memory+0x4a/0x215
 #2:  (xfs_iolock_reclaimable){+.+.-.}, at: [<ffffffffa013615d>] xfs_ilock+0x30/0xb9 [xfs]
 #3:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
INFO: task NetworkManager:958 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
NetworkManager  D ffff88007a481288  3312   958      1 0x00000000
 ffff88007a481278 0000000000000046 ffff88007a4811d8 ffffffff8107012f
 ffff88007a480010 ffff88007a481fd8 00000000001d21c0 ffff88007b4f0f80
 ffff88007b4f1318 ffff88007b4f1310 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffff8107012f>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffffa0150df7>] ? kmem_alloc+0x69/0xb1 [xfs]
 [<ffffffff810fad18>] ? __kmalloc+0x14e/0x160
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
 [<ffffffff810fa9f7>] ? kmem_cache_alloc+0xd1/0x145
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa011c5fa>] xfs_bmap_finish+0x87/0x16a [xfs]
 [<ffffffffa01389b9>] xfs_itruncate_finish+0x19e/0x2bd [xfs]
 [<ffffffffa014f202>] xfs_free_eofblocks+0x1ac/0x1f1 [xfs]
 [<ffffffffa014f707>] xfs_inactive+0x108/0x3a6 [xfs]
 [<ffffffff8106ff27>] ? lockdep_init_map+0xa6/0x11b
 [<ffffffffa015a87f>] xfs_fs_evict_inode+0xf6/0xfe [xfs]
 [<ffffffff81114766>] evict+0x24/0x8c
 [<ffffffff811147ff>] dispose_list+0x31/0xaf
 [<ffffffff81114e92>] shrink_icache_memory+0x1e5/0x215
 [<ffffffff810d1e14>] shrink_slab+0xe0/0x164
 [<ffffffff810d3282>] try_to_free_pages+0x27f/0x495
 [<ffffffff810cb3fc>] __alloc_pages_nodemask+0x4e3/0x767
 [<ffffffff810700a3>] ? trace_hardirqs_off_caller+0x1f/0x9e
 [<ffffffff810f5b14>] alloc_pages_current+0xa7/0xca
 [<ffffffff810c515c>] __page_cache_alloc+0x85/0x8c
 [<ffffffff810cd420>] __do_page_cache_readahead+0xdb/0x1df
 [<ffffffff810cd545>] ra_submit+0x21/0x25
 [<ffffffff810c66e3>] filemap_fault+0x176/0x396
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffff810e15e7>] __do_fault+0x54/0x354
 [<ffffffff810709bf>] ? mark_lock+0x2d/0x22d
 [<ffffffff810e2493>] handle_pte_fault+0x2cf/0x6e8
 [<ffffffff810e004e>] ? __pte_alloc+0xc3/0xd0
 [<ffffffff810e2986>] handle_mm_fault+0xda/0xed
 [<ffffffff81377c28>] do_page_fault+0x3b4/0x3d6
 [<ffffffff8118e6de>] ? fsnotify_perm+0x69/0x75
 [<ffffffff8118e74b>] ? security_file_permission+0x2e/0x33
 [<ffffffff813739e6>] ? trace_hardirqs_off_thunk+0x3a/0x3c
 [<ffffffff81374be5>] page_fault+0x25/0x30
5 locks held by NetworkManager/958:
 #0:  (&mm->mmap_sem){++++++}, at: [<ffffffff81377a36>] do_page_fault+0x1c2/0x3d6
 #1:  (shrinker_rwsem){++++..}, at: [<ffffffff810d1d71>] shrink_slab+0x3d/0x164
 #2:  (iprune_sem){++++.-}, at: [<ffffffff81114cf7>] shrink_icache_memory+0x4a/0x215
 #3:  (xfs_iolock_reclaimable){+.+.-.}, at: [<ffffffffa013615d>] xfs_ilock+0x30/0xb9 [xfs]
 #4:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
INFO: task xfssyncd/dm-0:1346 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfssyncd/dm-0   D ffff880072cb1a20  4824  1346      2 0x00000000
 ffff880072cb1a10 0000000000000046 ffff880072cb1970 ffffffff8107012f
 ffff880072cb0010 ffff880072cb1fd8 00000000001d21c0 ffff88007b22ca00
 ffff88007b22cd98 ffff88007b22cd90 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffff8107012f>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffff81065905>] ? sched_clock_local+0x1c/0x82
 [<ffffffff810700a3>] ? trace_hardirqs_off_caller+0x1f/0x9e
 [<ffffffff810709bf>] ? mark_lock+0x2d/0x22d
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa0150cdd>] ? kmem_zone_alloc+0x69/0xb1 [xfs]
 [<ffffffffa014af66>] ? xfs_trans_add_item+0x50/0x5c [xfs]
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa0133289>] xfs_fs_log_dummy+0x76/0x7d [xfs]
 [<ffffffffa015cd3d>] xfs_sync_worker+0x37/0x6f [xfs]
 [<ffffffffa015ccb0>] xfssyncd+0x15b/0x1b1 [xfs]
 [<ffffffffa015cb55>] ? xfssyncd+0x0/0x1b1 [xfs]
 [<ffffffff8105fb7c>] kthread+0xa0/0xa8
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
 [<ffffffff813749d4>] ? restore_args+0x0/0x30
 [<ffffffff8105fadc>] ? kthread+0x0/0xa8
 [<ffffffff81003a20>] ? kernel_thread_helper+0x0/0x10
no locks held by xfssyncd/dm-0/1346.
INFO: task ffsb:1355 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ffsb            D 000000010002503a  3648  1355   1322 0x00000000
 ffff88007baffae8 0000000000000046 ffff88007baffa48 ffffffff00000000
 ffff88007bafe010 ffff88007bafffd8 00000000001d21c0 ffff880071df4680
 ffff880071df4a18 ffff880071df4a10 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffff810727da>] ? __lock_acquire+0x3bc/0xd26
 [<ffffffff81065a7a>] ? local_clock+0x41/0x5a
 [<ffffffff81024167>] ? pvclock_clocksource_read+0x4b/0xb4
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffff81065905>] ? sched_clock_local+0x1c/0x82
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
 [<ffffffff810fa9f7>] ? kmem_cache_alloc+0xd1/0x145
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa0150cdd>] ? kmem_zone_alloc+0x69/0xb1 [xfs]
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa01566f8>] xfs_file_fsync+0x166/0x1e6 [xfs]
 [<ffffffff81122a8b>] vfs_fsync_range+0x54/0x7c
 [<ffffffff81122b15>] vfs_fsync+0x1c/0x1e
 [<ffffffff81122b45>] do_fsync+0x2e/0x43
 [<ffffffff81122b81>] sys_fsync+0x10/0x14
 [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
2 locks held by ffsb/1355:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
INFO: task ffsb:1364 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ffsb            D 0000000100024ff6  3776  1364   1322 0x00000000
 ffff880027d25ae8 0000000000000046 ffff880027d25a48 ffffffff00000000
 ffff880027d24010 ffff880027d25fd8 00000000001d21c0 ffff88002839c8c0
 ffff88002839cc58 ffff88002839cc50 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffffa0150df7>] ? kmem_alloc+0x69/0xb1 [xfs]
 [<ffffffff810fad18>] ? __kmalloc+0x14e/0x160
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
 [<ffffffff810fa9f7>] ? kmem_cache_alloc+0xd1/0x145
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa0150cdd>] ? kmem_zone_alloc+0x69/0xb1 [xfs]
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa01566f8>] xfs_file_fsync+0x166/0x1e6 [xfs]
 [<ffffffff81122a8b>] vfs_fsync_range+0x54/0x7c
 [<ffffffff81122b15>] vfs_fsync+0x1c/0x1e
 [<ffffffff81122b45>] do_fsync+0x2e/0x43
 [<ffffffff81122b81>] sys_fsync+0x10/0x14
 [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
2 locks held by ffsb/1364:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]

(and many more ffsb processes hung similar to the 2 above)

I just attempted a git command against the root volume, it hung:

git             D 0000000100252022  3440  1471   1461 0x00000004
 ffff88003ad611d8 0000000000000046 ffff88003ad61138 ffffffff00000000
 ffff88003ad60010 ffff88003ad61fd8 00000000001d21c0 ffff88003b498d40
 ffff88003b4990d8 ffff88003b4990d0 00000000001d21c0 00000000001d21c0
Call Trace:
 [<ffffffffa013e958>] xlog_wait+0x60/0x78 [xfs]
 [<ffffffff810404de>] ? default_wake_function+0x0/0x14
 [<ffffffff81373c5f>] ? _raw_spin_lock+0x62/0x69
 [<ffffffffa013f874>] xlog_state_get_iclog_space+0x9e/0x22c [xfs]
 [<ffffffffa013fb73>] xlog_write+0x171/0x4ae [xfs]
 [<ffffffffa0150df7>] ? kmem_alloc+0x69/0xb1 [xfs]
 [<ffffffff810fad18>] ? __kmalloc+0x14e/0x160
 [<ffffffffa013ff04>] xfs_log_write+0x54/0x7e [xfs]
 [<ffffffffa014b5b0>] xfs_trans_commit_iclog+0x195/0x2d8 [xfs]
 [<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
 [<ffffffff810fa9f7>] ? kmem_cache_alloc+0xd1/0x145
 [<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa014b7bc>] _xfs_trans_commit+0xc9/0x206 [xfs]
 [<ffffffffa011c5fa>] xfs_bmap_finish+0x87/0x16a [xfs]
 [<ffffffffa01389b9>] xfs_itruncate_finish+0x19e/0x2bd [xfs]
 [<ffffffffa014f202>] xfs_free_eofblocks+0x1ac/0x1f1 [xfs]
 [<ffffffffa014f707>] xfs_inactive+0x108/0x3a6 [xfs]
 [<ffffffff8106ff27>] ? lockdep_init_map+0xa6/0x11b
 [<ffffffffa015a87f>] xfs_fs_evict_inode+0xf6/0xfe [xfs]
 [<ffffffff81114766>] evict+0x24/0x8c
 [<ffffffff811147ff>] dispose_list+0x31/0xaf
 [<ffffffff81114e92>] shrink_icache_memory+0x1e5/0x215
 [<ffffffff810d1e14>] shrink_slab+0xe0/0x164
 [<ffffffff810d3282>] try_to_free_pages+0x27f/0x495
 [<ffffffff810cb3fc>] __alloc_pages_nodemask+0x4e3/0x767
 [<ffffffff810700a3>] ? trace_hardirqs_off_caller+0x1f/0x9e
 [<ffffffff810f5b14>] alloc_pages_current+0xa7/0xca
 [<ffffffff810c515c>] __page_cache_alloc+0x85/0x8c
 [<ffffffff810cd420>] __do_page_cache_readahead+0xdb/0x1df
 [<ffffffff8106fa35>] ? lock_release_holdtime+0x2c/0xd7
 [<ffffffff810cd545>] ra_submit+0x21/0x25
 [<ffffffff810cd92c>] ondemand_readahead+0x1e3/0x1f6
 [<ffffffff810cd9b8>] page_cache_async_readahead+0x79/0x82
 [<ffffffff810c6633>] filemap_fault+0xc6/0x396
 [<ffffffff81070ece>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffff810e15e7>] __do_fault+0x54/0x354
 [<ffffffff810709bf>] ? mark_lock+0x2d/0x22d
 [<ffffffff810e2493>] handle_pte_fault+0x2cf/0x6e8
 [<ffffffff810e004e>] ? __pte_alloc+0xc3/0xd0
 [<ffffffff810e2986>] handle_mm_fault+0xda/0xed
 [<ffffffff81377c28>] do_page_fault+0x3b4/0x3d6
 [<ffffffff810700a3>] ? trace_hardirqs_off_caller+0x1f/0x9e
 [<ffffffff8107012f>] ? trace_hardirqs_off+0xd/0xf
 [<ffffffff81065a7a>] ? local_clock+0x41/0x5a
 [<ffffffff813739e6>] ? trace_hardirqs_off_thunk+0x3a/0x3c
 [<ffffffff81374be5>] page_fault+0x25/0x30

And here is the summary of all the locks (via sysrq-t):

Showing all locks held in the system:
2 locks held by kworker/0:1/10:
 #0:  (xfsdatad){++++..}, at: [<ffffffff81059fba>] process_one_work+0x18a/0x37f
 #1:  ((&ioend->io_work)){+.+...}, at: [<ffffffff81059fba>] process_one_work+0x18a/0x37f
4 locks held by kswapd0/23:
 #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810d1d71>] shrink_slab+0x3d/0x164
 #1:  (iprune_sem){++++.-}, at: [<ffffffff81114cf7>] shrink_icache_memory+0x4a/0x215
 #2:  (xfs_iolock_reclaimable){+.+.-.}, at: [<ffffffffa013615d>] xfs_ilock+0x30/0xb9 [xfs]
 #3:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by multipathd/659:
 #0:  (&u->readlock){+.+.+.}, at: [<ffffffff813533dd>] unix_dgram_recvmsg+0x5a/0x27f
5 locks held by NetworkManager/958:
 #0:  (&mm->mmap_sem){++++++}, at: [<ffffffff81377a36>] do_page_fault+0x1c2/0x3d6
 #1:  (shrinker_rwsem){++++..}, at: [<ffffffff810d1d71>] shrink_slab+0x3d/0x164
 #2:  (iprune_sem){++++.-}, at: [<ffffffff81114cf7>] shrink_icache_memory+0x4a/0x215
 #3:  (xfs_iolock_reclaimable){+.+.-.}, at: [<ffffffffa013615d>] xfs_ilock+0x30/0xb9 [xfs]
 #4:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by agetty/1099:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1101:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1103:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1105:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1107:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1109:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by mingetty/1111:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
1 lock held by bash/1313:
 #0:  (&tty->atomic_read_lock){+.+.+.}, at: [<ffffffff8123c132>] n_tty_read+0x284/0x7ba
2 locks held by ffsb/1355:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1358:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by ffsb/1359:
 #0:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1362:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1364:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1365:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1366:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1367:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by ffsb/1368:
 #0:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1371:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1372:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1373:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1374:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1375:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1376:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1377:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1378:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1380:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1381:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1383:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
4 locks held by ffsb/1384:
 #0:  (&sb->s_type->i_mutex_key#13/1){+.+.+.}, at: [<ffffffff8110b9ba>] do_unlinkat+0x67/0x165
 #1:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81109eaf>] vfs_unlink+0x4f/0xcc
 #2:  (&(&ip->i_lock)->mr_lock/2){+.+...}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
 #3:  (&(&ip->i_lock)->mr_lock/3){+.+...}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1385:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1386:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1387:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1388:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: 
2 locks held by ffsb/1389:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by ffsb/1390:
 #0:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1391:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1392:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1393:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by ffsb/1394:
 #0:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1395:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1396:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1397:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1398:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1399:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1400:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1402:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1403:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
1 lock held by ffsb/1404:
 #0:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1405:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1406:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1407:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1409:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1410:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1411:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1412:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1413:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1414:
 #0:  (xfs_iolock_active){++++.+}, at: [<ffffffffa0136079>] xfs_ilock_nowait+0x2b/0xdf [xfs]
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1416:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by ffsb/1417:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff81122a7e>] vfs_fsync_range+0x47/0x7c
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
3 locks held by ffsb/1418:
 #0:  (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [<ffffffff8110a688>] do_last+0xb8/0x2f9
 #1:  (&(&ip->i_lock)->mr_lock/1){+.+.+.}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
 #2:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa01360a7>] xfs_ilock_nowait+0x59/0xdf [xfs]
2 locks held by flush-253:0/1350:
 #0:  (&type->s_umount_key#24){.+.+.+}, at: [<ffffffff8111f509>] writeback_inodes_wb+0xce/0x13d
 #1:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
5 locks held by git/1471:
 #0:  (&mm->mmap_sem){++++++}, at: [<ffffffff81377a36>] do_page_fault+0x1c2/0x3d6
 #1:  (shrinker_rwsem){++++..}, at: [<ffffffff810d1d71>] shrink_slab+0x3d/0x164
 #2:  (iprune_sem){++++.-}, at: [<ffffffff81114cf7>] shrink_icache_memory+0x4a/0x215
 #3:  (xfs_iolock_reclaimable){+.+.-.}, at: [<ffffffffa013615d>] xfs_ilock+0x30/0xb9 [xfs]
 #4:  (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffffa0136190>] xfs_ilock+0x63/0xb9 [xfs]
2 locks held by bash/1472:
 #0:  (sysrq_key_table_lock){......}, at: [<ffffffff81242275>] __handle_sysrq+0x28/0x15c
 #1:  (tasklist_lock){.+.+..}, at: [<ffffffff8107062c>] debug_show_all_locks+0x52/0x19b
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists