| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20111020224214.GC22772@hostway.ca> Date: Thu, 20 Oct 2011 15:42:14 -0700 From: Simon Kirby <sim@...tway.ca> To: Christoph Hellwig <hch@...radead.org> Cc: xfs@....sgi.com, linux-kernel@...r.kernel.org Subject: XFS read hangs in 3.1-rc10 Hello! We're seeing a big problem with read latency in 3.1-rc10 and 3.1-rc7 when compared with 2.6.36, to the point that the hung task detector is spitting out traces when set to 30 seconds. At first, we noticed this on SATA disks on a PERC5 controller and figured it was just really bad RAID latency, but it also seems to be happening on an SSD box. This is kind of a pain to debug since running "ps" also seems to cause reclaim from fork() which hits a lock held by the rest of the blocked stuff, so I wrote some hacky tools that do it without allocating (much) memory or forking: http://0x.ca/sim/ref/tools/ watch_utilization: full-screen iostat/vmstat type output pstack: full-screen "ps -eo pid,cmd,stat | grep D" with stack trace kswapd has a stack trace like this (and is almost always in D state), from cat /proc/`pidof kswapd0`/stack: [<ffffffff8126f205>] xfs_reclaim_inode+0x85/0x2b0 [<ffffffff8126f5b0>] xfs_reclaim_inodes_ag+0x180/0x2f0 [<ffffffff8126f74e>] xfs_reclaim_inodes_nr+0x2e/0x40 [<ffffffff8126ccf0>] xfs_fs_free_cached_objects+0x10/0x20 [<ffffffff81119a70>] prune_super+0x110/0x1b0 [<ffffffff810e4fa5>] shrink_slab+0x1e5/0x2a0 [<ffffffff810e5821>] kswapd+0x7c1/0xba0 [<ffffffff8107ada6>] kthread+0x96/0xb0 [<ffffffff816c0474>] kernel_thread_helper+0x4/0x10 [<ffffffffffffffff>] 0xffffffffffffffff All of the other processes that get stuck have this stack: [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b [<ffffffffffffffff>] 0xffffffffffffffff By the way, xfs_reclaim_inode+0x85 (133) disassembles as: 0xffffffff8128d4dd <+77>: nopl (%rax) 0xffffffff8128d4e0 <+80>: mov $0x4,%esi 0xffffffff8128d4e5 <+85>: mov %r12,%rdi 0xffffffff8128d4e8 <+88>: callq 0xffffffff812831c0 <xfs_ilock> 0xffffffff8128d4ed <+93>: mov %r15,%rdi 0xffffffff8128d4f0 <+96>: callq 0xffffffff8104c810 <try_wait_for_completion> 0xffffffff8128d4f5 <+101>: test %al,%al 0xffffffff8128d4f7 <+103>: jne 0xffffffff8128d515 <xfs_reclaim_inode+133> 0xffffffff8128d4f9 <+105>: test %r14d,%r14d 0xffffffff8128d4fc <+108>: nopl 0x0(%rax) 0xffffffff8128d500 <+112>: je 0xffffffff8128d6d8 <xfs_reclaim_inode+584> 0xffffffff8128d506 <+118>: mov %r15,%rdi 0xffffffff8128d509 <+121>: nopl 0x0(%rax) 0xffffffff8128d510 <+128>: callq 0xffffffff816f1bb0 <wait_for_completion> 0xffffffff8128d515 <+133>: mov -0x40(%rbp),%rdi 0xffffffff8128d519 <+137>: callq 0xffffffff81146e10 <is_bad_inode> 0xffffffff8128d51e <+142>: test %eax,%eax ...So the next function is wait_for_completion(), which is marked __sched and thus doesn't show up in the trace. When the clog happens, "iostat -x -k 1" shows no reads from the XFS devices, though writes keep happening. "vmstat 1" matches. I tried switching schedulers from CFQ to deadline -- no difference. Queue depth is empty on the devices and nothing is actually clogged up at the device -- it's not actually plugged at the controller or disk. I did a sysreq-w while this was happening. About 10 seconds later, everything unclogs and continues. Sysreq-W output below. I poked around at the various XFS tracepoints in /sys/kernel/debug/tracing/events/xfs, but I'm not sure which tracepoints to use and many of them scroll too fast to see anything. Any suggestions? Simon- SysRq : Show Blocked State task PC stack pid father kswapd0 D ffff88022fc52940 0 543 2 0x00000000 ffff8802261f1940 0000000000000046 ffff8802261f0010 ffff880226b5ac80 0000000000012940 ffff8802261f1fd8 ffff8802261f1fd8 0000000000012940 0000000000000001 ffff880226b5ac80 ffff8801e0831640 ffff880226b5b000 Call Trace: [<ffffffff816b482a>] ? schedule+0x3a/0x60 [<ffffffff8111167e>] ? __slab_free+0x1fe/0x320 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b3bae>] wait_for_common+0xee/0x180 [<ffffffff81050490>] ? try_to_wake_up+0x270/0x270 [<ffffffff810aa980>] ? call_rcu_sched+0x10/0x20 [<ffffffff816b3ce8>] wait_for_completion+0x18/0x20 [<ffffffff8126f205>] xfs_reclaim_inode+0x85/0x2b0 [<ffffffff8137663b>] ? radix_tree_gang_lookup_tag+0xab/0x120 [<ffffffff8126f5b0>] xfs_reclaim_inodes_ag+0x180/0x2f0 [<ffffffff8126f74e>] xfs_reclaim_inodes_nr+0x2e/0x40 [<ffffffff8126ccf0>] xfs_fs_free_cached_objects+0x10/0x20 [<ffffffff81119a70>] prune_super+0x110/0x1b0 [<ffffffff810e4fa5>] shrink_slab+0x1e5/0x2a0 [<ffffffff810e5821>] kswapd+0x7c1/0xba0 [<ffffffff8105049d>] ? default_wake_function+0xd/0x10 [<ffffffff8107b2c0>] ? wake_up_bit+0x40/0x40 [<ffffffff810e5060>] ? shrink_slab+0x2a0/0x2a0 [<ffffffff8107ada6>] kthread+0x96/0xb0 [<ffffffff816c0474>] kernel_thread_helper+0x4/0x10 [<ffffffff8107ad10>] ? kthread_worker_fn+0x190/0x190 [<ffffffff816c0470>] ? gs_change+0x13/0x13 pop3 D ffff88022fcd2940 0 4189 4182 0x00000000 ffff880226a8fa18 0000000000000082 ffff880226a8e010 ffff8802253a0000 0000000000012940 ffff880226a8ffd8 ffff880226a8ffd8 0000000000012940 0000000000000003 ffff8802253a0000 ffff880226949640 ffff8802253a0380 Call Trace: [<ffffffff81371163>] ? cpumask_next_and+0x23/0x40 [<ffffffff810498ca>] ? select_task_rq_fair+0x5aa/0xb30 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff812a3f58>] xfs_iunlink+0x48/0x150 [<ffffffff8105f1a2>] ? current_fs_time+0x22/0x30 [<ffffffff812b8ccd>] ? xfs_trans_ichgtime+0x1d/0xb0 [<ffffffff81271246>] xfs_droplink+0x56/0x70 [<ffffffff812731fd>] xfs_remove+0x2bd/0x3a0 [<ffffffff81269c73>] xfs_vn_unlink+0x43/0x90 [<ffffffff811250d1>] vfs_unlink+0x91/0x110 [<ffffffff81125313>] do_unlinkat+0x1c3/0x1e0 [<ffffffff8111b831>] ? sys_newstat+0x31/0x50 [<ffffffff81125341>] sys_unlink+0x11/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4190 4182 0x00000000 ffff880225475648 0000000000000082 ffff880225474010 ffff8802253a5900 0000000000012940 ffff880225475fd8 ffff880225475fd8 0000000000012940 0000000000000000 ffff8802253a5900 ffffffff819b3020 ffff8802253a5c80 Call Trace: [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff815e14f3>] ? dev_queue_xmit+0x153/0x5b0 [<ffffffff810d6e2a>] ? zone_watermark_ok+0x1a/0x20 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4191 4182 0x00000000 ffff880225e29648 0000000000000086 ffff880225e28010 ffff880226b09640 0000000000012940 ffff880225e29fd8 ffff880225e29fd8 0000000000012940 0000000000000000 ffff880226b09640 ffffffff819b3020 ffff880226b099c0 Call Trace: [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff81111fde>] ? kmem_cache_alloc+0xee/0x100 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4192 4182 0x00000000 ffff880224071648 0000000000000086 ffff880224070010 ffff880226b0c2c0 0000000000012940 ffff880224071fd8 ffff880224071fd8 0000000000012940 0000000000000000 ffff880226b0c2c0 ffffffff819b3020 ffff880226b0c640 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff816b4f6b>] ? __wait_on_bit+0x7b/0x90 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4193 4182 0x00000000 ffff8802255db648 0000000000000082 ffff8802255da010 ffff8802261c42c0 0000000000012940 ffff8802255dbfd8 ffff8802255dbfd8 0000000000012940 0000000000000000 ffff8802261c42c0 ffffffff819b3020 ffff8802261c4640 Call Trace: [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fcd2940 0 4194 4182 0x00000000 ffff880224121648 0000000000000086 ffff880224120010 ffff880226249640 0000000000012940 ffff880224121fd8 ffff880224121fd8 0000000000012940 0000000000000003 ffff880226249640 ffff880226949640 ffff8802262499c0 Call Trace: [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff81111fde>] ? kmem_cache_alloc+0xee/0x100 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81375790>] ? prop_norm_single+0x80/0xd0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4195 4182 0x00000000 ffff880224141648 0000000000000082 ffff880224140010 ffff8802260d0000 0000000000012940 ffff880224141fd8 ffff880224141fd8 0000000000012940 0000000000000000 ffff8802260d0000 ffffffff819b3020 ffff8802260d0380 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b pop3 D ffff88022fc12940 0 4196 4182 0x00000000 ffff880224143648 0000000000000086 ffff880224142010 ffff8802260d5900 0000000000012940 ffff880224143fd8 ffff880224143fd8 0000000000012940 0000000000000000 ffff8802260d5900 ffffffff819b3020 ffff8802260d5c80 Call Trace: [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff81111fde>] ? kmem_cache_alloc+0xee/0x100 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff8168a5b6>] ? vlan_dev_hard_start_xmit+0x76/0xe0 [<ffffffff815e10f8>] ? dev_hard_start_xmit+0x418/0x6c0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc92940 0 4205 4182 0x00000000 ffff88022433f648 0000000000000086 ffff88022433e010 ffff880225481640 0000000000012940 ffff88022433ffd8 ffff88022433ffd8 0000000000012940 0000000000000002 ffff880225481640 ffff880226929640 ffff8802254819c0 Call Trace: [<ffffffff8128ad9d>] ? xfs_bmbt_get_startoff+0xd/0x20 [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff81111fde>] ? kmem_cache_alloc+0xee/0x100 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff81011815>] ? __switch_to+0x1a5/0x340 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc12940 0 4206 4182 0x00000000 ffff88022435f648 0000000000000086 ffff88022435e010 ffff880225482c80 0000000000012940 ffff88022435ffd8 ffff88022435ffd8 0000000000012940 0000000000000000 ffff880225482c80 ffffffff819b3020 ffff880225483000 Call Trace: [<ffffffff8128ad56>] ? xfs_bmbt_get_all+0x16/0x20 [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fcd2940 0 4207 4182 0x00000000 ffff880224361648 0000000000000082 ffff880224360010 ffff8802261e9640 0000000000012940 ffff880224361fd8 ffff880224361fd8 0000000000012940 0000000000000003 ffff8802261e9640 ffff8802261c42c0 ffff8802261e99c0 Call Trace: [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff81011815>] ? __switch_to+0x1a5/0x340 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fcd2940 0 4208 4182 0x00000000 ffff880224383648 0000000000000086 ffff880224382010 ffff880226a7c2c0 0000000000012940 ffff880224383fd8 ffff880224383fd8 0000000000012940 0000000000000003 ffff880226a7c2c0 ffff8802260d0000 ffff880226a7c640 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff81111f65>] ? kmem_cache_alloc+0x75/0x100 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff812654ca>] ? xfs_iunlock+0x10a/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc52940 0 4209 4182 0x00000000 ffff8802243a3648 0000000000000086 ffff8802243a2010 ffff880224035900 0000000000012940 ffff8802243a3fd8 ffff8802243a3fd8 0000000000012940 0000000000000001 ffff880224035900 ffff880226911640 ffff880224035c80 Call Trace: [<ffffffff8128ad56>] ? xfs_bmbt_get_all+0x16/0x20 [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff812654ca>] ? xfs_iunlock+0x10a/0x130 [<ffffffff8125c250>] ? __xfs_get_blocks+0x460/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc52940 0 4210 4182 0x00000000 ffff8802243c5648 0000000000000082 ffff8802243c4010 ffff8802243c8000 0000000000012940 ffff8802243c5fd8 ffff8802243c5fd8 0000000000012940 0000000000000001 ffff8802243c8000 ffff880226911640 ffff8802243c8380 Call Trace: [<ffffffff8104beef>] ? set_next_entity+0xaf/0xd0 [<ffffffff812a1e7c>] ? xfs_iext_insert+0xbc/0xe0 [<ffffffff81285f3d>] ? xfs_bmap_add_extent+0xad/0x390 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc12940 0 4211 4182 0x00000000 ffff8802243c7648 0000000000000082 ffff8802243c6010 ffff8802243c9640 0000000000012940 ffff8802243c7fd8 ffff8802243c7fd8 0000000000012940 0000000000000000 ffff8802243c9640 ffffffff819b3020 ffff8802243c99c0 Call Trace: [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff810d56e1>] ? mempool_alloc_slab+0x11/0x20 [<ffffffff810d57f0>] ? mempool_alloc+0x60/0x140 [<ffffffff81111f65>] ? kmem_cache_alloc+0x75/0x100 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81265482>] ? xfs_iunlock+0xc2/0x130 [<ffffffff8125c201>] ? __xfs_get_blocks+0x411/0x4a0 [<ffffffff813758a6>] ? __prop_inc_single+0x46/0x60 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b lmtp D ffff88022fc12940 0 4212 4182 0x00000000 ffff8802243f3948 0000000000000082 ffff8802243f2010 ffff8802243cac80 0000000000012940 ffff8802243f3fd8 ffff8802243f3fd8 0000000000012940 0000000000000000 ffff8802243cac80 ffffffff819b3020 ffff8802243cb000 Call Trace: [<ffffffff810695f6>] ? lock_timer_base+0x36/0x70 [<ffffffff81069a78>] ? try_to_del_timer_sync+0xa8/0xd0 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4c62>] schedule_timeout+0x192/0x280 [<ffffffff81069c60>] ? del_timer+0xd0/0xd0 [<ffffffff816b4d69>] schedule_timeout_uninterruptible+0x19/0x20 [<ffffffff81265b56>] xfs_iget+0x176/0x670 [<ffffffff812a33aa>] xfs_ialloc+0x9a/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b imap D ffff88022fc52940 0 4215 4182 0x00000000 ffff88022043d648 0000000000000082 ffff88022043c010 ffff880225042c80 0000000000012940 ffff88022043dfd8 ffff88022043dfd8 0000000000012940 0000000000000001 ffff880225042c80 ffff880226911640 ffff880225043000 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81047ff0>] ? update_curr+0x100/0x1a0 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b imap D ffff88022fcd2940 0 4216 4182 0x00000000 ffff88022045f648 0000000000000082 ffff88022045e010 ffff880220460000 0000000000012940 ffff88022045ffd8 ffff88022045ffd8 0000000000012940 0000000000000003 ffff880220460000 ffff880226949640 ffff880220460380 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff812a1e7c>] ? xfs_iext_insert+0xbc/0xe0 [<ffffffff81285f3d>] ? xfs_bmap_add_extent+0xad/0x390 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff81371163>] ? cpumask_next_and+0x23/0x40 [<ffffffff812acb44>] ? xfs_perag_get+0x34/0xb0 [<ffffffff81265eef>] ? xfs_iget+0x50f/0x670 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff811228ba>] ? inode_permission+0x7a/0xd0 [<ffffffff81121bd7>] ? do_lookup+0xf7/0x3a0 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b doveadm D ffff88022fc12940 0 12727 11474 0x00000000 ffff880002ce5648 0000000000000082 ffff880002ce4010 ffff880218b30000 0000000000012940 ffff880002ce5fd8 ffff880002ce5fd8 0000000000012940 0000000000000000 ffff880218b30000 ffffffff819b3020 ffff880218b30380 Call Trace: [<ffffffff812a1181>] ? xfs_iext_bno_to_ext+0x91/0x180 [<ffffffff8114959b>] ? bvec_alloc_bs+0x6b/0x110 [<ffffffff816b482a>] schedule+0x3a/0x60 [<ffffffff816b4ced>] schedule_timeout+0x21d/0x280 [<ffffffff816b5cc0>] __down+0x70/0xb0 [<ffffffff812603ad>] ? _xfs_buf_find+0x16d/0x270 [<ffffffff81080587>] down+0x47/0x50 [<ffffffff8125e816>] xfs_buf_lock+0x66/0xd0 [<ffffffff812603ad>] _xfs_buf_find+0x16d/0x270 [<ffffffff81274c37>] ? kmem_zone_alloc+0x77/0xe0 [<ffffffff81260517>] xfs_buf_get+0x67/0x1a0 [<ffffffff8126067a>] xfs_buf_read+0x2a/0x120 [<ffffffff812b876f>] xfs_trans_read_buf+0x28f/0x3f0 [<ffffffff8129e161>] xfs_read_agi+0x71/0x100 [<ffffffff81111fde>] ? kmem_cache_alloc+0xee/0x100 [<ffffffff8129e219>] xfs_ialloc_read_agi+0x29/0xa0 [<ffffffff8129e59b>] xfs_ialloc_ag_select+0x1db/0x2a0 [<ffffffff8129fc95>] xfs_dialloc+0x415/0x8c0 [<ffffffff810b8215>] ? tracing_record_cmdline+0x35/0x110 [<ffffffff816b4385>] ? __schedule+0x585/0x950 [<ffffffff8112da50>] ? dput+0x40/0x320 [<ffffffff812a336c>] xfs_ialloc+0x5c/0x710 [<ffffffff812b381f>] ? xlog_grant_log_space+0x24f/0x530 [<ffffffff81274cd5>] ? kmem_zone_zalloc+0x35/0x50 [<ffffffff812712d5>] xfs_dir_ialloc+0x75/0x290 [<ffffffff812afa39>] ? xfs_trans_reserve+0x99/0x210 [<ffffffff8127390a>] xfs_create+0x46a/0x600 [<ffffffff8112b66f>] ? d_instantiate+0x5f/0x70 [<ffffffff81269a7a>] xfs_vn_mknod+0x7a/0x130 [<ffffffff81269b4b>] xfs_vn_create+0xb/0x10 [<ffffffff81124c99>] vfs_create+0x99/0xd0 [<ffffffff8112635b>] do_last+0x69b/0x810 [<ffffffff8112659f>] path_openat+0xcf/0x3f0 [<ffffffff8112401d>] ? user_path_at+0x8d/0xb0 [<ffffffff811269d3>] do_filp_open+0x43/0xa0 [<ffffffff81131d2a>] ? alloc_fd+0x4a/0x140 [<ffffffff81115202>] do_sys_open+0x152/0x1f0 [<ffffffff811152cb>] sys_open+0x1b/0x20 [<ffffffff816be392>] system_call_fastpath+0x16/0x1b -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists