| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 28 Aug 2013 15:45:03 -0700 From: kernel neophyte <neophyte.hacker001@...il.com> To: Stefan Priebe <s.priebe@...fihost.ag> Cc: Kent Overstreet <kmo@...erainc.com>, Jens Axboe <axboe@...nel.dk>, "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, "linux-bcache@...r.kernel.org" <linux-bcache@...r.kernel.org> Subject: Re: bcache: Fix a writeback performance regression On Wed, Aug 28, 2013 at 1:15 PM, Stefan Priebe <s.priebe@...fihost.ag> wrote: > sorry but that's completely wrong. > > please use branch bcache-for-3.10 > > http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-3.10 nope :-( doesn't help I still see deadlocks: [ 437.123184] bcache: bch_cached_dev_attach() Caching sda3 as bcache0 on set a9ad97be-0918-427a-8d2a-db6e0a2c0e7a [ 1442.355992] INFO: task kworker/0:0:4 blocked for more than 120 seconds. [ 1442.356003] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356009] kworker/0:0 D 0000000000000000 0 4 2 0x00000000 [ 1442.356023] Workqueue: bch_allocator bch_allocator_thread [ 1442.356027] ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8 ffffffff8107fd3d [ 1442.356033] ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8 ffff882fa6a85fd8 [ 1442.356037] ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80 ffff88211ce60db0 [ 1442.356042] Call Trace: [ 1442.356053] [<ffffffff8107fd3d>] ? dequeue_task_fair+0x2cd/0x530 [ 1442.356063] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356068] [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10 [ 1442.356073] [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0 [ 1442.356078] [<ffffffff816b829a>] mutex_lock+0x2a/0x50 [ 1442.356082] [<ffffffff81509432>] bch_allocator_thread+0x112/0xf30 [ 1442.356089] [<ffffffff8106a420>] ? add_wait_queue+0x60/0x60 [ 1442.356096] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1442.356101] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1442.356106] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356110] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1442.356115] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356122] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1442.356126] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356132] INFO: task kworker/1:1:47 blocked for more than 120 seconds. [ 1442.356136] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356140] kworker/1:1 D ffffffff8180c0c0 0 47 2 0x00000000 [ 1442.356150] Workqueue: bcache bch_journal [ 1442.356153] ffff882fa542d4f8 0000000000000046 ffff882fa542d518 0000000800000000 [ 1442.356157] ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8 ffff882fa542dfd8 [ 1442.356162] ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80 ffff88211ce60db0 [ 1442.356167] Call Trace: [ 1442.356172] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356177] [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10 [ 1442.356182] [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0 [ 1442.356187] [<ffffffff816b829a>] mutex_lock+0x2a/0x50 [ 1442.356191] [<ffffffff8150d6b5>] bch_mca_shrink+0x1c5/0x310 [ 1442.356198] [<ffffffff8117ba02>] ? prune_super+0x162/0x1b0 [ 1442.356206] [<ffffffff8112d054>] shrink_slab+0x154/0x300 [ 1442.356213] [<ffffffff81076118>] ? resched_task+0x68/0x70 [ 1442.356218] [<ffffffff81076a45>] ? check_preempt_curr+0x75/0xa0 [ 1442.356225] [<ffffffff811387c9>] ? fragmentation_index+0x19/0x70 [ 1442.356231] [<ffffffff8112f8af>] do_try_to_free_pages+0x20f/0x4b0 [ 1442.356236] [<ffffffff8112fd04>] try_to_free_pages+0xe4/0x1a0 [ 1442.356245] [<ffffffff8112536c>] __alloc_pages_nodemask+0x60c/0x990 [ 1442.356249] [<ffffffff81076118>] ? resched_task+0x68/0x70 [ 1442.356257] [<ffffffff8115e94a>] alloc_pages_current+0xba/0x170 [ 1442.356263] [<ffffffff81120a5e>] __get_free_pages+0xe/0x40 [ 1442.356268] [<ffffffff8150ac73>] mca_data_alloc+0x73/0x1d0 [ 1442.356272] [<ffffffff8150b2ba>] mca_bucket_alloc+0x14a/0x1f0 [ 1442.356276] [<ffffffff8150d3a0>] mca_alloc+0x320/0x470 [ 1442.356281] [<ffffffff8150e37b>] bch_btree_node_alloc+0x9b/0x200 [ 1442.356285] [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90 [ 1442.356290] [<ffffffff81510099>] btree_split+0x119/0x770 [ 1442.356298] [<ffffffff81025018>] ? native_smp_send_reschedule+0x48/0x60 [ 1442.356303] [<ffffffff81513cac>] ? __bch_bset_search+0x3fc/0x4a0 [ 1442.356307] [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90 [ 1442.356312] [<ffffffff81510adb>] bch_btree_insert_recurse+0x3eb/0x500 [ 1442.356316] [<ffffffff81510a78>] bch_btree_insert_recurse+0x388/0x500 [ 1442.356321] [<ffffffff81510d46>] bch_btree_insert+0x156/0x330 [ 1442.356327] [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0 [ 1442.356332] [<ffffffff81517ba5>] bch_journal+0x45/0x260 [ 1442.356337] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1442.356342] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1442.356347] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356350] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1442.356355] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356360] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1442.356364] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356373] INFO: task kworker/u64:7:349 blocked for more than 120 seconds. [ 1442.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356382] kworker/u64:7 D ffffffff8180bde0 0 349 2 0x00000000 [ 1442.356390] Workqueue: bcache_writeback write_dirty_finish [ 1442.356392] ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48 ffffffff81138789 [ 1442.356397] ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8 ffff882f88e8bfd8 [ 1442.356401] ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3 ffff882f88c8dc80 [ 1442.356406] Call Trace: [ 1442.356410] [<ffffffff81138789>] ? zone_statistics+0x99/0xc0 [ 1442.356415] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356421] [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5 [ 1442.356429] [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30 [ 1442.356434] [<ffffffff816b8b34>] ? down_read+0x24/0x2b [ 1442.356438] [<ffffffff81510d00>] bch_btree_insert+0x110/0x330 [ 1442.356444] [<ffffffff8151891a>] write_dirty_finish+0x27a/0x2c0 [ 1442.356450] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1442.356454] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1442.356459] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356463] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1442.356467] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356472] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1442.356476] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356485] INFO: task kworker/0:2:1967 blocked for more than 120 seconds. [ 1442.356489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356493] kworker/0:2 D 0000000000000000 0 1967 2 0x00000000 [ 1442.356500] Workqueue: bcache bch_journal [ 1442.356502] ffff882f890b7bf0 0000000000000046 0000000100000000 ffff882fbf273140 [ 1442.356507] ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8 ffff882f890b7fd8 [ 1442.356511] ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08 ffff882f88d8ae40 [ 1442.356516] Call Trace: [ 1442.356521] [<ffffffff81516b08>] ? journal_write_endio+0x38/0x40 [ 1442.356526] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356530] [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5 [ 1442.356536] [<ffffffff81053a90>] ? internal_add_timer+0x20/0x50 [ 1442.356541] [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30 [ 1442.356546] [<ffffffff816b8b34>] ? down_read+0x24/0x2b [ 1442.356550] [<ffffffff81510d00>] bch_btree_insert+0x110/0x330 [ 1442.356556] [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0 [ 1442.356561] [<ffffffff81517ba5>] bch_journal+0x45/0x260 [ 1442.356566] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1442.356570] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1442.356575] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356579] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1442.356583] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356588] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1442.356592] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356595] INFO: task kworker/u64:0:1980 blocked for more than 120 seconds. [ 1442.356600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356604] kworker/u64:0 D ffffffff8180bde0 0 1980 2 0x00000000 [ 1442.356610] Workqueue: bch_btree_io btree_write_work [ 1442.356612] ffff8822b4447cc8 0000000000000046 ffff8822b4447ca8 ffffffff81514c30 [ 1442.356616] ffff882f924d4560 ffff8822b4447fd8 ffff8822b4447fd8 ffff8822b4447fd8 [ 1442.356621] ffff882fa6ab0000 ffff882f924d4560 0000000000000000 ffff882f924d4560 [ 1442.356625] Call Trace: [ 1442.356630] [<ffffffff81514c30>] ? __btree_sort+0x160/0x250 [ 1442.356635] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356640] [<ffffffff816bae55>] rwsem_down_write_failed+0xf5/0x1a0 [ 1442.356645] [<ffffffff8132eb13>] call_rwsem_down_write_failed+0x13/0x20 [ 1442.356649] [<ffffffff816b8b01>] ? down_write+0x31/0x40 [ 1442.356654] [<ffffffff8150bfbf>] btree_write_work+0x2f/0x60 [ 1442.356658] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1442.356663] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1442.356668] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1442.356671] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1442.356676] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356680] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1442.356684] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1442.356688] INFO: task iozone:1993 blocked for more than 120 seconds. [ 1442.356692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1442.356696] iozone D 0000000000000000 0 1993 1470 0x00000000 [ 1442.356700] ffff881a3c981978 0000000000000082 ffff881a3c981938 ffffffff812fdc7a [ 1442.356705] ffff882f88d89720 ffff881a3c981fd8 ffff881a3c981fd8 ffff881a3c981fd8 [ 1442.356710] ffff882fa5f85c80 ffff882f88d89720 ffff881a3c981978 ffff882fbf2339f8 [ 1442.356714] Call Trace: [ 1442.356721] [<ffffffff812fdc7a>] ? generic_make_request+0xca/0x100 [ 1442.356726] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1442.356731] [<ffffffff816ba21f>] io_schedule+0x8f/0xd0 [ 1442.356740] [<ffffffff811b375c>] do_blockdev_direct_IO+0x1a7c/0x1fb0 [ 1442.356751] [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356757] [<ffffffff811b3ce5>] __blockdev_direct_IO+0x55/0x60 [ 1442.356763] [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356769] [<ffffffffa02e4349>] ext2_direct_IO+0x79/0xe0 [ext2] [ 1442.356775] [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2] [ 1442.356781] [<ffffffff8104ad96>] ? current_fs_time+0x16/0x60 [ 1442.356787] [<ffffffff8111d7b6>] generic_file_direct_write+0xc6/0x180 [ 1442.356793] [<ffffffff8111db4d>] __generic_file_aio_write+0x2dd/0x3b0 [ 1442.356798] [<ffffffff8111dc89>] generic_file_aio_write+0x69/0xd0 [ 1442.356803] [<ffffffff8117767a>] do_sync_write+0x7a/0xb0 [ 1442.356808] [<ffffffff8117842e>] vfs_write+0xce/0x1e0 [ 1442.356812] [<ffffffff81178912>] SyS_write+0x52/0xa0 [ 1442.356818] [<ffffffff816c38c2>] system_call_fastpath+0x16/0x1b [ 1562.355919] INFO: task kworker/0:0:4 blocked for more than 120 seconds. [ 1562.355927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.355935] kworker/0:0 D 0000000000000000 0 4 2 0x00000000 [ 1562.355945] Workqueue: bch_allocator bch_allocator_thread [ 1562.355949] ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8 ffffffff8107fd3d [ 1562.355954] ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8 ffff882fa6a85fd8 [ 1562.355959] ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80 ffff88211ce60db0 [ 1562.355963] Call Trace: [ 1562.355971] [<ffffffff8107fd3d>] ? dequeue_task_fair+0x2cd/0x530 [ 1562.355977] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1562.355982] [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10 [ 1562.355987] [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0 [ 1562.355992] [<ffffffff816b829a>] mutex_lock+0x2a/0x50 [ 1562.355996] [<ffffffff81509432>] bch_allocator_thread+0x112/0xf30 [ 1562.356001] [<ffffffff8106a420>] ? add_wait_queue+0x60/0x60 [ 1562.356006] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1562.356011] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1562.356016] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356020] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1562.356025] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356030] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1562.356035] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356040] INFO: task kworker/1:1:47 blocked for more than 120 seconds. [ 1562.356045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356051] kworker/1:1 D ffffffff8180c0c0 0 47 2 0x00000000 [ 1562.356059] Workqueue: bcache bch_journal [ 1562.356061] ffff882fa542d4f8 0000000000000046 ffff882fa542d518 0000000800000000 [ 1562.356066] ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8 ffff882fa542dfd8 [ 1562.356070] ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80 ffff88211ce60db0 [ 1562.356075] Call Trace: [ 1562.356080] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1562.356085] [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10 [ 1562.356089] [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0 [ 1562.356094] [<ffffffff816b829a>] mutex_lock+0x2a/0x50 [ 1562.356098] [<ffffffff8150d6b5>] bch_mca_shrink+0x1c5/0x310 [ 1562.356104] [<ffffffff8117ba02>] ? prune_super+0x162/0x1b0 [ 1562.356109] [<ffffffff8112d054>] shrink_slab+0x154/0x300 [ 1562.356115] [<ffffffff81076118>] ? resched_task+0x68/0x70 [ 1562.356120] [<ffffffff81076a45>] ? check_preempt_curr+0x75/0xa0 [ 1562.356125] [<ffffffff811387c9>] ? fragmentation_index+0x19/0x70 [ 1562.356130] [<ffffffff8112f8af>] do_try_to_free_pages+0x20f/0x4b0 [ 1562.356135] [<ffffffff8112fd04>] try_to_free_pages+0xe4/0x1a0 [ 1562.356142] [<ffffffff8112536c>] __alloc_pages_nodemask+0x60c/0x990 [ 1562.356147] [<ffffffff81076118>] ? resched_task+0x68/0x70 [ 1562.356153] [<ffffffff8115e94a>] alloc_pages_current+0xba/0x170 [ 1562.356159] [<ffffffff81120a5e>] __get_free_pages+0xe/0x40 [ 1562.356163] [<ffffffff8150ac73>] mca_data_alloc+0x73/0x1d0 [ 1562.356168] [<ffffffff8150b2ba>] mca_bucket_alloc+0x14a/0x1f0 [ 1562.356172] [<ffffffff8150d3a0>] mca_alloc+0x320/0x470 [ 1562.356177] [<ffffffff8150e37b>] bch_btree_node_alloc+0x9b/0x200 [ 1562.356181] [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90 [ 1562.356186] [<ffffffff81510099>] btree_split+0x119/0x770 [ 1562.356191] [<ffffffff81025018>] ? native_smp_send_reschedule+0x48/0x60 [ 1562.356196] [<ffffffff81513cac>] ? __bch_bset_search+0x3fc/0x4a0 [ 1562.356200] [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90 [ 1562.356205] [<ffffffff81510adb>] bch_btree_insert_recurse+0x3eb/0x500 [ 1562.356209] [<ffffffff81510a78>] bch_btree_insert_recurse+0x388/0x500 [ 1562.356214] [<ffffffff81510d46>] bch_btree_insert+0x156/0x330 [ 1562.356220] [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0 [ 1562.356225] [<ffffffff81517ba5>] bch_journal+0x45/0x260 [ 1562.356229] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1562.356234] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1562.356239] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356243] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1562.356247] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356252] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1562.356256] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356262] INFO: task kworker/u64:7:349 blocked for more than 120 seconds. [ 1562.356267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356273] kworker/u64:7 D ffffffff8180bde0 0 349 2 0x00000000 [ 1562.356280] Workqueue: bcache_writeback write_dirty_finish [ 1562.356282] ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48 ffffffff81138789 [ 1562.356286] ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8 ffff882f88e8bfd8 [ 1562.356291] ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3 ffff882f88c8dc80 [ 1562.356295] Call Trace: [ 1562.356300] [<ffffffff81138789>] ? zone_statistics+0x99/0xc0 [ 1562.356305] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1562.356310] [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5 [ 1562.356316] [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30 [ 1562.356320] [<ffffffff816b8b34>] ? down_read+0x24/0x2b [ 1562.356325] [<ffffffff81510d00>] bch_btree_insert+0x110/0x330 [ 1562.356330] [<ffffffff8151891a>] write_dirty_finish+0x27a/0x2c0 [ 1562.356336] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1562.356341] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1562.356346] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356350] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1562.356354] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356359] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1562.356363] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356373] INFO: task kworker/0:2:1967 blocked for more than 120 seconds. [ 1562.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1562.356383] kworker/0:2 D 0000000000000000 0 1967 2 0x00000000 [ 1562.356389] Workqueue: bcache bch_journal [ 1562.356391] ffff882f890b7bf0 0000000000000046 0000000100000000 ffff882fbf273140 [ 1562.356396] ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8 ffff882f890b7fd8 [ 1562.356400] ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08 ffff882f88d8ae40 [ 1562.356405] Call Trace: [ 1562.356410] [<ffffffff81516b08>] ? journal_write_endio+0x38/0x40 [ 1562.356414] [<ffffffff816ba149>] schedule+0x29/0x70 [ 1562.356419] [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5 [ 1562.356424] [<ffffffff81053a90>] ? internal_add_timer+0x20/0x50 [ 1562.356429] [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30 [ 1562.356433] [<ffffffff816b8b34>] ? down_read+0x24/0x2b [ 1562.356438] [<ffffffff81510d00>] bch_btree_insert+0x110/0x330 [ 1562.356444] [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0 [ 1562.356449] [<ffffffff81517ba5>] bch_journal+0x45/0x260 [ 1562.356453] [<ffffffff81062454>] process_one_work+0x174/0x490 [ 1562.356458] [<ffffffff8106351b>] worker_thread+0x11b/0x370 [ 1562.356463] [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0 [ 1562.356466] [<ffffffff81069c60>] kthread+0xc0/0xd0 [ 1562.356471] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 [ 1562.356476] [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0 [ 1562.356480] [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0 -Neo > Stefan > > Am 28.08.2013 22:12, schrieb kernel neophyte: > >> On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe <s.priebe@...fihost.ag> >> wrote: >>> >>> Am 28.08.2013 20:47, schrieb kernel neophyte: >>> >>>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG >>>> <s.priebe@...fihost.ag> wrote: >>>>> >>>>> >>>>> I don't had one for a few days. Which kernel so you use? >>>> >>>> >>>> >>>> 3.10 kernel with all of kent's stable patches and perf patches.. >>> >>> >>> >>> Which exact 3.10 version? Which patches on top? >> >> >> 3.10.0-rc7 >> with following patches: >> bcache: Fix a dumb CPU spinning bug in writeback >> bcache: Fix a flush/fua performance bug >> bcache: Fix a writeback performance regression >> bcache: FUA fixes >> >> I have also tried bcache perf branch i.e. >> (http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf) >> >>> >>> >>>> and all defaults except for: >>>> >>>> echo writeback > /sys/block/bcache0/bcache/cache_mode >>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff >>>> echo 0 > >>>> >>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us >>>> echo 0 > >>>> >>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us >>>> >>>> -Neo >>>> >>>>> >>>>> Stefan >>>>> >>>>> This mail was sent with my iPhone. >>>>> >>>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte >>>>> <neophyte.hacker001@...il.com>: >>>>> >>>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@...erainc.com> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi Kent, >>>>>>>> >>>>>>>> I am still seeing deadlock: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Seeing blk_throtl_bio (what asshole misspells words in their function >>>>>>> names, so when you go to grep for them you grep for the wrong thing?) >>>>>>> in >>>>>>> the backtrace - is that what it takes to hit it? >>>>>> >>>>>> >>>>>> >>>>>> I don't think so, >>>>>> I had another deadlock: >>>>>> >>>>>> [ 1179.193990] bio: create slab <bio-2> at 2 >>>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data >>>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1 >>>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3 >>>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0 >>>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c >>>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.497967] kworker/0:0 D 0000000000000000 0 4 2 >>>>>> 0x00000000 >>>>>> [ 2162.497982] Workqueue: events write_dirty_finish >>>>>> [ 2162.497986] ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.497992] ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8 >>>>>> ffff882fa6ac9fd8 >>>>>> [ 2162.497996] ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1 >>>>>> ffff882fa6a84cb0 >>>>>> [ 2162.498001] Call Trace: >>>>>> [ 2162.498010] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498015] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498019] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498028] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498034] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498043] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498048] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.498052] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498057] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498066] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498072] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498076] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498082] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498087] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498093] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498100] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.498105] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.498110] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498115] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498120] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498127] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498131] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than >>>>>> 120 seconds. >>>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498154] bcache_allocato D 0000000000000001 0 1766 2 >>>>>> 0x00000000 >>>>>> [ 2162.498158] ffff8822630f3d88 0000000000000046 ffff8822630f3da8 >>>>>> ffffffff810808ad >>>>>> [ 2162.498163] ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8 >>>>>> ffff8822630f3fd8 >>>>>> [ 2162.498167] ffff882f81a91990 ffff882f891de640 ffff882f88b33320 >>>>>> ffff8801fad20d98 >>>>>> [ 2162.498172] Call Trace: >>>>>> [ 2162.498177] [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530 >>>>>> [ 2162.498182] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498187] [<ffffffff816be57e>] >>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>> [ 2162.498192] [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0 >>>>>> [ 2162.498197] [<ffffffff816bc3da>] mutex_lock+0x2a/0x50 >>>>>> [ 2162.498202] [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20 >>>>>> [ 2162.498207] [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0 >>>>>> [ 2162.498211] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498215] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498220] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498224] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than >>>>>> 120 seconds. >>>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498237] bcache_writebac D 0000000000000000 0 1771 2 >>>>>> 0x00000000 >>>>>> [ 2162.498241] ffff88197d2adc58 0000000000000046 ffff88197d2add48 >>>>>> ffffffff811269e2 >>>>>> [ 2162.498245] ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8 >>>>>> ffff88197d2adfd8 >>>>>> [ 2162.498250] ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68 >>>>>> ffff882f89550ba0 >>>>>> [ 2162.498254] Call Trace: >>>>>> [ 2162.498264] [<ffffffff811269e2>] ? >>>>>> __alloc_pages_nodemask+0x152/0x9b0 >>>>>> [ 2162.498269] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498274] [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250 >>>>>> [ 2162.498280] [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0 >>>>>> [ 2162.498285] [<ffffffff816bcd09>] __down_common+0x8e/0xe3 >>>>>> [ 2162.498293] [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170 >>>>>> [ 2162.498298] [<ffffffff816bcdd1>] __down+0x1d/0x1f >>>>>> [ 2162.498304] [<ffffffff8106f351>] down+0x41/0x50 >>>>>> [ 2162.498310] [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0 >>>>>> [ 2162.498315] [<ffffffff8108458b>] ? idle_balance+0xeb/0x150 >>>>>> [ 2162.498321] [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0 >>>>>> [ 2162.498325] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498329] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498334] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498338] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498351] kworker/0:2 D ffffffff81813a60 0 1789 2 >>>>>> 0x00000000 >>>>>> [ 2162.498357] Workqueue: events write_dirty_finish >>>>>> [ 2162.498360] ffff88277062fb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498364] ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8 >>>>>> ffff88277062ffd8 >>>>>> [ 2162.498368] ffffffff81c10440 ffff882fa6786640 ffffffff815117a1 >>>>>> ffff882fa6786640 >>>>>> [ 2162.498373] Call Trace: >>>>>> [ 2162.498378] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498382] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498387] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498392] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498397] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498402] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498407] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.498411] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498416] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498420] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498425] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498430] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498435] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498439] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498444] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498449] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.498454] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.498459] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498463] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498467] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498472] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498476] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498488] iozone D 0000000000000001 0 2052 1944 >>>>>> 0x00000000 >>>>>> [ 2162.498492] ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938 >>>>>> ffffffff81301d7a >>>>>> [ 2162.498497] ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8 >>>>>> ffff882f7e2d3fd8 >>>>>> [ 2162.498501] ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978 >>>>>> ffff882fbf2339f8 >>>>>> [ 2162.498505] Call Trace: >>>>>> [ 2162.498511] [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100 >>>>>> [ 2162.498517] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498522] [<ffffffff816be36f>] io_schedule+0x8f/0xd0 >>>>>> [ 2162.498530] [<ffffffff811b797c>] >>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0 >>>>>> [ 2162.498539] [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498546] [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60 >>>>>> [ 2162.498552] [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498556] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498562] [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2] >>>>>> [ 2162.498567] [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>> [ext2] >>>>>> [ 2162.498572] [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0 >>>>>> [ 2162.498579] [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60 >>>>>> [ 2162.498585] [<ffffffff8111f126>] >>>>>> generic_file_direct_write+0xc6/0x180 >>>>>> [ 2162.498590] [<ffffffff8111f4bd>] >>>>>> __generic_file_aio_write+0x2dd/0x3b0 >>>>>> [ 2162.498595] [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20 >>>>>> [ 2162.498600] [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0 >>>>>> [ 2162.498607] [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0 >>>>>> [ 2162.498612] [<ffffffff8117c63e>] vfs_write+0xce/0x1e0 >>>>>> [ 2162.498617] [<ffffffff8117bba0>] ? >>>>>> generic_file_llseek_size+0x110/0x110 >>>>>> [ 2162.498621] [<ffffffff8117cb22>] SyS_write+0x52/0xa0 >>>>>> [ 2162.498627] [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b >>>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498639] kworker/0:1 D ffffffff81813a60 0 2293 2 >>>>>> 0x00000000 >>>>>> [ 2162.498645] Workqueue: events write_dirty_finish >>>>>> [ 2162.498647] ffff882f8a05bb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498652] ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8 >>>>>> ffff882f8a05bfd8 >>>>>> [ 2162.498656] ffffffff81c10440 ffff882f8a163320 ffffffff815117a1 >>>>>> ffff882f8a163320 >>>>>> [ 2162.498661] Call Trace: >>>>>> [ 2162.498665] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498669] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498674] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498679] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498684] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498689] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498693] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.498698] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498702] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498707] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498712] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498716] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498721] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498726] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498731] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498735] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.498740] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.498745] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498749] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498753] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498758] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498762] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498777] kworker/0:3 D ffffffff81813a60 0 2534 2 >>>>>> 0x00000000 >>>>>> [ 2162.498783] Workqueue: events write_dirty_finish >>>>>> [ 2162.498785] ffff88237d0bdb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498790] ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8 >>>>>> ffff88237d0bdfd8 >>>>>> [ 2162.498794] ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1 >>>>>> ffff882fa66fe640 >>>>>> [ 2162.498799] Call Trace: >>>>>> [ 2162.498803] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498807] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498812] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498817] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498822] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498826] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498831] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.498836] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498840] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498845] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498849] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498854] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498859] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.498864] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.498868] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.498873] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.498878] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.498882] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.498886] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.498890] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498896] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.498900] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.498914] kworker/0:4 D 0000000000000000 0 2537 2 >>>>>> 0x00000000 >>>>>> [ 2162.498920] Workqueue: events write_dirty_finish >>>>>> [ 2162.498922] ffff882f76d8fb30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.498926] ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8 >>>>>> ffff882f76d8ffd8 >>>>>> [ 2162.498931] ffff882f78beb320 ffff882f88b30000 ffffffff815117a1 >>>>>> ffff882f88b30000 >>>>>> [ 2162.498935] Call Trace: >>>>>> [ 2162.498940] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.498944] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498948] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.498953] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.498958] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.498963] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.498968] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.498972] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.498977] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.498981] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.498986] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.498990] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.498995] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.499000] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.499004] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.499009] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.499014] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.499019] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.499023] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.499027] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499032] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.499036] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.499050] kworker/1:1 D ffffffff81813d40 0 2544 2 >>>>>> 0x00000000 >>>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys >>>>>> [ 2162.499059] ffff882fa677b3f8 0000000000000046 ffff882fa677b418 >>>>>> ffff882f8a709000 >>>>>> [ 2162.499063] ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8 >>>>>> ffff882fa677bfd8 >>>>>> [ 2162.499068] ffff882fa6471990 ffff882f88b33320 ffff882f88b33320 >>>>>> ffff8801fad20d98 >>>>>> [ 2162.499072] Call Trace: >>>>>> [ 2162.499077] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.499082] [<ffffffff816be57e>] >>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>> [ 2162.499087] [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0 >>>>>> [ 2162.499092] [<ffffffff816bc3da>] mutex_lock+0x2a/0x50 >>>>>> [ 2162.499096] [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0 >>>>>> [ 2162.499102] [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0 >>>>>> [ 2162.499109] [<ffffffff8112ebb4>] shrink_slab+0x154/0x300 >>>>>> [ 2162.499113] [<ffffffff81076828>] ? resched_task+0x68/0x70 >>>>>> [ 2162.499118] [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0 >>>>>> [ 2162.499126] [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70 >>>>>> [ 2162.499131] [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0 >>>>>> [ 2162.499137] [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0 >>>>>> [ 2162.499143] [<ffffffff81126e9c>] >>>>>> __alloc_pages_nodemask+0x60c/0x9b0 >>>>>> [ 2162.499149] [<ffffffff8116062a>] alloc_pages_current+0xba/0x170 >>>>>> [ 2162.499155] [<ffffffff8112240e>] __get_free_pages+0xe/0x40 >>>>>> [ 2162.499159] [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0 >>>>>> [ 2162.499163] [<ffffffff81510f37>] mca_alloc+0x277/0x470 >>>>>> [ 2162.499168] [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0 >>>>>> [ 2162.499173] [<ffffffff81513020>] btree_split+0x110/0x5c0 >>>>>> [ 2162.499179] [<ffffffff81457d43>] ? >>>>>> scsi_pool_alloc_command+0x33/0x80 >>>>>> [ 2162.499184] [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50 >>>>>> [ 2162.499188] [<ffffffff8150fed6>] ? >>>>>> bch_btree_insert_keys+0x56/0x250 >>>>>> [ 2162.499193] [<ffffffff81082def>] ? update_group_power+0x13f/0x220 >>>>>> [ 2162.499198] [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0 >>>>>> [ 2162.499202] [<ffffffff815137e8>] btree_insert_fn+0x28/0x50 >>>>>> [ 2162.499207] [<ffffffff81511b8c>] >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.499211] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499216] [<ffffffff816bcc26>] ? down_write+0x16/0x40 >>>>>> [ 2162.499221] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.499225] [<ffffffff81511c30>] >>>>>> bch_btree_map_nodes_recurse+0x110/0x170 >>>>>> [ 2162.499230] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499235] [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90 >>>>>> [ 2162.499239] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>> [ 2162.499244] [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0 >>>>>> [ 2162.499248] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499254] [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0 >>>>>> [ 2162.499258] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.499264] [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160 >>>>>> [ 2162.499268] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.499273] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.499278] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.499282] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.499286] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499292] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.499296] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120 >>>>>> seconds. >>>>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>> disables this message. >>>>>> [ 2162.499310] kworker/0:5 D 0000000000000000 0 2552 2 >>>>>> 0x00000000 >>>>>> [ 2162.499411] Workqueue: events write_dirty_finish >>>>>> [ 2162.499413] ffff88226314db30 0000000000000046 ffffffff81511b8c >>>>>> ffffffff815137c0 >>>>>> [ 2162.499418] ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8 >>>>>> ffff88226314dfd8 >>>>>> [ 2162.499422] ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1 >>>>>> ffff882fa6781990 >>>>>> [ 2162.499427] Call Trace: >>>>>> [ 2162.499431] [<ffffffff81511b8c>] ? >>>>>> bch_btree_map_nodes_recurse+0x6c/0x170 >>>>>> [ 2162.499435] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499440] [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280 >>>>>> [ 2162.499445] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>> [ 2162.499450] [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5 >>>>>> [ 2162.499455] [<ffffffff81332c64>] >>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>> [ 2162.499460] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>> [ 2162.499464] [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0 >>>>>> [ 2162.499468] [<ffffffff815137c0>] ? >>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>> [ 2162.499473] [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10 >>>>>> [ 2162.499478] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>> [ 2162.499483] [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270 >>>>>> [ 2162.499488] [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150 >>>>>> [ 2162.499492] [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0 >>>>>> [ 2162.499497] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>> [ 2162.499502] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>> [ 2162.499507] [<ffffffff81063570>] ? >>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>> [ 2162.499511] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>> [ 2162.499515] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> [ 2162.499520] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>> [ 2162.499524] [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0 >>>>>> >>>>>> >>>>>> >>>>>>>> [ 930.169234] bcache: bch_cached_dev_attach() Caching sda3 as >>>>>>>> bcache0 >>>>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1 >>>>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than >>>>>>>> 120 seconds. >>>>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60 0 2058 2 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.956212] ffff881c39c1fcd8 0000000000000046 0000000000000001 >>>>>>>> 0000000000000001 >>>>>>>> [ 2522.956218] ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8 >>>>>>>> ffff881c39c1ffd8 >>>>>>>> [ 2522.956222] ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320 >>>>>>>> ffff882f8b56b320 >>>>>>>> [ 2522.956227] Call Trace: >>>>>>>> [ 2522.956241] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.956247] [<ffffffff816befb5>] >>>>>>>> rwsem_down_write_failed+0xf5/0x1a0 >>>>>>>> [ 2522.956255] [<ffffffff81332c93>] >>>>>>>> call_rwsem_down_write_failed+0x13/0x20 >>>>>>>> [ 2522.956261] [<ffffffff816bcc41>] ? down_write+0x31/0x40 >>>>>>>> [ 2522.956269] [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0 >>>>>>>> [ 2522.956278] [<ffffffff8108458b>] ? idle_balance+0xeb/0x150 >>>>>>>> [ 2522.956284] [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0 >>>>>>>> [ 2522.956290] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>>>> [ 2522.956294] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956302] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>>>> [ 2522.956306] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than >>>>>>>> 120 >>>>>>>> seconds. >>>>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.956322] kworker/u64:1 D ffffffff81813a60 0 2433 2 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn >>>>>>>> (flush-252:0) >>>>>>>> [ 2522.956336] ffff882f899d1560 0000000000000046 ffffffff81120635 >>>>>>>> 0000000000000100 >>>>>>>> [ 2522.956340] ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8 >>>>>>>> ffff882f899d1fd8 >>>>>>>> [ 2522.956345] ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55 >>>>>>>> ffff88219b488000 >>>>>>>> [ 2522.956349] Call Trace: >>>>>>>> [ 2522.956359] [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20 >>>>>>>> [ 2522.956367] [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540 >>>>>>>> [ 2522.956372] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.956378] [<ffffffff816bf0fd>] >>>>>>>> rwsem_down_read_failed+0x9d/0xe5 >>>>>>>> [ 2522.956383] [<ffffffff81332c64>] >>>>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>>>> [ 2522.956388] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>>>> [ 2522.956394] [<ffffffff8151fe32>] >>>>>>>> cached_dev_make_request+0x752/0xf00 >>>>>>>> [ 2522.956400] [<ffffffff81301af8>] ? >>>>>>>> generic_make_request_checks+0x1e8/0x3a0 >>>>>>>> [ 2522.956405] [<ffffffff81301d7a>] generic_make_request+0xca/0x100 >>>>>>>> [ 2522.956409] [<ffffffff81301e29>] submit_bio+0x79/0x160 >>>>>>>> [ 2522.956417] [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0 >>>>>>>> [ 2522.956421] [<ffffffff811abd3f>] _submit_bh+0x13f/0x200 >>>>>>>> [ 2522.956425] [<ffffffff811abe10>] submit_bh+0x10/0x20 >>>>>>>> [ 2522.956430] [<ffffffff811af6b8>] >>>>>>>> __block_write_full_page+0x1d8/0x360 >>>>>>>> [ 2522.956439] [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530 >>>>>>>> [ 2522.956444] [<ffffffff811ad3c0>] ? >>>>>>>> end_buffer_async_read+0x130/0x130 >>>>>>>> [ 2522.956449] [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10 >>>>>>>> [ 2522.956454] [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10 >>>>>>>> [ 2522.956459] [<ffffffff811af90a>] >>>>>>>> block_write_full_page_endio+0xca/0x100 >>>>>>>> [ 2522.956464] [<ffffffff811af955>] block_write_full_page+0x15/0x20 >>>>>>>> [ 2522.956469] [<ffffffff811b3608>] blkdev_writepage+0x18/0x20 >>>>>>>> [ 2522.956473] [<ffffffff81127d37>] __writepage+0x17/0x40 >>>>>>>> [ 2522.956477] [<ffffffff8112820e>] write_cache_pages+0x20e/0x460 >>>>>>>> [ 2522.956481] [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60 >>>>>>>> [ 2522.956486] [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0 >>>>>>>> [ 2522.956490] [<ffffffff811284aa>] generic_writepages+0x4a/0x70 >>>>>>>> [ 2522.956495] [<ffffffff81129cd0>] do_writepages+0x20/0x40 >>>>>>>> [ 2522.956501] [<ffffffff811a36f5>] >>>>>>>> __writeback_single_inode+0x45/0x280 >>>>>>>> [ 2522.956507] [<ffffffff811269e2>] ? >>>>>>>> __alloc_pages_nodemask+0x152/0x9b0 >>>>>>>> [ 2522.956513] [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0 >>>>>>>> [ 2522.956518] [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0 >>>>>>>> [ 2522.956523] [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320 >>>>>>>> [ 2522.956529] [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80 >>>>>>>> [ 2522.956534] [<ffffffff811a500f>] >>>>>>>> wb_check_old_data_flush+0x9f/0xb0 >>>>>>>> [ 2522.956539] [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0 >>>>>>>> [ 2522.956547] [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80 >>>>>>>> [ 2522.956552] [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200 >>>>>>>> [ 2522.956557] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>>>> [ 2522.956562] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>>>> [ 2522.956567] [<ffffffff81063570>] ? >>>>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>>>> [ 2522.956571] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>>>> [ 2522.956575] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956581] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>>>> [ 2522.956585] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 >>>>>>>> seconds. >>>>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.956597] iozone D 0000000000000001 0 2435 1545 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.956602] ffff882a75671978 0000000000000082 ffff882a75671938 >>>>>>>> ffffffff81301d7a >>>>>>>> [ 2522.956606] ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8 >>>>>>>> ffff882a75671fd8 >>>>>>>> [ 2522.956611] ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978 >>>>>>>> ffff882fbf2539f8 >>>>>>>> [ 2522.956615] Call Trace: >>>>>>>> [ 2522.956619] [<ffffffff81301d7a>] ? >>>>>>>> generic_make_request+0xca/0x100 >>>>>>>> [ 2522.956625] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.956630] [<ffffffff816be36f>] io_schedule+0x8f/0xd0 >>>>>>>> [ 2522.956636] [<ffffffff811b797c>] >>>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0 >>>>>>>> [ 2522.956645] [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>>>> [ext2] >>>>>>>> [ 2522.956652] [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60 >>>>>>>> [ 2522.956658] [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>>>> [ext2] >>>>>>>> [ 2522.956662] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>>>> [ 2522.956669] [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2] >>>>>>>> [ 2522.956674] [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 >>>>>>>> [ext2] >>>>>>>> [ 2522.956680] [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60 >>>>>>>> [ 2522.956685] [<ffffffff8111f126>] >>>>>>>> generic_file_direct_write+0xc6/0x180 >>>>>>>> [ 2522.956690] [<ffffffff8111f4bd>] >>>>>>>> __generic_file_aio_write+0x2dd/0x3b0 >>>>>>>> [ 2522.956696] [<ffffffff8111f5f9>] >>>>>>>> generic_file_aio_write+0x69/0xd0 >>>>>>>> [ 2522.956702] [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0 >>>>>>>> [ 2522.956706] [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0 >>>>>>>> [ 2522.956712] [<ffffffff8117c63e>] vfs_write+0xce/0x1e0 >>>>>>>> [ 2522.956716] [<ffffffff8117cb22>] SyS_write+0x52/0xa0 >>>>>>>> [ 2522.956722] [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b >>>>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 >>>>>>>> seconds. >>>>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.956735] kworker/2:1 D ffffffff81813d40 0 2470 2 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys >>>>>>>> [ 2522.956745] ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648 >>>>>>>> 0000004000000000 >>>>>>>> [ 2522.956749] ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8 >>>>>>>> ffff882a5b0c9fd8 >>>>>>>> [ 2522.956754] ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990 >>>>>>>> ffff8828dba00d98 >>>>>>>> [ 2522.956758] Call Trace: >>>>>>>> [ 2522.956763] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.956769] [<ffffffff816be57e>] >>>>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>>>> [ 2522.956774] [<ffffffff816bc862>] >>>>>>>> __mutex_lock_slowpath+0x112/0x1b0 >>>>>>>> [ 2522.956779] [<ffffffff816bc3da>] mutex_lock+0x2a/0x50 >>>>>>>> [ 2522.956783] [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0 >>>>>>>> [ 2522.956791] [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0 >>>>>>>> [ 2522.956799] [<ffffffff8112ebb4>] shrink_slab+0x154/0x300 >>>>>>>> [ 2522.956805] [<ffffffff81076828>] ? resched_task+0x68/0x70 >>>>>>>> [ 2522.956810] [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0 >>>>>>>> [ 2522.956816] [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70 >>>>>>>> [ 2522.956822] [<ffffffff8113140f>] >>>>>>>> do_try_to_free_pages+0x20f/0x4b0 >>>>>>>> [ 2522.956827] [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0 >>>>>>>> [ 2522.956833] [<ffffffff81126e9c>] >>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0 >>>>>>>> [ 2522.956843] [<ffffffff8116062a>] alloc_pages_current+0xba/0x170 >>>>>>>> [ 2522.956848] [<ffffffff8112240e>] __get_free_pages+0xe/0x40 >>>>>>>> [ 2522.956852] [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0 >>>>>>>> [ 2522.956857] [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0 >>>>>>>> [ 2522.956861] [<ffffffff81511020>] mca_alloc+0x360/0x470 >>>>>>>> [ 2522.956866] [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480 >>>>>>>> [ 2522.956871] [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280 >>>>>>>> [ 2522.956875] [<ffffffff81511c02>] >>>>>>>> bch_btree_map_nodes_recurse+0xe2/0x170 >>>>>>>> [ 2522.956880] [<ffffffff815137c0>] ? >>>>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>>>> [ 2522.956886] [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90 >>>>>>>> [ 2522.956890] [<ffffffff8107f991>] ? update_curr+0x141/0x1f0 >>>>>>>> [ 2522.956895] [<ffffffff81514dce>] >>>>>>>> __bch_btree_map_nodes+0x13e/0x1c0 >>>>>>>> [ 2522.956899] [<ffffffff815137c0>] ? >>>>>>>> bch_btree_insert_node+0x2f0/0x2f0 >>>>>>>> [ 2522.956905] [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0 >>>>>>>> [ 2522.956909] [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120 >>>>>>>> [ 2522.956915] [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160 >>>>>>>> [ 2522.956920] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>>>> [ 2522.956925] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>>>> [ 2522.956930] [<ffffffff81063570>] ? >>>>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>>>> [ 2522.956933] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>>>> [ 2522.956938] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956943] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>>>> [ 2522.956947] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 >>>>>>>> seconds. >>>>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.956959] kworker/3:2 D ffffffff81813a60 0 2471 2 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.956966] Workqueue: events update_writeback_rate >>>>>>>> [ 2522.956968] ffff882a5d125cd0 0000000000000046 0000000000000003 >>>>>>>> ffff882a5d125d68 >>>>>>>> [ 2522.956972] ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8 >>>>>>>> ffff882a5d125fd8 >>>>>>>> [ 2522.956977] ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003 >>>>>>>> ffff882f8ec94cb0 >>>>>>>> [ 2522.956981] Call Trace: >>>>>>>> [ 2522.956987] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.956992] [<ffffffff816bf0fd>] >>>>>>>> rwsem_down_read_failed+0x9d/0xe5 >>>>>>>> [ 2522.956997] [<ffffffff81332c64>] >>>>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>>>> [ 2522.957002] [<ffffffff816bcc74>] ? down_read+0x24/0x2b >>>>>>>> [ 2522.957007] [<ffffffff8151b010>] >>>>>>>> update_writeback_rate+0x30/0x230 >>>>>>>> [ 2522.957011] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>>>> [ 2522.957017] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>>>> [ 2522.957021] [<ffffffff81063570>] ? >>>>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>>>> [ 2522.957025] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>>>> [ 2522.957030] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.957035] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>>>> [ 2522.957039] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than >>>>>>>> 120 >>>>>>>> seconds. >>>>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>> disables this message. >>>>>>>> [ 2522.957056] kworker/u64:2 D ffffffff81813a60 0 2540 2 >>>>>>>> 0x00000000 >>>>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work >>>>>>>> [ 2522.957064] ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738 >>>>>>>> ffff882f86e3be58 >>>>>>>> [ 2522.957069] ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8 >>>>>>>> ffff882a4fdc9fd8 >>>>>>>> [ 2522.957073] ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00 >>>>>>>> ffff8828dba00d98 >>>>>>>> [ 2522.957078] Call Trace: >>>>>>>> [ 2522.957083] [<ffffffff816be299>] schedule+0x29/0x70 >>>>>>>> [ 2522.957088] [<ffffffff816be57e>] >>>>>>>> schedule_preempt_disabled+0xe/0x10 >>>>>>>> [ 2522.957092] [<ffffffff816bc862>] >>>>>>>> __mutex_lock_slowpath+0x112/0x1b0 >>>>>>>> [ 2522.957102] [<ffffffff81484090>] ? >>>>>>>> ata_scsiop_mode_sense+0x380/0x380 >>>>>>>> [ 2522.957107] [<ffffffff816bc3da>] mutex_lock+0x2a/0x50 >>>>>>>> [ 2522.957111] [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0 >>>>>>>> [ 2522.957116] [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0 >>>>>>>> [ 2522.957121] [<ffffffff8112ebb4>] shrink_slab+0x154/0x300 >>>>>>>> [ 2522.957126] [<ffffffff81076828>] ? resched_task+0x68/0x70 >>>>>>>> [ 2522.957130] [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0 >>>>>>>> [ 2522.957135] [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70 >>>>>>>> [ 2522.957140] [<ffffffff8113140f>] >>>>>>>> do_try_to_free_pages+0x20f/0x4b0 >>>>>>>> [ 2522.957146] [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0 >>>>>>>> [ 2522.957152] [<ffffffff81126e9c>] >>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0 >>>>>>>> [ 2522.957158] [<ffffffff8116062a>] alloc_pages_current+0xba/0x170 >>>>>>>> [ 2522.957163] [<ffffffff8112240e>] __get_free_pages+0xe/0x40 >>>>>>>> [ 2522.957168] [<ffffffff81517fc8>] __btree_sort+0x48/0x230 >>>>>>>> [ 2522.957173] [<ffffffff8151765c>] ? >>>>>>>> __bch_btree_iter_init+0x7c/0xc0 >>>>>>>> [ 2522.957178] [<ffffffff81518301>] >>>>>>>> bch_btree_sort_partial+0x101/0x120 >>>>>>>> [ 2522.957182] [<ffffffff8150f000>] ? >>>>>>>> __btree_node_write_done+0x100/0x100 >>>>>>>> [ 2522.957187] [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90 >>>>>>>> [ 2522.957191] [<ffffffff815109ba>] >>>>>>>> bch_btree_node_write+0x36a/0x4a0 >>>>>>>> [ 2522.957196] [<ffffffff8108458b>] ? idle_balance+0xeb/0x150 >>>>>>>> [ 2522.957201] [<ffffffff8106212c>] ? >>>>>>>> pwq_activate_delayed_work+0x4c/0xb0 >>>>>>>> [ 2522.957205] [<ffffffff81511477>] btree_node_write_work+0x57/0x80 >>>>>>>> [ 2522.957210] [<ffffffff810624d4>] process_one_work+0x174/0x490 >>>>>>>> [ 2522.957215] [<ffffffff8106368b>] worker_thread+0x11b/0x370 >>>>>>>> [ 2522.957219] [<ffffffff81063570>] ? >>>>>>>> manage_workers.isra.23+0x2d0/0x2d0 >>>>>>>> [ 2522.957223] [<ffffffff81069f40>] kthread+0xc0/0xd0 >>>>>>>> [ 2522.957228] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> [ 2522.957233] [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0 >>>>>>>> [ 2522.957237] [<ffffffff81069e80>] ? >>>>>>>> flush_kthread_worker+0xb0/0xb0 >>>>>>>> >>>>>>>> >>>>>>>> I can easily reproduce this, please let me know if you require any >>>>>>>> info/test any patch. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Neo >>>>>>>> >>>>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe >>>>>>>> <s.priebe@...fihost.ag> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Kent, >>>>>>>>> >>>>>>>>> a new one under 3.10: >>>>>>>>> >>>>>>>>> 2013-08-26 21:05:30 INFO: task ceph-osd:8939 blocked for more >>>>>>>>> than 120 >>>>>>>>> seconds. >>>>>>>>> 2013-08-26 21:05:30 "echo 0 > >>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>>> disables this message. >>>>>>>>> 2013-08-26 21:05:30 ceph-osd D ffffffff8160f760 0 8939 1 >>>>>>>>> 0x00000000 >>>>>>>>> 2013-08-26 21:05:30 ffff880bd9a61820 0000000000000086 >>>>>>>>> ffff880bd9a61fd8 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:30 ffff880bd9a60010 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:30 ffff880bd9a61fd8 0000000000012c80 >>>>>>>>> ffff880bd7abb1c0 >>>>>>>>> ffff880c48dee380 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff811e8128>] >>>>>>>>> xfs_vm_readpages+0x18/0x20 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810f1da3>] read_pages+0x43/0x100 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810e64df>] ? >>>>>>>>> __page_cache_alloc+0x9f/0xc0 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810f1fab>] >>>>>>>>> __do_page_cache_readahead+0x14b/0x160 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810f1fdc>] ra_submit+0x1c/0x20 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810f22a5>] >>>>>>>>> ondemand_readahead+0x115/0x240 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810f249e>] >>>>>>>>> page_cache_sync_readahead+0x2e/0x40 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810e7a5e>] T.1003+0x33e/0x430 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810e7c30>] >>>>>>>>> generic_file_aio_read+0xe0/0x220 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff811efd0a>] >>>>>>>>> xfs_file_aio_read+0x15a/0x2a0 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff8114095a>] do_sync_read+0x7a/0xb0 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff81140df1>] vfs_read+0xb1/0x130 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff81141757>] SyS_pread64+0x97/0xa0 >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff81526b12>] >>>>>>>>> system_call_fastpath+0x16/0x1b >>>>>>>>> 2013-08-26 21:05:29 INFO: task ceph-osd:8896 blocked for more >>>>>>>>> than 120 >>>>>>>>> seconds. >>>>>>>>> 2013-08-26 21:05:29 "echo 0 > >>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>>> disables this message. >>>>>>>>> 2013-08-26 21:05:29 ceph-osd D ffffffff8160f760 0 8896 1 >>>>>>>>> 0x00000000 >>>>>>>>> 2013-08-26 21:05:29 ffff880bd9a9dc28 0000000000000086 >>>>>>>>> ffff880bd9a9dfd8 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:29 ffff880bd9a9c010 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:29 ffff880bd9a9dfd8 0000000000012c80 >>>>>>>>> ffff880bd79298e0 >>>>>>>>> ffff880c48dd6380 >>>>>>>>> 2013-08-26 21:05:29 Call Trace: >>>>>>>>> 2013-08-26 21:05:29 [<ffffffff810e60f0>] ? >>>>>>>>> __lock_page+0x70/0x70 >>>>>>>>> 2013-08-26 21:05:28 "echo 0 > >>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>>> disables this message. >>>>>>>>> 2013-08-26 21:05:28 ceph-osd D ffffffff8160f760 0 8798 1 >>>>>>>>> 0x00000000 >>>>>>>>> 2013-08-26 21:05:28 ffff880bd998b740 0000000000000086 >>>>>>>>> ffff880bd998bfd8 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:28 ffff880bd998a010 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:28 ffff880bd998bfd8 0000000000012c80 >>>>>>>>> ffff880bd99c0000 >>>>>>>>> ffff880c48dd6380 >>>>>>>>> 2013-08-26 21:05:28 Call Trace: >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff81524874>] schedule+0x24/0x70 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff81525d7d>] >>>>>>>>> rwsem_down_read_failed+0x9d/0xe5 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff812aa0a4>] >>>>>>>>> call_rwsem_down_read_failed+0x14/0x30 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff81523a42>] ? down_read+0x12/0x20 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffffa01ead62>] >>>>>>>>> btree_read_async+0xa2/0x1a0 >>>>>>>>> [bcache] >>>>>>>>> 2013-08-26 21:05:28 [<ffffffffa01f7b73>] >>>>>>>>> closure_queue+0x43/0x60 >>>>>>>>> [bcache] >>>>>>>>> 2013-08-26 21:05:28 [<ffffffffa01eb903>] T.1050+0x63/0x70 >>>>>>>>> [bcache] >>>>>>>>> 2013-08-26 21:05:28 [<ffffffffa01ed6ea>] >>>>>>>>> cached_dev_make_request+0x20a/0x350 [bcache] >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff81281952>] >>>>>>>>> generic_make_request+0xc2/0x100 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff812819f7>] submit_bio+0x67/0x130 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff8117b240>] >>>>>>>>> do_mpage_readpage+0x2a0/0x660 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff81101cee>] ? >>>>>>>>> __inc_zone_page_state+0x2e/0x30 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff810e6581>] ? >>>>>>>>> add_to_page_cache_locked+0x81/0x110 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff8117b76a>] >>>>>>>>> mpage_readpages+0xfa/0x150 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff811e90c0>] ? >>>>>>>>> xfs_get_blocks_direct+0x20/0x20 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff811e90c0>] ? >>>>>>>>> xfs_get_blocks_direct+0x20/0x20 >>>>>>>>> 2013-08-26 21:05:28 [<ffffffff810eeda3>] ? >>>>>>>>> __alloc_pages_nodemask+0x123/0x240 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8115bf93>] ? __close_fd+0x43/0x80 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81526b12>] >>>>>>>>> system_call_fastpath+0x16/0x1b >>>>>>>>> 2013-08-26 21:05:27 INFO: task ceph-osd:8746 blocked for more >>>>>>>>> than 120 >>>>>>>>> seconds. >>>>>>>>> 2013-08-26 21:05:27 "echo 0 > >>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" >>>>>>>>> disables this message. >>>>>>>>> 2013-08-26 21:05:27 ceph-osd D ffffffff8160f760 0 8746 1 >>>>>>>>> 0x00000000 >>>>>>>>> 2013-08-26 21:05:27 ffff880c2dde1d38 0000000000000086 >>>>>>>>> ffff880c2dde1fd8 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:27 ffff880c2dde0010 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 0000000000012c80 >>>>>>>>> 2013-08-26 21:05:27 ffff880c2dde1fd8 0000000000012c80 >>>>>>>>> ffff880c239eb1c0 >>>>>>>>> ffff880c48dd6380 >>>>>>>>> 2013-08-26 21:05:27 Call Trace: >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81524874>] schedule+0x24/0x70 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81522bbd>] >>>>>>>>> schedule_timeout+0x16d/0x200 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81068eba>] ? >>>>>>>>> __queue_delayed_work+0xaa/0x1a0 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81068d19>] ? >>>>>>>>> try_to_grab_pending+0x109/0x190 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff815251e5>] >>>>>>>>> wait_for_completion+0x95/0x110 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8107fd00>] ? >>>>>>>>> try_to_wake_up+0x2a0/0x2a0 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81167797>] ? >>>>>>>>> bdi_queue_work+0x77/0xc0 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81167863>] >>>>>>>>> writeback_inodes_sb_nr+0x83/0xb0 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff811678ea>] >>>>>>>>> writeback_inodes_sb+0x5a/0x70 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8116dd0a>] >>>>>>>>> __sync_filesystem+0x4a/0x50 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8116dd42>] >>>>>>>>> sync_filesystem+0x32/0x60 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90 >>>>>>>>> 2013-08-26 21:05:27 [<ffffffff81526b12>] >>>>>>>>> system_call_fastpath+0x16/0x1b >>>>>>>>> 2013-08-26 21:05:27 INFO: task ceph-osd:8798 blocked for more >>>>>>>>> than 120 >>>>>>>>> seconds. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> >>>>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG: >>>>>>>>> >>>>>>>>>> great! >>>>>>>>>> >>>>>>>>>> Everything seems to work fine now! Except read_dirty always going >>>>>>>>>> to >>>>>>>>>> negative values after a reboot. >>>>>>>>>> >>>>>>>>>> Stefan >>>>>>>>>> >>>>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> schedule_timeout() is not the same as >>>>>>>>>>>>> schedule_timeout_interruptible(). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> just search and replace? So i can try on my own. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The one in read_dirty(), line ~330 >>>>>>>>>>> -- >>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>>>>> linux-bcache" >>>>>>>>>>> in >>>>>>>>>>> the body of a message to majordomo@...r.kernel.org >>>>>>>>>>> More majordomo info at >>>>>>>>>>> http://vger.kernel.org/majordomo-info.html >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>>> linux-bcache" in >>>>>>>>> the body of a message to majordomo@...r.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>> linux-bcache" >>>>>> in >>>>>> the body of a message to majordomo@...r.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists