| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Mon, 26 Aug 2013 21:21:47 +0200 From: Stefan Priebe <s.priebe@...fihost.ag> To: Kent Overstreet <kmo@...erainc.com> CC: Jens Axboe <axboe@...nel.dk>, linux-kernel@...r.kernel.org, linux-bcache@...r.kernel.org Subject: Re: bcache: Fix a writeback performance regression Hi Kent, a new one under 3.10: 2013-08-26 21:05:30 INFO: task ceph-osd:8939 blocked for more than 120 seconds. 2013-08-26 21:05:30 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2013-08-26 21:05:30 ceph-osd D ffffffff8160f760 0 8939 1 0x00000000 2013-08-26 21:05:30 ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8 0000000000012c80 2013-08-26 21:05:30 ffff880bd9a60010 0000000000012c80 0000000000012c80 0000000000012c80 2013-08-26 21:05:30 ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0 ffff880c48dee380 2013-08-26 21:05:29 [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20 2013-08-26 21:05:29 [<ffffffff810f1da3>] read_pages+0x43/0x100 2013-08-26 21:05:29 [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0 2013-08-26 21:05:29 [<ffffffff810f1fab>] __do_page_cache_readahead+0x14b/0x160 2013-08-26 21:05:29 [<ffffffff810f1fdc>] ra_submit+0x1c/0x20 2013-08-26 21:05:29 [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240 2013-08-26 21:05:29 [<ffffffff810f249e>] page_cache_sync_readahead+0x2e/0x40 2013-08-26 21:05:29 [<ffffffff810e7a5e>] T.1003+0x33e/0x430 2013-08-26 21:05:29 [<ffffffff810e7c30>] generic_file_aio_read+0xe0/0x220 2013-08-26 21:05:29 [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0 2013-08-26 21:05:29 [<ffffffff8114095a>] do_sync_read+0x7a/0xb0 2013-08-26 21:05:29 [<ffffffff81140df1>] vfs_read+0xb1/0x130 2013-08-26 21:05:29 [<ffffffff81141757>] SyS_pread64+0x97/0xa0 2013-08-26 21:05:29 [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b 2013-08-26 21:05:29 INFO: task ceph-osd:8896 blocked for more than 120 seconds. 2013-08-26 21:05:29 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2013-08-26 21:05:29 ceph-osd D ffffffff8160f760 0 8896 1 0x00000000 2013-08-26 21:05:29 ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8 0000000000012c80 2013-08-26 21:05:29 ffff880bd9a9c010 0000000000012c80 0000000000012c80 0000000000012c80 2013-08-26 21:05:29 ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0 ffff880c48dd6380 2013-08-26 21:05:29 Call Trace: 2013-08-26 21:05:29 [<ffffffff810e60f0>] ? __lock_page+0x70/0x70 2013-08-26 21:05:28 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2013-08-26 21:05:28 ceph-osd D ffffffff8160f760 0 8798 1 0x00000000 2013-08-26 21:05:28 ffff880bd998b740 0000000000000086 ffff880bd998bfd8 0000000000012c80 2013-08-26 21:05:28 ffff880bd998a010 0000000000012c80 0000000000012c80 0000000000012c80 2013-08-26 21:05:28 ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000 ffff880c48dd6380 2013-08-26 21:05:28 Call Trace: 2013-08-26 21:05:28 [<ffffffff81524874>] schedule+0x24/0x70 2013-08-26 21:05:28 [<ffffffff81525d7d>] rwsem_down_read_failed+0x9d/0xe5 2013-08-26 21:05:28 [<ffffffff812aa0a4>] call_rwsem_down_read_failed+0x14/0x30 2013-08-26 21:05:28 [<ffffffff81523a42>] ? down_read+0x12/0x20 2013-08-26 21:05:28 [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0 [bcache] 2013-08-26 21:05:28 [<ffffffffa01f7b73>] closure_queue+0x43/0x60 [bcache] 2013-08-26 21:05:28 [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache] 2013-08-26 21:05:28 [<ffffffffa01ed6ea>] cached_dev_make_request+0x20a/0x350 [bcache] 2013-08-26 21:05:28 [<ffffffff81281952>] generic_make_request+0xc2/0x100 2013-08-26 21:05:28 [<ffffffff812819f7>] submit_bio+0x67/0x130 2013-08-26 21:05:28 [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660 2013-08-26 21:05:28 [<ffffffff81101cee>] ? __inc_zone_page_state+0x2e/0x30 2013-08-26 21:05:28 [<ffffffff810e6581>] ? add_to_page_cache_locked+0x81/0x110 2013-08-26 21:05:28 [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150 2013-08-26 21:05:28 [<ffffffff811e90c0>] ? xfs_get_blocks_direct+0x20/0x20 2013-08-26 21:05:28 [<ffffffff811e90c0>] ? xfs_get_blocks_direct+0x20/0x20 2013-08-26 21:05:28 [<ffffffff810eeda3>] ? __alloc_pages_nodemask+0x123/0x240 2013-08-26 21:05:27 [<ffffffff8115bf93>] ? __close_fd+0x43/0x80 2013-08-26 21:05:27 [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20 2013-08-26 21:05:27 [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b 2013-08-26 21:05:27 INFO: task ceph-osd:8746 blocked for more than 120 seconds. 2013-08-26 21:05:27 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2013-08-26 21:05:27 ceph-osd D ffffffff8160f760 0 8746 1 0x00000000 2013-08-26 21:05:27 ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8 0000000000012c80 2013-08-26 21:05:27 ffff880c2dde0010 0000000000012c80 0000000000012c80 0000000000012c80 2013-08-26 21:05:27 ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0 ffff880c48dd6380 2013-08-26 21:05:27 Call Trace: 2013-08-26 21:05:27 [<ffffffff81524874>] schedule+0x24/0x70 2013-08-26 21:05:27 [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200 2013-08-26 21:05:27 [<ffffffff81068eba>] ? __queue_delayed_work+0xaa/0x1a0 2013-08-26 21:05:27 [<ffffffff81068d19>] ? try_to_grab_pending+0x109/0x190 2013-08-26 21:05:27 [<ffffffff815251e5>] wait_for_completion+0x95/0x110 2013-08-26 21:05:27 [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0 2013-08-26 21:05:27 [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0 2013-08-26 21:05:27 [<ffffffff81167863>] writeback_inodes_sb_nr+0x83/0xb0 2013-08-26 21:05:27 [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70 2013-08-26 21:05:27 [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50 2013-08-26 21:05:27 [<ffffffff8116dd42>] sync_filesystem+0x32/0x60 2013-08-26 21:05:27 [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90 2013-08-26 21:05:27 [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b 2013-08-26 21:05:27 INFO: task ceph-osd:8798 blocked for more than 120 seconds. Stefan Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG: > great! > > Everything seems to work fine now! Except read_dirty always going to > negative values after a reboot. > > Stefan > > Am 22.08.2013 08:02, schrieb Kent Overstreet: >> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote: >>> >>>> schedule_timeout() is not the same as >>>> schedule_timeout_interruptible(). >>> >>> just search and replace? So i can try on my own. >> >> The one in read_dirty(), line ~330 >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >> the body of a message to majordomo@...r.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists