linux-kernel - Re: livelock in __writeback_inodes

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20121211134113.GA15801@quack.suse.cz>
Date:	Tue, 11 Dec 2012 14:41:13 +0100
From:	Jan Kara <jack@...e.cz>
To:	Fengguang Wu <fengguang.wu@...el.com>
Cc:	Dave Jones <davej@...hat.com>, linux-mm@...ck.org,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Jan Kara <jack@...e.cz>, linux-fsdevel@...r.kernel.org
Subject: Re: livelock in __writeback_inodes_wb ?

On Tue 11-12-12 16:23:27, Wu Fengguang wrote:
> On Wed, Nov 28, 2012 at 09:55:15AM -0500, Dave Jones wrote:
> > We had a user report the soft lockup detector kicked after 22
> > seconds of no progress, with this trace..
> 
> Where is the original report? The reporter may help provide some clues
> on the workload that triggered the bug.
> 
> > :BUG: soft lockup - CPU#1 stuck for 22s! [flush-8:16:3137]
> > :Pid: 3137, comm: flush-8:16 Not tainted 3.6.7-4.fc17.x86_64 #1
> > :RIP: 0010:[<ffffffff812eeb8c>]  [<ffffffff812eeb8c>] __list_del_entry+0x2c/0xd0
> > :Call Trace:
> > : [<ffffffff811b783e>] redirty_tail+0x5e/0x80
> > : [<ffffffff811b8212>] __writeback_inodes_wb+0x72/0xd0
> > : [<ffffffff811b980b>] wb_writeback+0x23b/0x2d0
> > : [<ffffffff811b9b5c>] wb_do_writeback+0xac/0x1f0
> > : [<ffffffff8106c0e0>] ? __internal_add_timer+0x130/0x130
> > : [<ffffffff811b9d2b>] bdi_writeback_thread+0x8b/0x230
> > : [<ffffffff811b9ca0>] ? wb_do_writeback+0x1f0/0x1f0
> > : [<ffffffff8107fde3>] kthread+0x93/0xa0
> > : [<ffffffff81627e04>] kernel_thread_helper+0x4/0x10
> > : [<ffffffff8107fd50>] ? kthread_freezable_should_stop+0x70/0x70
> > : [<ffffffff81627e00>] ? gs_change+0x13/0x13
> > 
> > Looking over the code, is it possible that something could be
> > dirtying pages faster than writeback can get them written out,
> > keeping us in this loop indefitely ?
> 
> The bug reporter should know best whether there are heavy IO.
> 
> However I suspect it's not directly caused by heavy IO: we will
> release &wb->list_lock before each __writeback_single_inode() call,
> which starts writeback IO for each inode.
  Umm, it's not about releasing wb->list_lock I think. Softlockup will
trigger whenever we are looping in a kernel for more than given timeout
(e.g. those 22 s) without sleeping.

> > Should there be something in this loop periodically poking
> > the watchdog perhaps ?
> 
> It seems we failed to release &wb->list_lock in wb_writeback() for
> long time (dozens of seconds). That is, the inode_sleep_on_writeback()
> is somehow not called. However it's not obvious to me how come this
> can happen..
  Maybe, progress is always non-zero but small and nr_pages is high (e.g.
when writeback is triggered by wakeup_flusher_threads()). What filesystem
is the guy using? I remember e.g. btrfs used to have always-dirty inodes
which could confuse us.

>From the backtrace it is clear there's some superblock which has s_umount
locked and we cannot writeback inodes there. So if this superblock contains
most of the dirty pages we need to write and there's another superblock
with always dirty inode we would livelock like observed... So my question
would be about what filesystems are there in the system (/proc/mounts),
what load does trigger this, trigger sysrq-w when the lockup happens.

								Honza
-- 
Jan Kara <jack@...e.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/