[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 8 Sep 2011 23:05:54 -0400
From: Greg Freemyer <greg.freemyer@...il.com>
To: Jan Kara <jack@...e.cz>
Cc: Masayoshi MIZUMA <m.mizuma@...fujitsu.com>,
Andrew Morton <akpm@...ux-foundation.org>,
Andreas Dilger <adilger.kernel@...ger.ca>,
linux-ext4@...r.kernel.org, linux-fsdevel@...r.kernel.org,
Valerie Aurora <val@...consulting.com>
Subject: Re: [BUG] ext3: cannot unfreeze a filesystem due to a deadlock
On Wed, Sep 7, 2011 at 6:32 PM, Jan Kara <jack@...e.cz> wrote:
>
> On Wed 07-09-11 13:56:08, Greg Freemyer wrote:
> > On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara <jack@...e.cz> wrote:
> > > Hello,
> > >
> > > Thanks for report!
> > >
> > > On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
> > >> When I checked the freeze feature for ext3 filesystem using fsfreeze
> > >> command at 3.1.0-rc4, I think the following deadlock problem happened.
> > >>
> > >> How to reproduce:
> > >> # mkfs -t ext3 /dev/sdd1
> > >> # mount /dev/sdd1 /MNT
> > >> # ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
> > >> # fsfreeze -f /MNT
> > >> # fsfreeze -u /MNT
> > >>
> > >> If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
> > >>
> > >> The detail of deadlock:
> > >> o [flush-8:16:1523]
> > >> wb_do_writeback
> > >> wb_writeback
> > >> ...
> > >> ext3_journalled_writepage
> > >> journal_start
> > >> start_this_handle
> > >> # waiting until journal->j_barrier_count turns 0...
> > >> # j_barrier_count was incremented by journal_lock_updates()
> > >> # via ext3_freeze().
> > >>
> > >> o [fsstress:2673]
> > >> sys_sync
> > >> sync_filesystems
> > >> iterate_supers
> > >> down_read(sb->s_umount)
> > >> sync_one_sb
> > >> __sync_filesystem
> > >> writeback_inodes_sb
> > >> writeback_inodes_sb_nr
> > >> wait_for_completion
> > >> wait_for_common
> > >> # waiting for completion of [flush-8:16:1523]...
> > >>
> > >> o [fsfreeze:2749]
> > >> sys_ioctl
> > >> do_vfs_ioctl
> > >> thaw_super
> > >> # waiting for down_write(sb->s_umount)...
> > >> # [fsfreeze:2673] did down_read(sb->s_umount).
> > > Yes, this is a classical deadlock that can happen for any filesystem. The
> > > problem is flusher thread holds s_umount semaphore (either directly, or as
> > > in your case, indirectly via blocked sync) and tries to do some IO which
> > > blocks on frozen filesystem. It's particularly easy to hit for ext3 because
> > > it doesn't do vfs_check_frozen() checks but all other filesystems have the
> > > race window as well. Val Henson is working on fixing the problem - she even
> > > has some first version of patches I believe.
> > >
> > > Honza
> >
> > xfstests test 068 has been around since kernel 2.4 days and should
> > have caught it if xfs is impacted.
> >
> > I know I ran the 2002 version many times to prove to myself that
> > fsfreeze for xfs was stable when teamed with LVM. (It wasn't when I
> > first wrote 068 way back then).
> >
> > 068 has been greatly simplified since 2002, but it still looks like it
> > should do a good job.
> >
> > Is there a problem with 068? Does it need extra test coverage even for xfs?
> I believe at least mmapped writes can trigger the deadlock even for xfs
> and fsstress (slightly surprisingly) does not test that. It's a narrow race
> window but it is there and it has been triggered in practice (for ext4 but
> it's a race in VFS code used by both XFS and ext4). So maybe extending
> fsstress would be a way to go?
>
> Honza
That's a surprisingly large hole in xfstests.
That sounds like a pretty core and significant change. I'll have to
leave that to one of the main developers.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists