Date:	Sun, 24 Oct 2010 12:43:45 -0400
From:	Ric Wheeler <rwheeler@...hat.com>
To:	Bernd Schubert <bschubert@....com>
CC:	"Ted Ts'o" <tytso@....edu>, Amir Goldstein <amir73il@...il.com>,
	Bernd Schubert <bs_lists@...ef.fastmail.fm>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>,
	Andreas Dilger <adilger@....com>
Subject: Re: ext4_clear_journal_err: Filesystem error recorded from previous
 mount: IO failure

  On 10/24/2010 12:16 PM, Bernd Schubert wrote:
> On 10/24/2010 05:49 PM, Ric Wheeler wrote:
>>    On 10/24/2010 11:39 AM, Bernd Schubert wrote:
>>> On 10/24/2010 05:20 PM, Ric Wheeler wrote:
>>>> This still sounds more like a Lustre issue than an ext4 one, Andreas can fill in
>>>> the technical details.
>>> The underlying device handling is unrelated to Lustre. In that sense it
>>> is just a local filesystem.
>>>
>>>> What ever shared storage sits under ext4 is irrelevant to the fail over case.
>>>>
>>>> Unless Lustre does other magic, they still need to obey the basic cluster rules
>>>> - one mount per cluster.
>>> Yes, one mount per cluster.
>>>
>>>> If Lustre is doing the same trick you would do with active/passive failure over
>>>> clusters that export ext4 via NFS, you would still need to clean up the file
>>>> system before being able to re-export it from a fail over node.
>>> What exactly is your question here? We use pacemaker/stonith to do the
>>> fencing job.
>>> What exactly do you want to clean up? The device is recovered by
>>> journals, Lustre goes into recovery mode, clients reconnect, locks are
>>> updated and incomplete transactions resend.
>>>
>>>
>>> Cheers,
>>> Bernd
>>>
>> What I don't get (certainly might just be me) is why this is a unique issue when
>> used by lustre. Normally, any similar type of fail over will clean up the local
>> file system normally before trying to re-export from the second node.
> Of course that is not a Lustre specific issue, which is why I also did
> not open a Lustre bugzilla, but opened the thread here.
>
>> Why exactly can't you use the same type of recovery here? Is it the fencing
>> agent killing nodes on detection of the file system errors?
> But I'm using the same type of recovery! I just rewrote pacemakers
> default "Filesystem" agent to a lustre_server agent, to include more
> Lustre specific checks. When I then added last week a check for the
> dumpe2fs "Filesystem state", I noticed, that sometimes the error state
> is only set *after* mounting the filesystem, so difficult to script it.
> And as I also wrote, running e2fsck from that script and to do a
> complete fs check is not appropriate, as that might simply time out.
> Again not Lustre specific. So after some discussion, the proposed
> solution is to add a "journal recovery only" option to e2fsck and to do
> that before the mount. I will add that to the 'lustre_server' agent
> (which is part of Lustre now), but leave it to someone else to that for
> the 'Filesystem' agent script (I'm not using that script myself and IMHO
> it is already too complex, as it tries to support all filesystems -
> shell code is ideal anymore then).

Why not simply have your script attempt to mount the file system? If it 
succeeds, it will replay the journal. If it fails, you will need to fall back to 
the long fsck which is unavoidable.

We spend a lot of time and testing to make sure that ext* can be shot at any 
point and come back after a storage outage and still mount.

Ric

> Really, only Lustre specific here is the feature to have a proc file to
> see if filesystem errors came up on a node. That is a missing feature in
> extX and all other linux filesystems I have worked with. And Lustre
> server nodes just means the usage of dozens to hundreds of
> ext3/ext4/ldiskfs devices, so bugs are more likely exposed by that high
> number.
>
>
> Cheers,
> Bernd
>

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists