Message-ID: <20100118122437.GF7264@discord.disaster>
Date:	Mon, 18 Jan 2010 23:24:37 +1100
From:	Dave Chinner <david@...morbit.com>
To:	Nick Piggin <npiggin@...e.de>
Cc:	Jan Kara <jack@...e.cz>,
	Hidehiro Kawai <hidehiro.kawai.ez@...achi.com>,
	linux-kernel@...r.kernel.org, linux-ext4@...r.kernel.org,
	Andrew Morton <akpm@...ux-foundation.org>,
	Andreas Dilger <adilger@....com>,
	Theodore Ts'o <tytso@....edu>,
	Satoshi OSHIMA <satoshi.oshima.fk@...achi.com>,
	linux-fsdevel@...r.kernel.org
Subject: Re: IO error semantics

On Mon, Jan 18, 2010 at 05:05:18PM +1100, Nick Piggin wrote:
> On Mon, Jan 18, 2010 at 04:18:47PM +1100, Nick Piggin wrote:
> > We also need to remove some ClearPageUptodate calls I think (similar
> > issues), so keep those in mind too. Unfortunately it looks like there
> > are also a lot of filesystem specific tests of PageUptodate... but you
> > could also move those under the new compatibility s_flag.
> > 
> > I don't know of a really good way to inject and test filesystem errors.
> > Make request failures causes most fs to quickly go readonly or have
> > bigger problems. If you're careful like try to only fail read IOs for
> > data, or only fail write IOs not involved in integrity or journal
> > operations, then test programs just tend to abort pretty quickly. Does
> > anyone know of anything more systematic?
> 
> This might be a good time to bring up IO error behaviour again. I got
> into some debates I think on Andi's hwpoison thread a while back, but
> probably not appropriate thread to find a real solution to this.
> 
> The problem we have now is that IO error semantics are not well defined.
> It is hard to even enumerate all the issues.
> 
> read IOs
>   how to retry? appropriate defaults should happen at the block layer I
>   think. Should retry behaviour be tunable by the mm/fs, or should that
>   be coded explicitly as submission retry loops? Either way does imply
>   there is either similar defaults for all types (or maybe classes) of
>   drivers, or some way to query/set this.

It's more complex than that - there are classes of errors to
consider as well. e.g transient vs permanent.

Transient is from stuff like FC path failures - failover can take up
to 240s to occur, and then the IO will generally complete
successfully.  Permanent errors are those that involve data loss e.g
bad sectors on single disks or on degraded RAID devices.

The action to take is generally different for different error
classes - transient errors can be retried later, while permanent
errors won't change no matter how many retries you do.  IOWs, we'll
need help from the block layer to enable us to determine the error
class we are dealing with.

>   It would be nice to be able to set fs/driver behaviour from userspace
>   too, in a generic (not driver or fs specific way). But defaults should
>   be reasonable and similar between all, I guess.

I don't think generic handling is really possible - filesystems may
have different ways of recovering e.g. duplicate copies of data or
metadata or internal ECC that can be used to recovery the bad
region. Also, depending where the error occurs, the filesystem might
need to shutdown to be repaired....

> write IOs
>   This is more interesting. How to handle write IO errors. In my opinion
>   we must not invalidate the data before an IO error is returned to
>   somebody (whether it be fsync or a synchronous write syscall).

We already pass the error via mapping_set_error() calls when the
error occurs and checking in it filemap_fdatawait_range().  However,
where we check the error we've lost all context and what range the
error occurred on. I don't see any easy way to track such an
error for later invalidation except maybe by a new radix tree tag.
That would allow later invalidation of only the specific range the
error was reported from.

>   Any
>   earlier and the app just gets RAW consistency randomly violated. And I
>   think it is important to treat IO errors as transparently as possible
>   until the error can be detected.
> 
>   I happen to think that actually we should go further and not
>   invalidate the data at all. This makes implementation simpler, and
>   also allows us to retry writes like we can retry reads. It's also
>   problematic to throw out errors at that point because *sync syscalls
>   coming from elsewhere could result in loss of error reporting (think,
>   sys_sync).

The worst problem with this is what happens when you can't write
back to the filesystem because of IO errors, but you still allow more
incoming writes? It's not far from IO error to running out of memory
and deadlocking....

>   If we go this way, we probably need another syscall and fs helper call
>   to invalidate the dirty data when we give up on retries. truncate_range
>   probably not appropriate because it is much harder to implement and
>   maybe we want to try to get at the most recent data that is on disk.

First we need to track what needs invalidating...

>   Also do we need to think about O_SYNC or -o sync type of writes that
>   are implemented via writeback cache? We could invalidate the dirtied
>   cache ASAP, which would leave a window where a concurrent read can see
>   first new, then old data. It would also kind of break the above scheme
>   in case the pagecache was already dirty via a descriptor without
>   O_SYNC. It might just make sense to leave the pagecache dirty. Either
>   way it should be documented I think.

How to handle this comes down to the type of error that occurred. In
the case of permanent error, the second read after the invalidation
probably should return EIO because you have no idea whether what is on
disk is the old, the new, some combination of the two or some other
random or stale garbage....

> Do we even care enough to bother thinking about this now? (serious question)

It's a damn hard problem and many of the details are filesystem
specific. However, if we want high grade reliability from our
systems then we have to tackle these problems at some point in time.

FWIW, I started to document some of what I've just been talking
(from a XFS metadata reliability context) about a year and a half
ago. The relevant section is here:

http://xfs.org/index.php/Reliable_Detection_and_Repair_of_Metadata_Corruption#Exception_Handling

Though the entire page is probably somewhat relevant.  I only got as
far as documenting methods for handling transient and permanent read
errors, and the TODO includes handling:

	- Transient write error
	- Permanent write error
	- Corrupted data on read
	- Corrupted data on write (detected during guard calculation)
	- I/O timeouts
	- Memory corruption

If we want to handle these sort of errors, I think first we've got
to understand what we have to do. A rough outline of the approach I
was taking to the above document was:

	- work out how to detect each type of error effectively
	- determine strategies of how to report each type of error
	- find the best way to recovery from the error
	- work out what additional information was needed on disk
	  to enable successful recovery from the error.
	- work out what addition functionality was required from
	  the lower layers to allow reliable detection and recovery
	  to ぽccur.

In the case of reporting data errors through pagecache paths, I'd add:

	- determine if the error can be handled in a generic way
	- work out how filesystem specific handlers can be invoked
	  for those that can do filesystem level recovery
	- work out how to consistently report the same result after
	  an error.

There's plenty more I could write, but I need to sleep now....

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists