linux-ext4 - Re: fsync() errors is unsafe and risks data loss

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 12 Apr 2018 11:16:46 -0400
From:   "Theodore Y. Ts'o" <tytso@....edu>
To:     Dave Chinner <david@...morbit.com>
Cc:     Jeff Layton <jlayton@...hat.com>,
        Matthew Wilcox <willy@...radead.org>,
        Andres Freund <andres@...razel.de>,
        Andreas Dilger <adilger@...ger.ca>,
        20180410184356.GD3563@...nk.org,
        Ext4 Developers List <linux-ext4@...r.kernel.org>,
        Linux FS Devel <linux-fsdevel@...r.kernel.org>,
        "Joshua D. Drake" <jd@...mandprompt.com>
Subject: Re: fsync() errors is unsafe and risks data loss

On Thu, Apr 12, 2018 at 10:01:22PM +1000, Dave Chinner wrote:
> On Thu, Apr 12, 2018 at 07:09:14AM -0400, Jeff Layton wrote:
> > When there is a writeback error, what should be done with the dirty
> > page(s)? Right now, we usually just mark them clean and carry on. Is
> > that the right thing to do?
> 
> There isn't a right thing. Whatever we do will be wrong for someone.

That's the problem.  The best that could be done (and it's not enough)
would be to have a mode which does with the PG folks want (or what
they *think* they want).  It seems what they want is to have an error
result in the page being marked clean.  When they discover the outcome
(OOM-city and the unability to unmount a file system on a failed
drive), then they will complain to us *again*, at which point we can
tell them that want they really want is another variation on O_PONIES,
and welcome to the real world and real life.

Which is why, even if they were to pay someone to implement what they
want, I'm not sure we would want to accept it upstream --- or distro's
might consider it a support nightmare, and refuse to allow that mode
to be enabled on enterprise distro's.  But at least, it will have been
some PG-based company who will have implemented it, so they're not
wasting other people's time or other people's resources...

We could try to get something like what Google is doing upstream,
which is to have the I/O errors sent to userspace via a netlink
channel (without changing anything else about how buffered writeback
is handled in the face of errors).  Then userspace applications could
switch to Direct I/O like all of the other really serious userspace
storage solutions I'm aware of, and then someone could try to write
some kind of HDD health monitoring system that tries to do the right
thing when a disk is discovered to have developed some media errors or
something more serious (e.g., a head failure).  That plus some kind of
RAID solution is I think the only thing which is really realistic for
a typical PG site.

It's certainly that's what *I* would do if I didn't decide to use a
hosted cloud solution, such as Cloud SQL for Postgres, and let someone
else solve the really hard problems of dealing with real-world HDD
failures.   :-)

					- Ted