linux-ext4 - Re: fsync() errors is unsafe and risks data loss

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180412220359.zqefic4gvoimig34@alap3.anarazel.de>
Date:   Thu, 12 Apr 2018 15:03:59 -0700
From:   Andres Freund <andres@...razel.de>
To:     "Theodore Y. Ts'o" <tytso@....edu>
Cc:     Andreas Dilger <adilger@...ger.ca>,
        Ext4 Developers List <linux-ext4@...r.kernel.org>,
        Linux FS Devel <linux-fsdevel@...r.kernel.org>,
        Jeff Layton <jlayton@...hat.com>,
        "Joshua D. Drake" <jd@...mandprompt.com>
Subject: Re: fsync() errors is unsafe and risks data loss

Hi,

On 2018-04-12 17:52:52 -0400, Theodore Y. Ts'o wrote:
> We did something *really* simple/stupid.  We just sent essentially an
> ascii test string out the netlink socket.  That's because what we were
> doing before was essentially scraping the output of dmesg
> (e.g. /dev/kmssg).
> 
> That's actually probably the simplest thing to do, and it has the
> advantage that it will work even on ancient enterprise kernels that PG
> users are likely to want to use.  So you will need to implement the
> dmesg text scraper anyway, and that's probably good enough for most
> use cases.

The worst part of that is, as you mention below, needing to handle a lot
of different error message formats. I guess it's reasonable enough if
you control your hardware, but no such luck.

Aren't there quite realistic scenarios where one could miss kmsg style
messages due to it being a ringbuffer?

> Right, it's a little challenging because the actual regexp's you would
> need to use do vary from device driver to device driver.  Fortunately
> nearly everything is a SCSI/SATA device these days, so there isn't
> _that_ much variability.

There's also SAN / NAS type stuff - not all of that presents as a
SCSI/SATA device, right?

> > Yea, agreed on all that. I don't think anybody actually involved in
> > postgres wants to do anything like that. Seems far outside of postgres'
> > remit.
> 
> Some people on the pg-hackers list were talking about wanting to retry
> the fsync() and hoping that would cause the write to somehow suceed.
> It's *possible* that might help, but it's not likely to be helpful in
> my experience.

Depends on the type of error and storage. ENOSPC, especially over NFS,
has some reasonable chances of being cleared up. And for networked block
storage it's also not impossible to think of scenarios where that'd
work for EIO.

But I think besides hope of clearing up itself, it has the advantage
that it trivially can give *some* feedback to the user. The user'll get
back strerror(ENOSPC) with some decent SQL error code, which'll
hopefully cause them to investigate (well, once monitoring detects high
error rates).   It's much nicer for the user to type COMMIT; get an
appropriate error back etc, than if the database just commits suicide.

Greetings,

Andres Freund