linux-ext4 - Re: fsync() errors is unsafe and risks data loss

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180412194626.xq3j3lma446tvvpi@alap3.anarazel.de>
Date:   Thu, 12 Apr 2018 12:46:27 -0700
From:   Andres Freund <andres@...razel.de>
To:     Lukas Czerner <lczerner@...hat.com>
Cc:     Dave Chinner <david@...morbit.com>,
        Andreas Dilger <adilger@...ger.ca>,
        20180410184356.GD3563@...nk.org,
        "Theodore Y. Ts'o" <tytso@....edu>,
        Ext4 Developers List <linux-ext4@...r.kernel.org>,
        Linux FS Devel <linux-fsdevel@...r.kernel.org>,
        Jeff Layton <jlayton@...hat.com>,
        "Joshua D. Drake" <jd@...mandprompt.com>
Subject: Re: fsync() errors is unsafe and risks data loss

Hi,

On 2018-04-12 12:19:26 +0200, Lukas Czerner wrote:
> On Wed, Apr 11, 2018 at 07:32:21PM -0700, Andres Freund wrote:
> 
> > And there's cases where that just doesn't help at all. Being able to
> > untar a database from backup / archive / timetravel / whatnot, and then
> > fsyncing the directory tree to make sure it's actually safe, is really
> > not an insane idea.  Or even just cp -r ing it, and then starting up a
> > copy of the database.  What you're saying is that none of that is doable
> > in a safe way, unless you use special-case DIO using tooling for the
> > whole operation (or at least tools that fsync carefully without ever
> > closing a fd, which certainly isn't the case for cp et al).
> 
> Does not seem like a problem to me, just checksum the thing if you
> really need to be extra safe. You should probably be doing it anyway if
> you backup / archive / timetravel / whatnot.

That doesn't really help, unless you want to sync() and then re-read all
the data to make sure it's the same. Rereading multi-TB backups just to
know whether there was an error that the OS knew about isn't
particularly fun. Without verifying after sync it's not going to improve
the situation measurably, you're still only going to discover that $data
isn't available when it's needed.

What you're saying here is that there's no way to use standard linux
tools to manipulate files and know whether it failed, without filtering
kernel logs for IO errors. Or am I missing something?

Greetings,

Andres Freund