lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 12 Apr 2018 12:46:27 -0700
From:   Andres Freund <andres@...razel.de>
To:     Lukas Czerner <lczerner@...hat.com>
Cc:     Dave Chinner <david@...morbit.com>,
        Andreas Dilger <adilger@...ger.ca>,
        20180410184356.GD3563@...nk.org,
        "Theodore Y. Ts'o" <tytso@....edu>,
        Ext4 Developers List <linux-ext4@...r.kernel.org>,
        Linux FS Devel <linux-fsdevel@...r.kernel.org>,
        Jeff Layton <jlayton@...hat.com>,
        "Joshua D. Drake" <jd@...mandprompt.com>
Subject: Re: fsync() errors is unsafe and risks data loss

Hi,

On 2018-04-12 12:19:26 +0200, Lukas Czerner wrote:
> On Wed, Apr 11, 2018 at 07:32:21PM -0700, Andres Freund wrote:
> 
> > And there's cases where that just doesn't help at all. Being able to
> > untar a database from backup / archive / timetravel / whatnot, and then
> > fsyncing the directory tree to make sure it's actually safe, is really
> > not an insane idea.  Or even just cp -r ing it, and then starting up a
> > copy of the database.  What you're saying is that none of that is doable
> > in a safe way, unless you use special-case DIO using tooling for the
> > whole operation (or at least tools that fsync carefully without ever
> > closing a fd, which certainly isn't the case for cp et al).
> 
> Does not seem like a problem to me, just checksum the thing if you
> really need to be extra safe. You should probably be doing it anyway if
> you backup / archive / timetravel / whatnot.

That doesn't really help, unless you want to sync() and then re-read all
the data to make sure it's the same. Rereading multi-TB backups just to
know whether there was an error that the OS knew about isn't
particularly fun. Without verifying after sync it's not going to improve
the situation measurably, you're still only going to discover that $data
isn't available when it's needed.

What you're saying here is that there's no way to use standard linux
tools to manipulate files and know whether it failed, without filtering
kernel logs for IO errors. Or am I missing something?

Greetings,

Andres Freund

Powered by blists - more mailing lists