linux-ext4 - Re: Atomic non-durable file write API

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 24 Dec 2010 12:14:21 +0100
From:	Olaf van der Spek <olafvdspek@...il.com>
To:	"Ted Ts'o" <tytso@....edu>
Cc:	linux-fsdevel <linux-fsdevel@...r.kernel.org>,
	linux-ext4@...r.kernel.org
Subject: Re: Atomic non-durable file write API

On Fri, Dec 24, 2010 at 10:51 AM, Ted Ts'o <tytso@....edu> wrote:
> On Fri, Dec 24, 2010 at 02:00:13AM +0100, Christian Stroetmann wrote:
>> I really do know what you want to say, despite that this example is
>> based on a bug in another system than the FS. But there will be
>> other examples, for sure.
>
> Sure, but this thread started because someone wanted an "atomic
> non-durable file write API", apparently because it was too slow to use
> fsync().  If people use databases, it's not a problem; databases use
> fsync(), but they use it properly and they provide the proper
> transactional interfaces that people want.
>
> The problem comes when people try to implement their own databases
> using small files for each row and column of the database, or for each
> registry variable.  Then they complain when fsync() is to expensive,
> because they need to use fsync() for every single 3 bytes of data they
> store in their badly implemented database.
>
> The bottom line is that if you want atomic updates of state
> information, you need to use fsync() or fdatasync().  If this is a
> performance bottleneck, then you're doing something wrong.  Maybe you
> shouldn't be writing a third of a megabyte on every URL click, on the
> main GUI thread; maybe the user doesn't need to remember every single
> URL that was visited even if the power suddenly fails (maybe it's
> enough if you write that information to disk every 3-5 minutes, and
> less if you're running on battery).  Or maybe you shouldn't be using
> hundreds of small state files, and screw up the dirty flag handling.
> But regardless, you're doing something wrong/stupid.

Hi Ted,

Thanks for taking the time to answer. The thread was started due to
the dpkg issue.
The questions were:
> What is the recommended way for atomic non-durable (complete) file writes?

It seems you're saying fsync is required, but why can't atomic be
provided without durable? Is it just an API issue?

If rename is recommended, how does one preserve meta-data including file owner?

> I'm also wondering why FSs commit after open/truncate but before
write/close. AFAIK this isn't necessary and thus suboptimal.

Olaf
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html