linux-kernel - Re: Wislist for Linux from the mold linker's POV

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2c33be3f-8c41-48f1-a6ad-b4ea00ec515f@nh2.me>
Date: Fri, 29 Nov 2024 06:38:47 +0100
From: Niklas Hambüchen <mail@....me>
To: Rui Ueyama <rui314@...il.com>
Cc: LKML <linux-kernel@...r.kernel.org>, Florian Weimer <fw@...eb.enyo.de>
Subject: Re: Wislist for Linux from the mold linker's POV

Hi Rui,

On 2024-11-29 01:44, Rui Ueyama wrote:
> ext4 on a PCIe Gen.5 SSD, but I guess it probably doesn't matter much
> because we observed similar results even on tmpfs

When dealing with ext4, there's another behaviour useful to know, which is the opposite of what you observed:

When files are overwritten, it can suddenly be ~10x slower than if they are deleted and written from scratch.

I thought that shouldn't be, because the files are re-written from scratch using `O_WRONLY|O_CREAT|O_TRUNC` in both cases.

Turns out, `ext4` has built in a feature to work around bad applications forgetting `fsync()`:

`close()`ing new files is fast.
But if you `close()` existing files after writing them from scratch, or atomic-rename something replacing them, ext4 will insert an `fsync()`!

Sources:
* https://superuser.com/questions/865710/write-to-newfile-vs-overwriting-performance-issue/872056
* https://www.kernel.org/doc/html/latest/admin-guide/ext4.html section `auto_da_alloc`
* https://en.wikipedia.org/wiki/Ext4#Delayed_allocation_and_potential_data_loss