linux-ext4 - Re: Bug#605009: serious performance regression with ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Mon, 29 Nov 2010 13:50:11 -0700
From:	Andreas Dilger <adilger@...ger.ca>
To:	Bernd Schubert <bernd.schubert@...tmail.fm>
Cc:	Ted Ts'o <tytso@....edu>, Jonathan Nieder <jrnieder@...il.com>,
	linux-ext4@...r.kernel.org
Subject: Re: Bug#605009: serious performance regression with ext4

On 2010-11-29, at 08:18, Bernd Schubert wrote:
> Wouldn't it make sense to modify ext4 or even the vfs to do that on
> close() itself? Most applications expect the file to be on disk after
> a close anyway and I also don't see a good reason why one should delay
> a disk write-back after close any longer (well, there are exeption if
> the application is broken, for example such as ha-logd used by pacemaker,
> which did for each line of logs an open, seek, write, flush, close
> sequence..., but at least we have fixed that in -hg now).

This would be terrible for applications like tar that create many hundreds or thousands of files.  Also, doesn't NFS also internally open/close the file for every write?

There would now be an implicit fsync and disk cache flush for every created file.  It would be impossible to create or extract more than about 100 files/second on an HDD due to seek limitations, even if the files are tiny and do not fill the memory.

I can imagine that it might make sense to _start_ writeback sooner than what the VM currently does, if an application is not repeatedly opening, writing, and closing the same file, since this is otherwise dead time in the IO pipeline that could be better utilized.  This kind of background writeout shouldn't trigger a cache flush each, so that multiple writes can be aggregated more efficiently.

Lustre has always been more aggressive than the VM in starting writeout when there are good-sized chunks of data to me written, or if there are a lot of small files that are not being modified, and this significantly improves performance when IO is bursty, which it is in most real-world cases.

Cheers, Andreas

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html