linux-kernel - ext4 perf regression on LTS kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20200430223458.GY13035@sasha-vm>
Date:   Thu, 30 Apr 2020 18:34:58 -0400
From:   Sasha Levin <sashal@...nel.org>
To:     tytso@....edu, adilger.kernel@...ger.ca
Cc:     linux-ext4@...r.kernel.org, stable@...r.kernel.org,
        linux-kernel@...r.kernel.org, runderwood@...kedin.com
Subject: ext4 perf regression on LTS kernels

Hi folks,

We're working on trying to figure out a severe performance regression in
the 5.4 and older LTS trees. The regression seems to happen only on
physical spinning rust disks, which is why it was probably went
unnoticed.

The regression seems to be introduced in v4.7 with:

        1f60fbe72749 ("ext4: allow readdir()'s of large empty directories to be interrupted")

The fio test used to reproduce it is:

	sync; i=0; while [ $i -lt 4 ]; do ( ( time fio
	--name=disk-burner --readwrite=write --bs=4096 --invalidate=1
	--end_fsync=0 --filesize=800M --runtime=120 --ioengine=libaio
	--thread --numjobs=20 --iodepth=1 --unlink=1 ) 2>&1 | grep
	'^real' ); ((i++)); done

When run with the offending commit, it'll take 3-4x longer to complete.

The regression was fixed upstream somewhere in this merge:

        e5da4c933c50 ("Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4")

but it seems to be a combination of commits that fix it rather than a
single one.

Now, here's the tricky part... reverting these two commits on top of
v4.19.118 "fixes" the issue:

        06bd3c36a733 ("ext4: fix data exposure after a crash")
        1f60fbe72749 ("ext4: allow readdir()'s of large empty directories to be interrupted")

but clearly this is not something we want to do in the stable trees, so
we're trying to figure out the proper way to fix this.

-- 
Thanks,
Sasha