linux-ext4 - Quadrant write performance degradation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 16 Jun 2014 09:02:08 +0300
From:	Tanya Brokhman <tlinder@...eaurora.org>
To:	linux-fsdevel@...r.kernel.org, linux-ext4@...r.kernel.org
CC:	kdorfman@...eaurora.org, merez@...eaurora.org,
	Dolev Raviv <draviv@...eaurora.org>, tlinder@...eaurora.org
Subject: Quadrant write performance degradation - kernel3.10 vs kernel3.4

Hello,
Recently we encountered a performance degradation on 3.10kernel based 
build, compared to 3.4 based one, when running the fs_write Quadrant 
benchmark.
We profiled the test and came to the conclusion that the root cause of 
the degradation is in the vfs_write call stack (overhead of 2611.2us is 
observed in 3.10 kernel compared to 3.4):

ret_fast_syscall
SyS_write
vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us)
do_sync_write
ext4_file_write
generic_file_aio_write (total time spent: 3.10kernel-19124.4us, 
3.4kernel-16815us)
__generic_file_aio_write
generic_file_buffered_write
ext4_da_write_begin (total time spent: 3.10kernel-10935.2us, 
3.4kernel-8444.6us)
__block_write_begin
ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us, 
3.4kernel-3576.8us)
ext4_es_lookup_extent  (total time spent: 3.10kernel-2219.7us, 
3.4kernel-0us)

We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel) 
build and got an improvement of 50% in the test result.
When looking deeper into the changes made to the ext4 FS between 3.4 and 
3.10 versions we stumbled across two major features making an explicit 
tradeoff in favor of robustness and good design over performance in some 
use cases:

1) Metadata Checksums 
http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8
“As far as performance impact goes, it shouldn't be noticeable for 
common desktop and server workloads. A mail server ffsb simulation show 
nearly no change. On a test doing only file creation and deletion and 
extent tree modifications, a performance drop of about 20 percent was 
measured. However, it's a workload very heavily oriented towards 
metadata, in most real-world workloads metadata is usually a small 
fraction of total IO, so unless your workload is metadata-oriented, the 
cost of enabling this feature should be negligible.”

2) Extents status tracking: 
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20
“There is a cache extent for write access, so if writes are not very 
random, adding space operations are in O(1) time.”

We tried pick up several performance-enhancement patches from the 
community, released between 3.10 and 3.14 kernel versions. The 
performance was almost the same.

I was wondering what performance tests were performed on these features? 
Has anyone encountered same issue?

Best Regards
Tanya Brokhman
-- 
QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html