linux-ext4 - RE: High CPU Utilization When Copying to Ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <341DAA96EE3A8444B6E4657BE8A846EA4B3DA1270A@NDJSSCC06.ndc.nasa.gov>
Date:	Tue, 28 Jun 2011 15:20:39 -0500
From:	"Mccauliff, Sean D. (ARC-PX)[Lockheed Martin Space OPNS]" 
	<sean.d.mccauliff@...a.gov>
To:	Theodore Tso <tytso@....EDU>
CC:	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
Subject: RE: High CPU Utilization When Copying to Ext4

Last time I benchmarked cp and tar with the respective sparse file options they where extremely slow as they (claim) to identify sparseness by contiguous regions of zeros.  This was quite sometime ago so perhaps cp and tar have changed.

Sean
________________________________________
From: Theodore Tso [tytso@....EDU]
Sent: Tuesday, June 28, 2011 1:14 PM
To: Mccauliff, Sean D. (ARC-PX)[Lockheed Martin Space OPNS]
Cc: linux-ext4@...r.kernel.org
Subject: Re: High CPU Utilization When Copying to Ext4

On Jun 28, 2011, at 2:37 PM, Mccauliff, Sean D. (ARC-PX)[Lockheed Martin Space OPNS] wrote:

> uname -a
> Linux sasr200-2.arc.nasa.gov 2.6.38.7-30.fc15.x86_64 #1 SMP Fri May 27 05:15:53 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
>
> There are about 10M files.  Many are small.  There are about 2M files that are sparse files.  It's hen the copy program gets to these files that the cpu usage gets very high.  There are no links of any kind.
>
> The copy program is written in Java, but uses the fiemap to get the logical address ranges that have actually been allocated.  It merges any contiguous logical address ranges when it reads and writes to the new file.

Fiemap?!?   What kind of copy algorithm are you using?

Why aren't you just doing a "read 10 megs from ext3",
"write 10 megs to ext4"?    How does fiemap figure into this?

-- Ted

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html