linux-kernel - Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4D4F1A3D.2090107@hardwarefreak.com>
Date:	Sun, 06 Feb 2011 16:01:33 -0600
From:	Stan Hoeppner <stan@...dwarefreak.com>
To:	Justin Piszcz <jpiszcz@...idpixels.com>
CC:	"Dr. David Alan Gilbert" <linux@...blig.org>,
	Emmanuel Florac <eflorac@...ellique.com>,
	linux-kernel@...r.kernel.org, linux-scsi@...r.kernel.org,
	linux-net@...r.kernel.org, Alan Piszcz <ap@...arrain.com>
Subject: Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?

Justin Piszcz put forth on 2/6/2011 4:16 AM:

> Workflow process-
> 
> Migrate data from old/legacy RAID sets to new ones, possibly also 2TB->3TB, so
> the faster the transfer speed, the better.

This type of data migration is probably going to include many many files of
various sizes from small to large.  You have optimized your system performance
only for individual large file xfers.  Thus, when you go to copy directories
containing hundreds or thousands of files of various sizes, you will likely see
much lower throughput using a single copy stream.  Thus if you want to keep that
10 GbE pipe full, you'll likely need to run multiple copies in parallel, one per
large parent directory.  Or, run a single copy from say, 10 legacy systems to
one new system simultaneously, etc.

Given this situation, you may want to consider tar'ing up entire directories
with gz or bz compression, if you have enough free space on the legacy machines,
and copying the tarballs to the new system.  This will maximize your throughput,
although I don't know if it will decrease your total work flow completion time,
which should really be your overall goal.

-- 
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/