linux-kernel - Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <AANLkTim8Qt25C+L+R3e-R2qW0K6XxQBv-Fsy8K=aVA2H@mail.gmail.com>
Date:	Mon, 7 Feb 2011 14:59:26 +1100
From:	Julian Calaby <julian.calaby@...il.com>
To:	Stan Hoeppner <stan@...dwarefreak.com>
Cc:	Justin Piszcz <jpiszcz@...idpixels.com>,
	"Dr. David Alan Gilbert" <linux@...blig.org>,
	Emmanuel Florac <eflorac@...ellique.com>,
	linux-kernel@...r.kernel.org, linux-scsi@...r.kernel.org,
	linux-net@...r.kernel.org, Alan Piszcz <ap@...arrain.com>
Subject: Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over 10GbE?

On Mon, Feb 7, 2011 at 09:01, Stan Hoeppner <stan@...dwarefreak.com> wrote:
> Justin Piszcz put forth on 2/6/2011 4:16 AM:
>
>> Workflow process-
>>
>> Migrate data from old/legacy RAID sets to new ones, possibly also 2TB->3TB, so
>> the faster the transfer speed, the better.
>
> This type of data migration is probably going to include many many files of
> various sizes from small to large.  You have optimized your system performance
> only for individual large file xfers.  Thus, when you go to copy directories
> containing hundreds or thousands of files of various sizes, you will likely see
> much lower throughput using a single copy stream.  Thus if you want to keep that
> 10 GbE pipe full, you'll likely need to run multiple copies in parallel, one per
> large parent directory.  Or, run a single copy from say, 10 legacy systems to
> one new system simultaneously, etc.
>
> Given this situation, you may want to consider tar'ing up entire directories
> with gz or bz compression, if you have enough free space on the legacy machines,
> and copying the tarballs to the new system.  This will maximize your throughput,
> although I don't know if it will decrease your total work flow completion time,
> which should really be your overall goal.

Another option might be to use tar and gzip to bundle the data up,
then pipe it through netcat or ssh. When I have to transfer large
chunks of data I find this is the fastest method. That said, if the
connection is interrupted, then you're on your own. rsync might also
be a good option.

Thanks,

-- 
Julian Calaby

Email: julian.calaby@...il.com
Profile: http://www.google.com/profiles/julian.calaby/
.Plan: http://sites.google.com/site/juliancalaby/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/