Solved

Raid 1 - 5Mil+ File Transfer

Posted on 2012-12-31
6
451 Views
Last Modified: 2013-01-01
I am trying to move some data from one external hard drive(which I removed and installed the drive in the machine) onto a new raid 1 array. The transfer speeds seem very slow, but I am not sure as I have never moved this amount of files before.

I am trying to move 5,000,000+ 75-150kb files onto the array, the transfer speeds are about 5MB, is this normal? It seems slow as 160GB has taken 5hours...
0
Comment
Question by:ThermoDust84
6 Comments
 
LVL 8

Assisted Solution

by:teomcam
teomcam earned 400 total points
ID: 38734123
Hi,

Yes that is very very normal as your number of files are a lot. For example;

1GB data as 1 File only= 1 minute
1GB data as 500 files =8 minutes
0
 

Author Comment

by:ThermoDust84
ID: 38734125
Darn, any way to improve the speed?
0
 
LVL 47

Expert Comment

by:dlethe
ID: 38734130
Get a faster external HDD interface.  Looks like you are using USB2.  Go USB3 or eSATA and you'll be at least 10X faster.

(Or just leave the HDD inside the machine and copy via 1GigE ethernet to another computer. That also will be at least 10X faster.
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 

Author Comment

by:ThermoDust84
ID: 38734139
dlethe Posted on 2013-01-01 at 00:10:21ID: 38734130

 Get a faster external HDD interface.  Looks like you are using USB2.  Go USB3 or eSATA and you'll be at least 10X faster.

(Or just leave the HDD inside the machine and copy via 1GigE ethernet to another computer. That also will be at least 10X faster.

It is in the machine, I removed it from the external.
0
 
LVL 8

Accepted Solution

by:
teomcam earned 400 total points
ID: 38734156
There is not much you can do, that is nature of the data transfer. If the number of files are so much then you will experince surprisingly slow speeds as all files indivudally have their own attiributes to be written. Upon completion of each file those attributes also written one by one thats why the data transfer speed is unbelivably slow but it is normal, just let it finish its work. You can test that fact by using 1 single file from the same source HDD to same destination. You may use a iso file with a size of a couple of gigabytes. You will see 1 single file will be copied much much faster.
Your number of files are crazy, 5.000.000+ of 150KB files waoow :) Everything just normal dont worry.
0
 
LVL 20

Assisted Solution

by:wolfcamel
wolfcamel earned 100 total points
ID: 38734158
lots of small files also means a lot of directory and FAT updates, this typically means the disk heads are going back and forth with a lot of random read and writes.

The only things you can really do to improve speed..
stop other services on the machine - particularly antivirus - but then give the drive a manual full scan when you are finished.
If the AV is scanning each file as it gets copied that will represent an overhead you can most likely be comfortable in turning off.
0

Featured Post

Too many email signature updates to deal with?

Do you feel like you are taking up all of your time constantly visiting users’ desks to make changes to email signatures? Wish you could manage all signatures from one central location, easily design them and deploy them quickly to users? Well, there is an easy way!

Join & Write a Comment

Suggested Solutions

In this article we have discussed the manual scenarios to recover data from Windows 10 through some backup and recovery tools which are offered by it.
Sometimes drives fill up and we don't know why.  If you don't understand the best way to use the tools available, you may end up being stumped as to why your drive says it's not full when you have no space left!  Here's how you can find out...
This video Micro Tutorial explains how to clone a hard drive using a commercial software product for Windows systems called Casper from Future Systems Solutions (FSS). Cloning makes an exact, complete copy of one hard disk drive (HDD) onto another d…
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now