We help IT Professionals succeed at work.

Check out our new AWS podcast with Certified Expert, Phil Phillips! Listen to "How to Execute a Seamless AWS Migration" on EE or on your favorite podcast platform. Listen Now

x

Why do large files take longer to copy than small files?

Medium Priority
867 Views
Last Modified: 2012-05-11
We are looking to use Veeam to backup our current VMWare infrastructure and I was researching cheap NAS solutions.  Well, after 2 days I'm starting to realize just how slow these things are.  It seems to have much to do with copying LARGE files as opposed to small files.  I haven't been able to understand why.

Example - one vendor (won't name names) sells an 8TB solution with 4x2TB SATA drives running at 7200RPM.  I found out that when copying large files (50-500GB each), the throuput is a paltry 35-45Mbps which means a 2TB backup could take over 100 hours!

1) whats the technical reason why large files take so much longe rto copy than small files and
2) can anyone recommend a cost effecient network storage solution?  I have a 10 hour backup window.

Thanks!
Comment
Watch Question

Commented:
the nas solution use a sync mode, that become more the copy so slow, if you want to increase the speed, try to recofigurate the nas server, in async mode, if you nas device, permits.
it's a common know problem, that present nas+vmware

some example here.
http://blog.laspina.ca/ubiquitous/running-zfs-over-nfs-as-a-vmware-store

as a solution, try to reconfigure in async mode your nas server, or use another I/O method, directly attach eSATA, USB (maybe v3), iscsi, obviously if you can do it.

i hope this helps
Principal
CERTIFIED EXPERT
Commented:
Unlock this solution with a free trial preview.
(No credit card required)
Get Preview

Commented:
Look for solutions that offer both nfs and iSCSI.  This was, depending on how you provision you LUNs on your NAS, you can offer up both protocols if you want  to your ESX environment.  Otherwise, iSCSI will probably give you the best performace.
Could there also be some issues maybe with the network config?  
Have you done a packet capture on your host to see what MTU the server is using or if there are any errors with protocol negotiation?  
Also, possiblty try a higher MTU than the standard 1500 (9000 if your switch supports it).  That might require some reconfig on your ESX servers though so be sure to test on a dev box first.

You should be getting faster bandwidth with larger files than smaller.  What rates were you getting with small files?

Unlock the solution to this question.
Thanks for using Experts Exchange.

Please provide your email to receive a free trial preview!

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.