Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 7160
  • Last Modified:

File transfer speed limit

Hi

I am trying to transfer the vmdk file from a VMware host, (130 GB) using SCP transfer protocol.
The highest speed I get is around 12 MB/s

My setup:
Dell PowerEdge 2950 Server, with onboard Gbit NIC's (VMware ESXi)
Dell Optiplex 755 with Intel Gbit NIC (Windows 2k3)
The computers are connected directly by cable. NO Switch

I know it is impossible to use 100% of the bandwith available, but 12 MB/s is roughly about 10% of a Gbit bandwith.

I have, for testing purpose, tried transferring a large file between 2 Windows XP's with the same result. around 12 MB/s.

Is this a limitation to the file-system ? or an issue when copying large files ?
0
TriGGy79
Asked:
TriGGy79
  • 8
  • 4
  • 3
  • +3
5 Solutions
 
zantakCommented:
I'd say its more the copying method used, then anything else. This site has more documentation on it, then what I can provide. Tho I reckon switching to a more unsecure method, ie. FTP would give you a better speed on this then using SCP.

http://www.psc.edu/networking/projects/hpn-ssh/
0
 
TriGGy79Author Commented:
Thanks for the quick reply.
It definately looks like something I should try.
I am a bit confused by where to change the buffer size, and how.
Is it to be changed on the ESXi host, or on my Win2k3 server?
The tool I use for copying the file is pscp.exe
0
 
za_mkhCommented:
Try using veeam FASTSCP ... the fastest SCP copy tool in town. It's free too ...
 
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
za_mkhCommented:
It would have helped if I put the url in ...www.veeam.com
For ESXi you will need the beta version of FastSCP which has just started support for ESXi.
0
 
jfields71Commented:
I agree that SCP is probably the issue.  SCP uses encryption, which is going to add considerable overhead and slow down the transfer.  FTP would definitely be faster, provided you don't mind the traffic being sent unencrypted.  

Also, if you want to keep using SCP, then FastSCP is recommended: http://www.veeam.com/esx-fastscp.html.  
0
 
TriGGy79Author Commented:
Ill Try FastSCP tomorrow, when I get in.
I tried activating FTP in inetd.conf, but it doesn't seem to work. I can't get any connection to the host.
I don't mind unencrypted transfer, as we have pretty strict firewall rules set up on the wan anyway.
0
 
Paul SolovyovskyCommented:
FastSCP is about 4-5 times faster than regular SCP
0
 
larstrCommented:
FastSCP is not using SCP. It's using FTP, so no wonder why it's faster.
0
 
TriGGy79Author Commented:
FastSCP Beta is closed for downloading.

Does anybody have a guide for enabling FTP on ESXi (if that's possible)
0
 
jfields71Commented:
See this thread about FTP on ESXi:
http://communities.vmware.com/message/1092088#1092088
0
 
jfields71Commented:
I should point out that you do not need to enable FTP in ESXi for FastSCP to work (once they release ver 3.0), despite what larstr's comments might indicate.  From Veeam's site:
Q: Do I need to switch ESXi into "unsupported" mode and enable additional service console daemons in order to be able to use FastSCP v3?
A: No, FastSCP v3 does not rely on ESXi service console components. Instead, it uses VMware file management API to copy files.
0
 
TriGGy79Author Commented:
Ok. looks like it is too difficult to get FTP to work on ESXi.
Does any of you have the publiv beta of FastSCP ?
It would be nice to test it, to know if it is worth waiting for.
0
 
za_mkhCommented:
Ah they (veeam) say that the product will be marked for General Availability during VMWorld Europe which is next week ... in the mean time .. I don't know what you can do .. maybe try a tor site ?
0
 
larstrCommented:
I wasn't refering to FastSCP v3, but to the previous versions. v3 probably uses the same methods as Veeam backup 3.0 that also supports ESXi.
Lars
0
 
larstrCommented:
Just got this info from Veeam: "Actually, FastSCP for ESXi is already here, just download Backup 3.0, stand-alone install coming soon."
0
 
TriGGy79Author Commented:
FastSCP does not do the trick.
We decided for another solution wich is not to use snapshots from vmware.
Instead we are going to use Symantecs System Recovery, to backup "snapshots" of the server, on to another location.
This solution has all the functionalities we wanted from vmware's snapshots, but it is much more efficient in our setup
0
 
Paul SolovyovskyCommented:
If you're going to use BESR keep in mind a few things.  

Do not use the symantec converter, it doesn't work when you push directly to ESX.  If you can't convert a virtual machine using the SV2I index file (sometimes gets corrupt) you can create a VM and use the BESR SRD ISO file to restore as you would be restoring to a physical machine.

I have used BESR a lot in smaller environment and it does the job but gets pricey when dealing with a lot of servers.  The 5 pack license that includes GRO functionaility is price competative but if you start getting a lot of virtual machines you may want to look at Vizioncore's VRanger or ESXPress
0
 
TriGGy79Author Commented:
I know it is a bit pricy, but in out case, we need it only for 2 database servers, the rest are backed up traditionally with Backup Exec.
I did indeed have some trouble restoring directly to esx, and also found, that using the boot iso, is the easiest way to do the restore.
Now I have a backup time of roughly 1 hour, instead of the 10 hours I had when copying the vmdk file.
+ I have no downtime when performing the snapshot.  (Don't know if that is a bug in the esx og a configuration fault, But when I take vmware snapshots, the server does not respond during the snapshot creation. That causes some serious problems in our production environment, as all users are disconnected from the oracle database)
0
 
larstrCommented:
If you take a snapshot (from the GUI) that also includes a memory dump, the VM will pause during the memory dump saving. If you only snapshot the disk, the VM will not pause.
0
 
Paul SolovyovskyCommented:
If you use BESR you should use vmware converter to convert the image, it understands v2i and sv2i files.  The nice thing about use vmware converter is that it keeps all the settings compared to using the "restore anywhere" feature
0
 
zantakCommented:
Original poster had his answers already in the first 3-5 comments.
0
 
TriGGy79Author Commented:
I have no problem awarding the participants for the efforts.
But the comments added in this question did not work.
You wrote about using FTP instead...  FTP, as it turned out, is not supported on ESXi.
And the FastSCP tool, might have worked for others, but in my sittuation, the transfer speeds where the same.
And I definately can rule out hardware issues, because with BESR I get very high transfer speeds.

So what would you like me to do in this case ?
If I mark your comment as the solution, others experiencing the same problem, might waste time pursueing a solution that is not working (at least for me)
0
 
zantakCommented:
Ok, I'll agree to the fact that it didnt solve your problem. However I do feel that the question(s) asked was answered. But I'll leave this up to you to decide what to actually do with it.
0
 
TriGGy79Author Commented:
Points divide to participants
0

Featured Post

Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

  • 8
  • 4
  • 3
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now