Link to home
Start Free TrialLog in
Avatar of TriGGy79
TriGGy79

asked on

File transfer speed limit

Hi

I am trying to transfer the vmdk file from a VMware host, (130 GB) using SCP transfer protocol.
The highest speed I get is around 12 MB/s

My setup:
Dell PowerEdge 2950 Server, with onboard Gbit NIC's (VMware ESXi)
Dell Optiplex 755 with Intel Gbit NIC (Windows 2k3)
The computers are connected directly by cable. NO Switch

I know it is impossible to use 100% of the bandwith available, but 12 MB/s is roughly about 10% of a Gbit bandwith.

I have, for testing purpose, tried transferring a large file between 2 Windows XP's with the same result. around 12 MB/s.

Is this a limitation to the file-system ? or an issue when copying large files ?
ASKER CERTIFIED SOLUTION
Avatar of zantak
zantak
Flag of Norway image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of TriGGy79
TriGGy79

ASKER

Thanks for the quick reply.
It definately looks like something I should try.
I am a bit confused by where to change the buffer size, and how.
Is it to be changed on the ESXi host, or on my Win2k3 server?
The tool I use for copying the file is pscp.exe
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
It would have helped if I put the url in ...www.veeam.com
For ESXi you will need the beta version of FastSCP which has just started support for ESXi.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Ill Try FastSCP tomorrow, when I get in.
I tried activating FTP in inetd.conf, but it doesn't seem to work. I can't get any connection to the host.
I don't mind unencrypted transfer, as we have pretty strict firewall rules set up on the wan anyway.
SOLUTION
Avatar of Paul Solovyovsky
Paul Solovyovsky
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
FastSCP Beta is closed for downloading.

Does anybody have a guide for enabling FTP on ESXi (if that's possible)
I should point out that you do not need to enable FTP in ESXi for FastSCP to work (once they release ver 3.0), despite what larstr's comments might indicate.  From Veeam's site:
Q: Do I need to switch ESXi into "unsupported" mode and enable additional service console daemons in order to be able to use FastSCP v3?
A: No, FastSCP v3 does not rely on ESXi service console components. Instead, it uses VMware file management API to copy files.
Ok. looks like it is too difficult to get FTP to work on ESXi.
Does any of you have the publiv beta of FastSCP ?
It would be nice to test it, to know if it is worth waiting for.
Ah they (veeam) say that the product will be marked for General Availability during VMWorld Europe which is next week ... in the mean time .. I don't know what you can do .. maybe try a tor site ?
I wasn't refering to FastSCP v3, but to the previous versions. v3 probably uses the same methods as Veeam backup 3.0 that also supports ESXi.
Lars
Just got this info from Veeam: "Actually, FastSCP for ESXi is already here, just download Backup 3.0, stand-alone install coming soon."
FastSCP does not do the trick.
We decided for another solution wich is not to use snapshots from vmware.
Instead we are going to use Symantecs System Recovery, to backup "snapshots" of the server, on to another location.
This solution has all the functionalities we wanted from vmware's snapshots, but it is much more efficient in our setup
If you're going to use BESR keep in mind a few things.  

Do not use the symantec converter, it doesn't work when you push directly to ESX.  If you can't convert a virtual machine using the SV2I index file (sometimes gets corrupt) you can create a VM and use the BESR SRD ISO file to restore as you would be restoring to a physical machine.

I have used BESR a lot in smaller environment and it does the job but gets pricey when dealing with a lot of servers.  The 5 pack license that includes GRO functionaility is price competative but if you start getting a lot of virtual machines you may want to look at Vizioncore's VRanger or ESXPress
I know it is a bit pricy, but in out case, we need it only for 2 database servers, the rest are backed up traditionally with Backup Exec.
I did indeed have some trouble restoring directly to esx, and also found, that using the boot iso, is the easiest way to do the restore.
Now I have a backup time of roughly 1 hour, instead of the 10 hours I had when copying the vmdk file.
+ I have no downtime when performing the snapshot.  (Don't know if that is a bug in the esx og a configuration fault, But when I take vmware snapshots, the server does not respond during the snapshot creation. That causes some serious problems in our production environment, as all users are disconnected from the oracle database)
If you take a snapshot (from the GUI) that also includes a memory dump, the VM will pause during the memory dump saving. If you only snapshot the disk, the VM will not pause.
If you use BESR you should use vmware converter to convert the image, it understands v2i and sv2i files.  The nice thing about use vmware converter is that it keeps all the settings compared to using the "restore anywhere" feature
Original poster had his answers already in the first 3-5 comments.
I have no problem awarding the participants for the efforts.
But the comments added in this question did not work.
You wrote about using FTP instead...  FTP, as it turned out, is not supported on ESXi.
And the FastSCP tool, might have worked for others, but in my sittuation, the transfer speeds where the same.
And I definately can rule out hardware issues, because with BESR I get very high transfer speeds.

So what would you like me to do in this case ?
If I mark your comment as the solution, others experiencing the same problem, might waste time pursueing a solution that is not working (at least for me)
Ok, I'll agree to the fact that it didnt solve your problem. However I do feel that the question(s) asked was answered. But I'll leave this up to you to decide what to actually do with it.
Points divide to participants