jswan
asked on
How can I transfer large files over the internet with resume capabilities?
The end goal is the ability to transfer large files (5-10gb) over the internet between two computers. The ability to pause the transfer would be a nice feature to have but isn't a requirement. The resume feature is a requirement, especially while uploading. I was thinking an FTP server might be the right choice but I'm not able to confirm which software has the ability to pause and resume file transfers. I did test with Windows Server 2008 R2 (IIS 7.5) as the FTP server and WinSCP as the client. While uploading a file from the client to the server I would break the connection and then reconnect it. WinSCP acknowledges the loss connection and restarts the upload when the connection is live again. The problem is; it starts over from the beginning instead of resuming where it left off. To me resume means; pick up where you left off, not start over.
It doesnt have to be FTP, Im open to try anything if it will accomplish what is needed. Please be specific on the server and client software needed.
It doesnt have to be FTP, Im open to try anything if it will accomplish what is needed. Please be specific on the server and client software needed.
ASKER
I'm already using Microsoft's IIS FTP server in my testing, it's the one included with Server 2008 R2. I tried FileZilla before trying WinSCP and it wouldn't resume either, I see an option to resume ACSII files but I'm tranfering binary files. CutePDF did not work either and IPswitch is the only one that will pause, when you hit play it want to overwite the file so I don;t see the point to that.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
CSecurity; I tested ALFTP and resume only works on downloads, on uploads it restarts from the beginning like the others.
Tolomir; have you used Cerberus FTP Server? What FTP client did you use? Did it do what I'm asking for?
The point of me asking the experts is to get real solutions that you have experience with, not just marketing material from websites. I appreciate the fast responses! Keep them coming!
Tolomir; have you used Cerberus FTP Server? What FTP client did you use? Did it do what I'm asking for?
The point of me asking the experts is to get real solutions that you have experience with, not just marketing material from websites. I appreciate the fast responses! Keep them coming!
The best free tool for you is robocopy
Robocopy will resume file transfer and has a default retry period of 30 seconds.
Mottai
ASKER
mottakutty; does Robocoy work over the internet (WAN)? I assume it will need some port open on the firewall??? Can you be more specific on the use of it?
As far as I know, robocopy is for networkshares at max.
In the tread above, I've mentioned they suggest the free filezilla.
In the tread above, I've mentioned they suggest the free filezilla.
ASKER
Tolomir; In the thread it mentions it works for downloads but not uploads. I'm looking for resume on uploads... maybe I'm asking for the impossible. Do you have any experience with Cerberus FTP? I think I'll send their support an email to see what they say.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I assumes the FTP portion of IIS 7.5 fully supports resume on uploads too, can you confirm if it does or does not? I'm willing to setup Cerberus FTP if I can confirm it works... already have Windows Server 2008 R2 with IIS 7.5 up and running.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Robocopy is most suitable for file transfers over the WAN. This is because of its bulit in retry on disconnection feature. If you are able to ping the two computers and access the drives by unc robocopy will do the job.
Cheers
Mottakutty
Ok wan is not Internet.
Also the need for encryption during transfer is not needed, since data will be at no time accessible by unauthorized externals.
Please tell us more about your setup. If you can use cifs/smb there is no need to install an ftp service.
Tolomir
Also the need for encryption during transfer is not needed, since data will be at no time accessible by unauthorized externals.
Please tell us more about your setup. If you can use cifs/smb there is no need to install an ftp service.
Tolomir
ASKER
I don't have UNC access; the transfer will be done over the internet so FTP does seem appropriate.
I sent this to Cerberus tech;
Before I purchase your software I have some technical questions about a feature (Resumes failed file transfers). I want the ability to resume files that have broken while uploading. Example; FTP client is used to transfer a 4GB media file (binary) to a Cerberus FTP server, during the upload the connection is lost, when the connection is restored I want the upload to continue automatically from the place it was when the connection was lost. If the file was 75% uploaded it should only need to upload the other 25%. Is this possible? Will it work as I explained? What version of your product do I need to use? What FTP client do you recommend?
Here is their response;
Yes, Cerberus FTP Server fully supports server resumes and resuming a failed file transfer is possible as you explained it. You can use nearly any popular FTP client (though I would not recommend a web browser-based FTP client). You can use Filezilla FTP client to easily resume failed file transfers. SmartFTP and WS_FTP client are also popular solutions.
Testing Cerberus FTP;
I installed Cerberus FTP and it looks like it will work! I've tested with Filezilla, SmartFTP and WS_FTP. Not sure which client I like best, leaning towards SmartFTP but then again Filezilla is free. I can connect and resume uploads with FTP and FTP/SSL (AUTH SSL), at this point I'm unable to connect using FTPS/Implicit SSL. Not sure why... seems to be a problem with a configuration within Cerberus FTP as I disabled the firewall on the server and made it a DMZ on the router / firewall and it still wouldn't work. Any suggestions on setting up FTPS/Implicit SSL with Cerberus FTP?
I sent this to Cerberus tech;
Before I purchase your software I have some technical questions about a feature (Resumes failed file transfers). I want the ability to resume files that have broken while uploading. Example; FTP client is used to transfer a 4GB media file (binary) to a Cerberus FTP server, during the upload the connection is lost, when the connection is restored I want the upload to continue automatically from the place it was when the connection was lost. If the file was 75% uploaded it should only need to upload the other 25%. Is this possible? Will it work as I explained? What version of your product do I need to use? What FTP client do you recommend?
Here is their response;
Yes, Cerberus FTP Server fully supports server resumes and resuming a failed file transfer is possible as you explained it. You can use nearly any popular FTP client (though I would not recommend a web browser-based FTP client). You can use Filezilla FTP client to easily resume failed file transfers. SmartFTP and WS_FTP client are also popular solutions.
Testing Cerberus FTP;
I installed Cerberus FTP and it looks like it will work! I've tested with Filezilla, SmartFTP and WS_FTP. Not sure which client I like best, leaning towards SmartFTP but then again Filezilla is free. I can connect and resume uploads with FTP and FTP/SSL (AUTH SSL), at this point I'm unable to connect using FTPS/Implicit SSL. Not sure why... seems to be a problem with a configuration within Cerberus FTP as I disabled the firewall on the server and made it a DMZ on the router / firewall and it still wouldn't work. Any suggestions on setting up FTPS/Implicit SSL with Cerberus FTP?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
This what I've settled on;
Cerberus FTP server
Filezilla for FTP client
FTP over Explicit TSL/SSL
Firewall Settings;
Forward Ports (FTP 21), (FTPS 989 & 990) and (FTP Passive 11000 - 13000) all are TCP
I can pause and resume uploads by toggling the process queue (CTRL-P) in Filezilla and if the connection is lost between the client and server, uploads will resume automatically once it's restored.
More from Cerberus;
We've long since deprecated and removed Implicit SSL from Cerberus FTP Server. Implicit SSL was a stopgap measure that has long since been discouraged for normal FTP use. We've thought about adding it back in but we haven't seen much interest. Explicit SSL is currently the only method of establishing an SSL connection. We are also in the process of adding SFTP but it will be a few more months before that is ready.
More information;
What is the difference between Implicit SSL and Explicit SSL?
http://www.smartftp.com/support/kb/what-is-the-difference-between-implicit-ssl-and-explicit-ssl-f189.html
I believe that Cerberus only supports explicit SSL and not implicit SSL if that helps any.
http://www.cerberusftp.com/phpBB3/viewtopic.php?f=2&t=1553
Definitions;
FTP = File Tranfer Protocol
SFTP = SSH FTP
FTPS = FTP over Implicit TSL/SSL
FTPES (AUTH SSL) = FTP over Explicit TSL/SSL
Thanks for all your help!
Cerberus FTP server
Filezilla for FTP client
FTP over Explicit TSL/SSL
Firewall Settings;
Forward Ports (FTP 21), (FTPS 989 & 990) and (FTP Passive 11000 - 13000) all are TCP
I can pause and resume uploads by toggling the process queue (CTRL-P) in Filezilla and if the connection is lost between the client and server, uploads will resume automatically once it's restored.
More from Cerberus;
We've long since deprecated and removed Implicit SSL from Cerberus FTP Server. Implicit SSL was a stopgap measure that has long since been discouraged for normal FTP use. We've thought about adding it back in but we haven't seen much interest. Explicit SSL is currently the only method of establishing an SSL connection. We are also in the process of adding SFTP but it will be a few more months before that is ready.
More information;
What is the difference between Implicit SSL and Explicit SSL?
http://www.smartftp.com/support/kb/what-is-the-difference-between-implicit-ssl-and-explicit-ssl-f189.html
I believe that Cerberus only supports explicit SSL and not implicit SSL if that helps any.
http://www.cerberusftp.com/phpBB3/viewtopic.php?f=2&t=1553
Definitions;
FTP = File Tranfer Protocol
SFTP = SSH FTP
FTPS = FTP over Implicit TSL/SSL
FTPES (AUTH SSL) = FTP over Explicit TSL/SSL
Thanks for all your help!
ASKER
Had to do too much research on my own.
ASKER
FYI - Cerberus was very responsive in answering my questions and that was also consider when chossing Cerberus FTP.
I know Cerberus is one of this great old tools, still active though.
Support either by forum or by email is something one cannot appreciate enough.
Great that you found your solution and also thank you for the details you provided from their support, for the users here..
Tolomir
Support either by forum or by email is something one cannot appreciate enough.
Great that you found your solution and also thank you for the details you provided from their support, for the users here..
Tolomir
As client use FileZilla's own client or any other professional client, like CuteFTP, IPSwitch FTP.
Then you'll have pause/resume/continue feature.