Assistance With FTP Scripting / Synchronization (Currently Using WS_FTP)

Hey Guys -

I have a remote FTP server which I connect to 1 - 5 times a day , download all files & folders, then delete them.  The contents randomly range between a few kilobytes to up to 30gb in size each and may be located in the root of the folder or nested in 1 or more folders.

What I'm trying to do is find a way to automate this process for me.  The best option I've found so far is WS_FTP's Scripting Utility.  I tried using WS_FTP's Synchronization, but when testing it failed to match (and therefore not mark to download) files I already had locally.

Note: I'm not dedicated to using WS_FTP, it's just the best option I've found so far.   I'll use any application / script that provides the most features I am wanting - even if I have to purchase it.

#1 Scheduling - As mentioned, I manually do the above task as needed.  If I get a script working the way I want and schedule it as a task to run every X hours, what happens if it starts to download a 20gb file at 3:00pm then re-runs per schedule again at 4:00pm?  Would there not be confliction?  Perhaps I can add a clause to only run if the file/script isn't currently active.  If so, how?

#2 Delete Folders Post Transfer - I've almost got the script where I want it.  It logs into the FTP site, downloads all files and folders, runs a command to delete transferred files, then closes.  When testing different scenarios, it worked great - except for one thing:  when it deletes post transfer, it only deletes files - not folders.  It deletes the files from within transferred folders, but leaves them on the server empty.  Perhaps even a way to set variables in script for folders appearing in DIR and a command after transfer to delete them?

#3 Transfer Control - During the day, I frequently connect to this server via RDP from a remote site.  It would be nice to be able to view the progress of the transfer is any type of GUI - even if in a terminal - just something better than the log file if possible.  Any way I could Pause it?  If I stop the script (therefore stopping the download) and started it again a few minutes later, would it resume transfers or overwrite?  

#4 Speed Control - I can't find a way to limit the download speed in WS_FTP not only in scripting, but even when using the full application / GUI.  It's the #1 thing I don't like about the app and have looked everywhere locally and on Goggle.  Is it possible?  My connection isn't great, but am hoping to get QOS on the router level working well to help with this, too.

My Current Script
Here's the script I am currently testing which has the results mentioned above:
; FTP Sync to C:\- Complete -   /   Revised 11/1/14  
LOG E:\Logs\FTPSync.log
USER user
PASS mypassword
CD /Complete
LCD "C:\- Complete -\Test2"
RGET *.*

Open in new window

Well, there's all of the details.  The system this will run on is a Windows 8.1 Update 1 x64 system.  If I can get this working, it would be a HUGE help!

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Shalom CarmelCTOCommented:
For the scheduling issue, I would use a semaphore.
In this case, I would wrap the ws_ftp script in a batch file or a powershell script.
The wrapper first checks for the existence of a certain local file.
If it exists, the wrapper ends.
If it does not exist, the wrapper creates it, executes ws_ftp,  and upon ending deletes the semaphore file.
Bill BachPresident and Btrieve GuruCommented:
I might suggest at least trying out Beyond Compare from Scooter Software.  I use it for FTP synchronization all the time from a script.  Specific to your issues:
1) If you schedule the task from the Windows Task Schedule, this should ensure that it only runs one at a time.  However, the semaphore idea from shalomc is a great double-check.  However, one problem with using a specific file as a sempahore is that if the task fails to complete (i.e. crashes), then manual intervention is required.  A true OS-level sempahore uses a lock on a system resource, so that if the process dies, the lock dies, too.  Hard to do in a script, but the combination may be good enough for your needs.
2) The BC scripting language supports both the MOVE command, as well as the DELETE command.  I haven't tested the MOVE myself (I use BC to synchronize folders and files only), but it sounds like it would work, and it should handle folders, too.  
3) Even with the batch script, there is a GUI for BC.  So, you can at least stop the job and resume it again.  With the synchronization options, it will have to restart the file it was on, but should otherwise avoid doing any extra work.
4) I see no way to limit the download speed in BC, so that one still won't be covered.  For most FTP clients, they want to pull the data as fast as they can -- which is why SERVER-SIDE speed limits are commonly available.  Most people wouldn't want their own transfer to take longer than it needs to, so I cannot imagine anyone building this into an FTP Client.  However, you might find one somewhere....
Bill BachPresident and Btrieve GuruCommented:
By the way, a sample script in BC would look like this:
# Turn logging on
log normal "L:\Backups\FTPDownload.log"
# Load the FTPDownload session (all connection details embedded within) 
load "FTPDownload"
option confirm:yes-to-all
# Move all files from left to right
move lt->rt

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Scripting Languages

From novice to tech pro — start learning today.