Assistance With FTP Scripting / Synchronization (Currently Using WS_FTP)

Posted on 2014-11-01
Last Modified: 2014-11-08
Hey Guys -

I have a remote FTP server which I connect to 1 - 5 times a day , download all files & folders, then delete them.  The contents randomly range between a few kilobytes to up to 30gb in size each and may be located in the root of the folder or nested in 1 or more folders.

What I'm trying to do is find a way to automate this process for me.  The best option I've found so far is WS_FTP's Scripting Utility.  I tried using WS_FTP's Synchronization, but when testing it failed to match (and therefore not mark to download) files I already had locally.

Note: I'm not dedicated to using WS_FTP, it's just the best option I've found so far.   I'll use any application / script that provides the most features I am wanting - even if I have to purchase it.

#1 Scheduling - As mentioned, I manually do the above task as needed.  If I get a script working the way I want and schedule it as a task to run every X hours, what happens if it starts to download a 20gb file at 3:00pm then re-runs per schedule again at 4:00pm?  Would there not be confliction?  Perhaps I can add a clause to only run if the file/script isn't currently active.  If so, how?

#2 Delete Folders Post Transfer - I've almost got the script where I want it.  It logs into the FTP site, downloads all files and folders, runs a command to delete transferred files, then closes.  When testing different scenarios, it worked great - except for one thing:  when it deletes post transfer, it only deletes files - not folders.  It deletes the files from within transferred folders, but leaves them on the server empty.  Perhaps even a way to set variables in script for folders appearing in DIR and a command after transfer to delete them?

#3 Transfer Control - During the day, I frequently connect to this server via RDP from a remote site.  It would be nice to be able to view the progress of the transfer is any type of GUI - even if in a terminal - just something better than the log file if possible.  Any way I could Pause it?  If I stop the script (therefore stopping the download) and started it again a few minutes later, would it resume transfers or overwrite?  

#4 Speed Control - I can't find a way to limit the download speed in WS_FTP not only in scripting, but even when using the full application / GUI.  It's the #1 thing I don't like about the app and have looked everywhere locally and on Goggle.  Is it possible?  My connection isn't great, but am hoping to get QOS on the router level working well to help with this, too.

My Current Script
Here's the script I am currently testing which has the results mentioned above:
; FTP Sync to C:\- Complete -   /   Revised 11/1/14  
LOG E:\Logs\FTPSync.log
USER user
PASS mypassword
CD /Complete
LCD "C:\- Complete -\Test2"
RGET *.*

Open in new window

Well, there's all of the details.  The system this will run on is a Windows 8.1 Update 1 x64 system.  If I can get this working, it would be a HUGE help!

Question by:BzowK
  • 2
LVL 33

Expert Comment

ID: 40417951
For the scheduling issue, I would use a semaphore.
In this case, I would wrap the ws_ftp script in a batch file or a powershell script.
The wrapper first checks for the existence of a certain local file.
If it exists, the wrapper ends.
If it does not exist, the wrapper creates it, executes ws_ftp,  and upon ending deletes the semaphore file.
LVL 28

Assisted Solution

by:Bill Bach
Bill Bach earned 500 total points
ID: 40418110
I might suggest at least trying out Beyond Compare from Scooter Software.  I use it for FTP synchronization all the time from a script.  Specific to your issues:
1) If you schedule the task from the Windows Task Schedule, this should ensure that it only runs one at a time.  However, the semaphore idea from shalomc is a great double-check.  However, one problem with using a specific file as a sempahore is that if the task fails to complete (i.e. crashes), then manual intervention is required.  A true OS-level sempahore uses a lock on a system resource, so that if the process dies, the lock dies, too.  Hard to do in a script, but the combination may be good enough for your needs.
2) The BC scripting language supports both the MOVE command, as well as the DELETE command.  I haven't tested the MOVE myself (I use BC to synchronize folders and files only), but it sounds like it would work, and it should handle folders, too.  
3) Even with the batch script, there is a GUI for BC.  So, you can at least stop the job and resume it again.  With the synchronization options, it will have to restart the file it was on, but should otherwise avoid doing any extra work.
4) I see no way to limit the download speed in BC, so that one still won't be covered.  For most FTP clients, they want to pull the data as fast as they can -- which is why SERVER-SIDE speed limits are commonly available.  Most people wouldn't want their own transfer to take longer than it needs to, so I cannot imagine anyone building this into an FTP Client.  However, you might find one somewhere....
LVL 28

Accepted Solution

Bill Bach earned 500 total points
ID: 40418112
By the way, a sample script in BC would look like this:
# Turn logging on
log normal "L:\Backups\FTPDownload.log"
# Load the FTPDownload session (all connection details embedded within) 
load "FTPDownload"
option confirm:yes-to-all
# Move all files from left to right
move lt->rt

Open in new window


Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
When you start your Windows 10 PC and got an "Operating system not found" error or just saw  "Auto repair for startup" or a blinking cursor with black screen. A loop for Auto repair will start but fix nothing.  You will be panic as there are no back…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…
Internet Business Fax to Email Made Easy - With  eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now