Go Premium for a chance to win a PS4. Enter to Win


Assistance With FTP Scripting / Synchronization (Currently Using WS_FTP)

Posted on 2014-11-01
Medium Priority
Last Modified: 2014-11-08
Hey Guys -

I have a remote FTP server which I connect to 1 - 5 times a day , download all files & folders, then delete them.  The contents randomly range between a few kilobytes to up to 30gb in size each and may be located in the root of the folder or nested in 1 or more folders.

What I'm trying to do is find a way to automate this process for me.  The best option I've found so far is WS_FTP's Scripting Utility.  I tried using WS_FTP's Synchronization, but when testing it failed to match (and therefore not mark to download) files I already had locally.

Note: I'm not dedicated to using WS_FTP, it's just the best option I've found so far.   I'll use any application / script that provides the most features I am wanting - even if I have to purchase it.

#1 Scheduling - As mentioned, I manually do the above task as needed.  If I get a script working the way I want and schedule it as a task to run every X hours, what happens if it starts to download a 20gb file at 3:00pm then re-runs per schedule again at 4:00pm?  Would there not be confliction?  Perhaps I can add a clause to only run if the file/script isn't currently active.  If so, how?

#2 Delete Folders Post Transfer - I've almost got the script where I want it.  It logs into the FTP site, downloads all files and folders, runs a command to delete transferred files, then closes.  When testing different scenarios, it worked great - except for one thing:  when it deletes post transfer, it only deletes files - not folders.  It deletes the files from within transferred folders, but leaves them on the server empty.  Perhaps even a way to set variables in script for folders appearing in DIR and a command after transfer to delete them?

#3 Transfer Control - During the day, I frequently connect to this server via RDP from a remote site.  It would be nice to be able to view the progress of the transfer is any type of GUI - even if in a terminal - just something better than the log file if possible.  Any way I could Pause it?  If I stop the script (therefore stopping the download) and started it again a few minutes later, would it resume transfers or overwrite?  

#4 Speed Control - I can't find a way to limit the download speed in WS_FTP not only in scripting, but even when using the full application / GUI.  It's the #1 thing I don't like about the app and have looked everywhere locally and on Goggle.  Is it possible?  My connection isn't great, but am hoping to get QOS on the router level working well to help with this, too.

My Current Script
Here's the script I am currently testing which has the results mentioned above:
; FTP Sync to C:\- Complete -   /   Revised 11/1/14  
LOG E:\Logs\FTPSync.log
USER user
PASS mypassword
CONNECT ftp.server.com
CD /Complete
LCD "C:\- Complete -\Test2"
RGET *.*

Open in new window

Well, there's all of the details.  The system this will run on is a Windows 8.1 Update 1 x64 system.  If I can get this working, it would be a HUGE help!

Question by:BzowK
  • 2
LVL 33

Expert Comment

ID: 40417951
For the scheduling issue, I would use a semaphore.
In this case, I would wrap the ws_ftp script in a batch file or a powershell script.
The wrapper first checks for the existence of a certain local file.
If it exists, the wrapper ends.
If it does not exist, the wrapper creates it, executes ws_ftp,  and upon ending deletes the semaphore file.
LVL 28

Assisted Solution

by:Bill Bach
Bill Bach earned 2000 total points
ID: 40418110
I might suggest at least trying out Beyond Compare from Scooter Software.  I use it for FTP synchronization all the time from a script.  Specific to your issues:
1) If you schedule the task from the Windows Task Schedule, this should ensure that it only runs one at a time.  However, the semaphore idea from shalomc is a great double-check.  However, one problem with using a specific file as a sempahore is that if the task fails to complete (i.e. crashes), then manual intervention is required.  A true OS-level sempahore uses a lock on a system resource, so that if the process dies, the lock dies, too.  Hard to do in a script, but the combination may be good enough for your needs.
2) The BC scripting language supports both the MOVE command, as well as the DELETE command.  I haven't tested the MOVE myself (I use BC to synchronize folders and files only), but it sounds like it would work, and it should handle folders, too.  
3) Even with the batch script, there is a GUI for BC.  So, you can at least stop the job and resume it again.  With the synchronization options, it will have to restart the file it was on, but should otherwise avoid doing any extra work.
4) I see no way to limit the download speed in BC, so that one still won't be covered.  For most FTP clients, they want to pull the data as fast as they can -- which is why SERVER-SIDE speed limits are commonly available.  Most people wouldn't want their own transfer to take longer than it needs to, so I cannot imagine anyone building this into an FTP Client.  However, you might find one somewhere....
LVL 28

Accepted Solution

Bill Bach earned 2000 total points
ID: 40418112
By the way, a sample script in BC would look like this:
# Turn logging on
log normal "L:\Backups\FTPDownload.log"
# Load the FTPDownload session (all connection details embedded within) 
load "FTPDownload"
option confirm:yes-to-all
# Move all files from left to right
move lt->rt

Open in new window


Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you’re involved with your company’s wide area network (WAN), you’ve probably heard about SD-WANs. They’re the “boy wonder” of networking, ostensibly allowing companies to replace expensive MPLS lines with low-cost Internet access. But, are they …
How to fix a SonicWall Gateway Anti-Virus firewall blocking automatic updates to apps like Windows, Adobe, Symantec, etc.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
Suggested Courses

916 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question