Solved

Assistance With FTP Scripting / Synchronization (Currently Using WS_FTP)

Posted on 2014-11-01
3
456 Views
Last Modified: 2014-11-08
Hey Guys -

I have a remote FTP server which I connect to 1 - 5 times a day , download all files & folders, then delete them.  The contents randomly range between a few kilobytes to up to 30gb in size each and may be located in the root of the folder or nested in 1 or more folders.

What I'm trying to do is find a way to automate this process for me.  The best option I've found so far is WS_FTP's Scripting Utility.  I tried using WS_FTP's Synchronization, but when testing it failed to match (and therefore not mark to download) files I already had locally.

Note: I'm not dedicated to using WS_FTP, it's just the best option I've found so far.   I'll use any application / script that provides the most features I am wanting - even if I have to purchase it.

#1 Scheduling - As mentioned, I manually do the above task as needed.  If I get a script working the way I want and schedule it as a task to run every X hours, what happens if it starts to download a 20gb file at 3:00pm then re-runs per schedule again at 4:00pm?  Would there not be confliction?  Perhaps I can add a clause to only run if the file/script isn't currently active.  If so, how?

#2 Delete Folders Post Transfer - I've almost got the script where I want it.  It logs into the FTP site, downloads all files and folders, runs a command to delete transferred files, then closes.  When testing different scenarios, it worked great - except for one thing:  when it deletes post transfer, it only deletes files - not folders.  It deletes the files from within transferred folders, but leaves them on the server empty.  Perhaps even a way to set variables in script for folders appearing in DIR and a command after transfer to delete them?

#3 Transfer Control - During the day, I frequently connect to this server via RDP from a remote site.  It would be nice to be able to view the progress of the transfer is any type of GUI - even if in a terminal - just something better than the log file if possible.  Any way I could Pause it?  If I stop the script (therefore stopping the download) and started it again a few minutes later, would it resume transfers or overwrite?  

#4 Speed Control - I can't find a way to limit the download speed in WS_FTP not only in scripting, but even when using the full application / GUI.  It's the #1 thing I don't like about the app and have looked everywhere locally and on Goggle.  Is it possible?  My connection isn't great, but am hoping to get QOS on the router level working well to help with this, too.

My Current Script
Here's the script I am currently testing which has the results mentioned above:
; FTP Sync to C:\- Complete -   /   Revised 11/1/14  
;
ONERROR GOTO END
LOG E:\Logs\FTPSync.log
USER user
PASS mypassword
CONNECT ftp.server.com
CD /Complete
LCD "C:\- Complete -\Test2"
ONERROR GOTO ENDCLOSE
POSTXFER PXDELETE
RGET *.*
LABEL END_CLOSE
CLOSE
LABEL END

Open in new window


Well, there's all of the details.  The system this will run on is a Windows 8.1 Update 1 x64 system.  If I can get this working, it would be a HUGE help!

Thanks!
0
Comment
Question by:BzowK
  • 2
3 Comments
 
LVL 32

Expert Comment

by:shalomc
Comment Utility
For the scheduling issue, I would use a semaphore.
In this case, I would wrap the ws_ftp script in a batch file or a powershell script.
The wrapper first checks for the existence of a certain local file.
If it exists, the wrapper ends.
If it does not exist, the wrapper creates it, executes ws_ftp,  and upon ending deletes the semaphore file.
0
 
LVL 28

Assisted Solution

by:Bill Bach
Bill Bach earned 500 total points
Comment Utility
I might suggest at least trying out Beyond Compare from Scooter Software.  I use it for FTP synchronization all the time from a script.  Specific to your issues:
1) If you schedule the task from the Windows Task Schedule, this should ensure that it only runs one at a time.  However, the semaphore idea from shalomc is a great double-check.  However, one problem with using a specific file as a sempahore is that if the task fails to complete (i.e. crashes), then manual intervention is required.  A true OS-level sempahore uses a lock on a system resource, so that if the process dies, the lock dies, too.  Hard to do in a script, but the combination may be good enough for your needs.
2) The BC scripting language supports both the MOVE command, as well as the DELETE command.  I haven't tested the MOVE myself (I use BC to synchronize folders and files only), but it sounds like it would work, and it should handle folders, too.  
3) Even with the batch script, there is a GUI for BC.  So, you can at least stop the job and resume it again.  With the synchronization options, it will have to restart the file it was on, but should otherwise avoid doing any extra work.
4) I see no way to limit the download speed in BC, so that one still won't be covered.  For most FTP clients, they want to pull the data as fast as they can -- which is why SERVER-SIDE speed limits are commonly available.  Most people wouldn't want their own transfer to take longer than it needs to, so I cannot imagine anyone building this into an FTP Client.  However, you might find one somewhere....
0
 
LVL 28

Accepted Solution

by:
Bill Bach earned 500 total points
Comment Utility
By the way, a sample script in BC would look like this:
# Turn logging on
log normal "L:\Backups\FTPDownload.log"
# Load the FTPDownload session (all connection details embedded within) 
load "FTPDownload"
option confirm:yes-to-all
# Move all files from left to right
move lt->rt

Open in new window

0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now