Solved

Assistance With FTP Scripting / Synchronization (Currently Using WS_FTP)

Posted on 2014-11-01
3
490 Views
Last Modified: 2014-11-08
Hey Guys -

I have a remote FTP server which I connect to 1 - 5 times a day , download all files & folders, then delete them.  The contents randomly range between a few kilobytes to up to 30gb in size each and may be located in the root of the folder or nested in 1 or more folders.

What I'm trying to do is find a way to automate this process for me.  The best option I've found so far is WS_FTP's Scripting Utility.  I tried using WS_FTP's Synchronization, but when testing it failed to match (and therefore not mark to download) files I already had locally.

Note: I'm not dedicated to using WS_FTP, it's just the best option I've found so far.   I'll use any application / script that provides the most features I am wanting - even if I have to purchase it.

#1 Scheduling - As mentioned, I manually do the above task as needed.  If I get a script working the way I want and schedule it as a task to run every X hours, what happens if it starts to download a 20gb file at 3:00pm then re-runs per schedule again at 4:00pm?  Would there not be confliction?  Perhaps I can add a clause to only run if the file/script isn't currently active.  If so, how?

#2 Delete Folders Post Transfer - I've almost got the script where I want it.  It logs into the FTP site, downloads all files and folders, runs a command to delete transferred files, then closes.  When testing different scenarios, it worked great - except for one thing:  when it deletes post transfer, it only deletes files - not folders.  It deletes the files from within transferred folders, but leaves them on the server empty.  Perhaps even a way to set variables in script for folders appearing in DIR and a command after transfer to delete them?

#3 Transfer Control - During the day, I frequently connect to this server via RDP from a remote site.  It would be nice to be able to view the progress of the transfer is any type of GUI - even if in a terminal - just something better than the log file if possible.  Any way I could Pause it?  If I stop the script (therefore stopping the download) and started it again a few minutes later, would it resume transfers or overwrite?  

#4 Speed Control - I can't find a way to limit the download speed in WS_FTP not only in scripting, but even when using the full application / GUI.  It's the #1 thing I don't like about the app and have looked everywhere locally and on Goggle.  Is it possible?  My connection isn't great, but am hoping to get QOS on the router level working well to help with this, too.

My Current Script
Here's the script I am currently testing which has the results mentioned above:
; FTP Sync to C:\- Complete -   /   Revised 11/1/14  
;
ONERROR GOTO END
LOG E:\Logs\FTPSync.log
USER user
PASS mypassword
CONNECT ftp.server.com
CD /Complete
LCD "C:\- Complete -\Test2"
ONERROR GOTO ENDCLOSE
POSTXFER PXDELETE
RGET *.*
LABEL END_CLOSE
CLOSE
LABEL END

Open in new window


Well, there's all of the details.  The system this will run on is a Windows 8.1 Update 1 x64 system.  If I can get this working, it would be a HUGE help!

Thanks!
0
Comment
Question by:BzowK
  • 2
3 Comments
 
LVL 33

Expert Comment

by:shalomc
ID: 40417951
For the scheduling issue, I would use a semaphore.
In this case, I would wrap the ws_ftp script in a batch file or a powershell script.
The wrapper first checks for the existence of a certain local file.
If it exists, the wrapper ends.
If it does not exist, the wrapper creates it, executes ws_ftp,  and upon ending deletes the semaphore file.
0
 
LVL 28

Assisted Solution

by:Bill Bach
Bill Bach earned 500 total points
ID: 40418110
I might suggest at least trying out Beyond Compare from Scooter Software.  I use it for FTP synchronization all the time from a script.  Specific to your issues:
1) If you schedule the task from the Windows Task Schedule, this should ensure that it only runs one at a time.  However, the semaphore idea from shalomc is a great double-check.  However, one problem with using a specific file as a sempahore is that if the task fails to complete (i.e. crashes), then manual intervention is required.  A true OS-level sempahore uses a lock on a system resource, so that if the process dies, the lock dies, too.  Hard to do in a script, but the combination may be good enough for your needs.
2) The BC scripting language supports both the MOVE command, as well as the DELETE command.  I haven't tested the MOVE myself (I use BC to synchronize folders and files only), but it sounds like it would work, and it should handle folders, too.  
3) Even with the batch script, there is a GUI for BC.  So, you can at least stop the job and resume it again.  With the synchronization options, it will have to restart the file it was on, but should otherwise avoid doing any extra work.
4) I see no way to limit the download speed in BC, so that one still won't be covered.  For most FTP clients, they want to pull the data as fast as they can -- which is why SERVER-SIDE speed limits are commonly available.  Most people wouldn't want their own transfer to take longer than it needs to, so I cannot imagine anyone building this into an FTP Client.  However, you might find one somewhere....
0
 
LVL 28

Accepted Solution

by:
Bill Bach earned 500 total points
ID: 40418112
By the way, a sample script in BC would look like this:
# Turn logging on
log normal "L:\Backups\FTPDownload.log"
# Load the FTPDownload session (all connection details embedded within) 
load "FTPDownload"
option confirm:yes-to-all
# Move all files from left to right
move lt->rt

Open in new window

0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Layer 2 versus layer 3 10 42
Cisco ASA inside & outside to same switch 3 37
Vlan to Vlan communication 9 71
VIRTUAL NETWORKING 3 23
Today, still in the boom of Apple, PC's and products, nearly 50% of the computer users use Windows as graphical operating systems. If you are among those users who love windows, but are grappling to keep the system's hard drive optimized, then you s…
I had an issue with InstallShield not being able to use Computer Browser service on Windows Server 2012. Here is the solution I found.
The viewer will learn how to successfully download and install the SARDU utility on Windows 8, without downloading adware.
The Task Scheduler is a powerful tool that is built into Windows. It allows you to schedule tasks (actions) on a recurring basis, such as hourly, daily, weekly, monthly, at log on, at startup, on idle, etc. This video Micro Tutorial is a brief intro…

929 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now