Batch File : Sync FTP to local folder

ReneGe
ReneGe used Ask the Experts™
on
Hi there,

I need to make a batch file to do a one way sync from the content of an FTP site, to a local folder.

Something like the following.

Thanks for your help,
Rene

Robocopy ftp://username:password@domainname.com:1111 "%~dp0Source" *.* /E /ZB /W:10 /R:10 /MIR /TEE /TS /NP /V /LOG+:"%~n0.log"

Open in new window

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
hi Renege,
u can find many ftp sync utility out there, syncing FTP through batch would be a tedious task i guess if the files are inside more than one directory on ftp server.

On the ftp server, the files are just in root folder or in different sub dirs ?
Most Valuable Expert 2018
Distinguished Expert 2018

Commented:
You won't get very far with robocopy, because it doesn't know anything about ftp.
Try wget, I've used it some years ago for a similar purpose.
GNU Wget
http://www.gnu.org/software/wget/
Obda, I guess he wants to use  Batch only..as usual may be another experiment  by ReneGe
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
@subhashchy:
-You are almost right about the experiment part. However, I need this to be done quickly.
-Yes, it does have subfolders, so I guess It may get messy...

@oBdA:
-I used robocopy in my example so you get the idea.
-wget seems to be a good solution for this.

Would either on of you mind providing me with the script by using wget?  Onless you have a better idea.

Thanks again,
Rene
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
I think your best shot if you need a pure BAT solution would be to FTP down the entire contents of the remote FTP folder with MGET to a work folder locally, and then use robocopy from that work folder to the folder to be synced.

But there really are better tools out there for this task , if you want to go that way let me know and I'll pass along a couple of suggestions.

~bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
If you want a tool to get a solution quickly how about the free version here:

http://www.cyberkiko.com/page/FTPSync.aspx

~bp
Hi Bill, I was trying out a script to pull the directory list of the FTP server in a text file and then compare the files with local directory list, and mget only missing files on local.  I am stuck how to get a complete list of files like dir /s /b from inside FTP.

not sure if i can make this.  May be you guys continue and i will just watch.
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Wouldn't that just the the LS command?  But I think I've seen the format of the output can be hard to parse afterwards.

~bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Actually sorry, it's the DIR subcommand you want, not LS.

~bp
yeah. i tried LS as well but didn't sound good to me , gonna be at least 20 trip to ftp and back to local cmd line if try to make it using LS.

there are only two sub commands for dir inside FTP, -a and -l , both not gonna help :(

Kent DyerIT Security Analyst Senior
Commented:
Best one that I have used is WinSCP..  And you can use it as a stand-alone too (you don't need to install, portable version is available)..  You can use it FTP (port 21) or even SFTP (port 22) too..

http://winscp.net

Here is an example:

Here is the main script (VBScript)..
Option Explicit

Dim oFSO, WshNetwork, objWS, ScriptDir, ServerName, DataCenter, ORG, Customer, LogLocation, DateVar
Dim Today, Yesterday, yMonth, yDay, yYear, yDateVar, AuditLogPart, ForensicLogPart, BillingLogPart, BackUpPath, ZipFile, zipProgram
Dim objFolder, colFiles, objFile, FLName, strCommand, iResult, LoggerFile, ResultFile, fileOutput, sCommand
Dim fileOutput2, strServer, intPort, strFrom, strTo, objMail

Set oFSO = CreateObject("Scripting.FileSystemObject")
Set WshNetwork = CreateObject("Wscript.Network")
Set objWS = CreateObject("WScript.Shell")
ScriptDir = Replace(WScript.ScriptFullName, WScript.ScriptName, "")
ServerName = WshNetwork.ComputerName
DataCenter = "datacenter"
ORG = "somenanme"
Customer = customer"
LogLocation = "C:\logs\"
DateVar = Year(Date) & Right(100 + Month(Date), 2) & Right(100 + Day(Date), 2)
Today = Replace(Date(), "/", "")
Yesterday = Replace(Date() - 1, "/", "")
yMonth = Right(100 + Month(Date() - 1), 2)
yDay = Right(100 + Day(Date() - 1), 2)
yYear = Year(Date() - 1)
yDateVar = yYear & yMonth & yDay

'yMonth = Right(100 + Month(Date()), 2)
'yDay = Right(100 + Day(Date()), 2)
'yYear = Year(Date())

Log = "server" & yYear & "-" & yMonth & "-" & yDay & ".log"
BackUpPath = "C:\Backup\"
ZipFile = BackUpPath & DateVar & "_" & Customer & "_" & ServerName & "@" & DataCenter & ".zip"
zipProgram = "C:\Progra~1\WinZip\WZZIP.EXE"


Set objFolder = oFSO.GetFolder(LogLocation)
Set colFiles = objFolder.Files
For Each objFile In colFiles
	FLName = objFile.Name
	If InStr(Log, FLName) Then
		strCommand = zipProgram & " " & Chr(34) & ZipFile & Chr(34) & " " & Chr(34) & LogLocation & FLName & Chr(34)
		iResult = objWS.Run(strCommand, 0, "true")
	End If
Next

If oFSO.FileExists(ZipFile) Then
	LoggerFile = ScriptDir & "logs\" & Today & "result_file.txt"
	Set fileOutput = oFSO.CreateTextFile(LoggerFile, True)
	fileOutput.WriteLine iResult
	ResultFile = ScriptDir & "WinSCP\send_file.txt"
	Set fileOutput2 = oFSO.CreateTextFile(ResultFile, True)
			
	fileOutput2.WriteLine "option batch abort"
	fileOutput2.WriteLine "option confirm off"
	fileOutput2.WriteLine "open somelogin@1.1.1.1"
	fileOutput2.WriteLine "send " & BackupPath & DateVar & "_" & Customer & "_" & ServerName & "@" & DataCenter & ".zip /home/incoming/"
	fileOutput2.WriteLine "exit"
	fileOutput2.close
	fileOutput.WriteLine "send_file.txt creation is complete"
			
	sCommand = "WinSCP\send_file.bat"
	iResult = objWS.Run("CMD /C " & sCommand,, "true")
	fileOutput.WriteLine iResult
	fileOutput.close
			
	strServer = "2.2.2.2"
	intPort = 25
	strFrom = "you@company.com"
	strTo = "them@company.com"
	'The email bit...
	On Error Goto 0
	Set objMail = CreateObject("CDO.Message") 
	With objMail
		.Subject = "Send file to Report from some server"
		.From = strFrom
		.To = strTo 
		.TextBody = LoggerFile
		'.AddAttachment = LoggerFile
		.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 '2 = use a specified SMTP server, 1 = use local SMTP service
		.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strServer 'Your SMTP Server
		.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = intPort 'Server port
		.Configuration.Fields.Update
		.Send
	End With
		 
	Set objMail = Nothing
		
End If

Open in new window


The file at WinSCP (send_file.txt)..

option batch abort
option confirm off
open somelogin@1.1.1.1
send C:\Backup\20110901_customer_servername@datacenter.zip /home/incoming/
exit

Open in new window


The scheduled task is:

"wscript.exe D:\Scripts\ftp_send.vbs"

HTH,

Kent

Author

Commented:
@kdyer:
Thanks for your hard work, but I am lost in VBScript, so I can't use it.

@subhashchy:
Thanks for scripting this for me.

@billprew:
Thanks for the app, I had it a quick try and did not work. I guess I need more time to figure it out.
If you know it, would you mind sharing me an example of a ini file with the required command line so I can sync files from the FTP site to a local folder?

Cheers,
Rene
Commented:
Here is a Robo-FTP script that will do it:
WORKINGDIR "c:\destination\folder"
IFERROR GOTO done
FTPLOGON "ftp.mydomain.com" /user="UserID" /pw="secret!"
IFERROR GOTO done
RCVFILE "*" /subdirs /emptydirs
:done
FTPLOGOFF
EXIT

Open in new window

Of course you'll need to change the name of the local destination folder on the first line as well as the address, username and password used to connect to the FTP site on the third line.  http://www.robo-ftp.com/download/

Most Valuable Expert 2018
Distinguished Expert 2018

Commented:
A simple
wget.exe -m ftp://username:password@domainname.com:1111

Open in new window

should do the trick.
User name and password could be passed as --ftp-user and --ftp-password as well.

Author

Commented:
@AlexPace:
Thanks for your script. However, RoboFTP is not free, but rather very expensive,  so I can't use it

@oBdA:
I don't know how I can do the required sync with your command line. Sorry for that ;-(

Cheers,
Rene
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
==> Thanks for the app, I had it a quick try and did not work. I guess I need more time to figure it out.
==> If you know it, would you mind sharing me an example of a ini file with the required command line ==> so I can sync files from the FTP site to a local folder?

What files do you want to sync from the ftp host?  Is it all the files in the default folder when you connect via FTP, or is it a particular named subfolder from there? Do you want all files or just certain ones matching a patern?

~bp

Author

Commented:
@billprew:
All files, folders, sub-folders...

Thanks bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Okay, here's the batch commands:

@echo off
ftpsync "EE27287357" /FTPSYNCDATA:"C:\EE\TEMP" /QUIET
if errorlevel 1 echo Errors in FTP sync

Open in new window

and here's the INI file (C:\EE\TEMP\EE27287357.ini):

[Source]
Server=ftp.server.com
User=username
Pass=password
Dir=/
Type=U
Case=Sensitive
IncludeSubDirs=Yes

[Destination]
Type=F
Dir=C:\EE\TEMP\Files\
Case=Sensitive

Open in new window

~bp

Commented:
Another program that will do want you want and is much more reasonably priced is Beyond Compare.

http://www.scootersoftware.com/moreinfo.php?zz=pricing

It can run in batch mode and is also very handy for a lot of other things as well.
Commented:
A sample from the Beyond Compare help file shows how it would be done using Beyond Compare ...


Synching two folders

Here is a more advanced script which syncs two specific folders using advanced filters.  This works the same as loading the folders in the Folder Viewer and performing a "Mirror Left to Right" folder synchronization.

  # Turn logging on
  log normal "C:\My Documents\Webpage\Synclog.txt"
  # Load the default session and exclude all but certain file types
  load <default>
  filter "*.html;*.htm;*.png;*.jpg"
  # Load the base folders
  load "C:\My Documents\Webpage" "ftp://user@ftp.some-isp.com/webroot"
  # Copy different files left to right, delete orphans on right
  sync mirror:left->right

Author

Commented:
@vk3kjc:
Thanks for your efforts and code. Considering that I am looking for a free solution and this software is way to greeddy... So I can't use it.

@bp:
Trying it now!

Thanks and cheers
Most Valuable Expert 2018
Distinguished Expert 2018
Commented:
"I don't know how I can do the required sync with your command line." - it would help tremendously if you could specify what exactly it is that is unclear. The "-m" should tell wget to mirror the ftp site, and it should then simply download the page given into a subfolder "domain.com" in the folder from which you started the command (at least it did last time I used it like that).
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
FWIW, I was able to do something useful with WGET as well, following the example from oBdA earlier.

~bp

Author

Commented:
@bp

I did not work for me (Errors in FTP sync).

See below:

Thanks

@echo off

REM CREATING INI FILE
	SET IniFile=%~dpn0.ini
	IF EXIST "%IniFile%" DEL "%IniFile%"
	
	SET ini.1=[Source]
	SET ini.2=Server=ftpsite.com:2222
	SET ini.3=User=username
	SET ini.4=Pass=password
	SET ini.5=Dir=/eetest
	SET ini.6=Type=U
	SET ini.7=Case=Sensitive
	SET ini.8=IncludeSubDirs=Yes
	SET ini.9=[Destination]
	SET ini.a=Type=F
	SET ini.b=Dir=C:\temp\ee
	SET ini.c=Case=Sensitive

	FOR /F "tokens=1* delims==" %%A in ('SET ini.') DO ECHO %%B>>"%IniFile%"

	REM CALL :RealIniValues


ftpsync "%IniFile%" /FTPSYNCDATA:"C:\temp\ee" /QUIET
if errorlevel 1 echo Errors in FTP sync
pause

Open in new window

Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
On the ftpsync command line, the INI file name parm does not allow the INI extension, it is assumed.  Notice my example line of:

ftpsync "EE27287357" /FTPSYNCDATA:"C:\EE\TEMP" /QUIET

which read C:\EE\TEMP\EE27287357.ini

~bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Maybe something like this:

@echo off
REM CREATING INI FILE
SET IniName=%~dpn0
SET IniFile=%IniName%.ini
(
ECHO [Source]
ECHO Server=ftpsite.com:2222
ECHO User=username
ECHO Pass=password
ECHO Dir=/eetest
ECHO Type=U
ECHO Case=Sensitive
ECHO IncludeSubDirs=Yes
ECHO [Destination]
ECHO Type=F
ECHO Dir=C:\temp\ee
ECHO Case=Sensitive
)>"%IniFile%"
ftpsync "%IniName%" /FTPSYNCDATA:"C:\temp\ee" /QUIET
if errorlevel 1 echo Errors in FTP sync
pause

Open in new window

~bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Hold on, that's not going to work either, since you have the path in the ININAME, just a sec...

~bp
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
This should be closer.  The INI file needs to be in the folder specified by /FTPSYNCDATA.

@echo off
REM CREATING INI FILE
SET IniName=%~n0
SET IniFile=C:\temp\ee\%~n0.ini
(
ECHO [Source]
ECHO Server=ftpsite.com:2222
ECHO User=username
ECHO Pass=password
ECHO Dir=/eetest
ECHO Type=U
ECHO Case=Sensitive
ECHO IncludeSubDirs=Yes
ECHO [Destination]
ECHO Type=F
ECHO Dir=C:\temp\ee
ECHO Case=Sensitive
)>"%IniFile%"
ftpsync "%IniName%" /FTPSYNCDATA:"C:\temp\ee" /QUIET
if errorlevel 1 echo Errors in FTP sync
pause

Open in new window

~bp

Author

Commented:
@bp:

Since I am not using the default FTP port 21, port parameter may not be entered correctly.

http://www.cyberkiko.com/docs/ftpsync28/pPort.htm?MenuState=X3BAAAAAAEEAAAAAQAFUVAAAAUA

However, they do not give instructions as in the ini file, like for example "port=2222".

Have any new ideas?

Thanks

Author

Commented:
Here is the /DEBUG log file content

I changed the domain name for privacy.

And by the way, by using the same parameters, I can access the FTP site with FTP client Filezilla.

Cheers
CyberKiko FTPSync 2.8f
Copyright © 1996-2011 by Kristof Gajsek, http://www.cyberkiko.com. All Rights Reserved.
This product includes software developed by Francois Piette & TeamICS (http://www.overbyte.be/)

Priority: Normal
Data folder: C:\temp\ee\
Loading MTB file: C:\temp\ee\sync.mtb

Start: 02-sept.-2011 14:47:05
  Syncing C:\Temp\ee\rene\ with ftpsite.com:2222/
  
  Detecting FTP mode for ftpsite.com:2222, please wait (this may take up to 10 minutes)...
  Once FTPSync determines FTP mode it will save it for all subsequent transfers.
  WARNING: FTPSync can not determine FTP mode. Passive mode assumed. Please set manually!
  Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /

Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /

Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /

Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /

Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /

Too many connection failures. Last error: 500 DNS lookup error - No Data (#11004) (CONNECT) on /


  
  Source FTP unreachable: 500 DNS lookup error - No Data (#11004)
  Source FTP unreachable: 500 DNS lookup error - No Data (#11004)
  Source FTP unreachable: 500 DNS lookup error - No Data (#11004)
  Source FTP unreachable: 500 DNS lookup error - No Data (#11004)

Open in new window

Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Looks to me like you would replace:

ECHO Server=ftpsite.com:2222

with:

ECHO Server=ftpsite.com
ECHO Port=2222

~bp

Author

Commented:
@bp:

It works... youhoooo...

1 last detail (please):

The auto detect, detects "passive mode".
FTP mode: Passive
Debug: > PASV
Debug: < 227 Entering Passive Mode


So I added "Passive=Yes". But still starts auto-detecting again and takes an eternity.

Have any ideas?
Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
Passive=Yes in the [source] section seems like it would be the right approach. When I did a little testing here it didn't take long at all to autodetect, how long is this "eternity" you mention ;-).

~bp

Author

Commented:
2 minutes 5 seconds
Test your restores, not your backups...
Top Expert 2016
Commented:
After I ran it once with no Passive= setting, it took make 10 to 15 seconds to autodetect.  But then it adds the Passive=No in my case since I'm active, and following executions skip that delay.

Which brings me to an interesting point. This utility is really designed not to start with a clean INI file on each execution.  It actually updates the INI when it runs and adds some useful stuff in there for future runs.  Including encrypting the password!  So I'd suggest you not build the INI in the BAT, but precreate it and reference it.

~bp

Author

Commented:
Thanks BP, you nailed it !!

Would you mind sharing what you found about WGET that would sync FTP site to local folder?

Here is my final E.E. working version.

Thanks a million and cheers!


@echo off

SET Retry=10
SET WorkingFolder=%~dp0WorkingFolder
SET IniFile=%WorkingFolder%\%~n0.ini
SET IniName=%~n0
SET IniTempFile=%~dpn0.ini
SET FTPLocalSyncFolder=C:\FTPLocalSyncFolder

IF NOT EXIST "%WorkingFolder%" MD "%WorkingFolder%"
IF NOT EXIST "%FTPLocalSyncFolder%" MD "%FTPLocalSyncFolder%"

REM IF NOT EXIST, CREATING INI FILE
	IF NOT EXIST "%IniFile%" (
		ECHO FIRST TIME RUNNING. MAY TAKE FEW MINUTES TO START
		SET /P User=ENTER FTP USER NAME: 
		SET /P Pass=ENTER FTP PASSWORD : 
	)
	
	IF NOT EXIST "%IniFile%" (
		ECHO [Source]
		ECHO Server=ftpserver.com
		ECHO Port=22
		ECHO User=%User%
		ECHO Pass=%Pass%
		ECHO Dir=/
		ECHO Type=U
		ECHO Case=Sensitive
		ECHO IncludeSubDirs=Yes
		ECHO Passive=Yes

		ECHO [Destination]
		ECHO Type=F
		ECHO Dir=%FTPLocalSyncFolder%
		ECHO Case=Sensitive
	)>"%IniFile%"

CLS
ECHO STARTING FTP SYNC

:Repeat
IF %Retry% EQU 0 GOTO End
SET /a Retry-=1
ftpsync "%IniName%" /DEBUG /FTPSYNCDATA:"%WorkingFolder%" /QUIET
if errorlevel 4 ECHO ERROR NUMBER [%errorlevel%] RETRY NUMBER [%Retry% OF 10] & goto Inierror
if errorlevel 3 ECHO ERROR NUMBER [%errorlevel%] RETRY NUMBER [%Retry% OF 10] & goto Repeat
if errorlevel 2 ECHO ERROR NUMBER [%errorlevel%] RETRY NUMBER [%Retry% OF 10] & goto Repeat
if errorlevel 1 ECHO ERROR NUMBER [%errorlevel%] RETRY NUMBER [%Retry% OF 10] & goto Repeat
GOTO end

:Inierror
ECHO "ERROR: Most likely there's a problem in ["%IniFile%"] file!"

:End
EXIT

Open in new window

Bill PrewTest your restores, not your backups...
Top Expert 2016

Commented:
I just CD to the folder I want the files in, and then did as mentioned above:

wget.exe -m ftp://username:password@domainname.com:1111

~bp

Author

Commented:
@ bp and oBdA

With:
wget.exe -m --passive -P LocalFolder ftp://username:password@domainname.com:1111

I was able to mirror from the FTP site. However, when I delete a file at the FTP site, it is not deleted at the local folder.  This is why I was unable to use WGET. Unless I did something wrong?

Have any ideas about this?

Thanks

Author

Commented:
You have done ennuf on this thread. I will close it and open a new one for WGET.

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/MS_DOS/Q_27289892.html

I will split points according to contribution and effort.

Thanks and cheers,
Rene

Author

Commented:
@oBdA

Sorry for not responding to your message. I just didn'nt see it until bp quoted your solution:
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/MS_DOS/Q_27287357.html?cid=1575#a36473643

Going to bed now!  Zzzzzz

Thanks and cheers,
Rene

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial