Automated FTP download through HTTP proxy (for mirroring McAfee DAT site)

I would like to mirror the mcafee ftp dat site (ftp://ftp.nai.com/pub/antivirus/datfiles/4.x) so that the 2000 server (running netshield 4.5) and xp clients (running vshield 4.5.1 sp1) can update themselves without connecting to the net. We are a school and behind several firewalls.
On the server, the autoupdate program will connect, but fails when it gets the the last file. Also, this version can't mirror the ftp site (therefore, not much use for the clients).
The xp client does have a usefull mirror utiltiy, which works through the firewall. However, only works on XP, and i want it all to come from the server.
The answer should be simple: use a scheduled ftp program (e.g. ws_ftp, scriptftp) to connect to ftp.nai.com and download the right folder then set the virus scan do update from my server. I've tried all sorts of configuration settings to setup the proxy, but with no luck. It must be possible, because the xp version of mcafee's software can do it. I can also access mcafee's site through internet explorer (but that can't be scripted - can it?).
thanks, ian.

p.s. ideally, the solution is free(!)
LVL 2
Ian_Stewart_UKAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

oBdACommented:
wget will allow you to make an ftp download through a proxy.
Note that you can define the environment variable "wgetrc" to point to an ini file, where you can pre-define some standard settings tha wget should use (keeps the command line a bit less complicated).
For example:

====8<----[wget.ini]----
use_proxy = on
passive_ftp = on
http_proxy = http://your.proxy.server:8080/
ftp_proxy = http://your.proxy.server:8080/
proxy_user=SomeUser
proxy_passwd=SomePassword
====8<----[wget.ini]----

GNU wget
http://www.gnu.org/software/wget/wget.html
0
Ian_Stewart_UKAuthor Commented:
thanks, i will try this on when i get back to work on monday.
0
Ian_Stewart_UKAuthor Commented:
I've been trying this at home (where i'm not forced to use a proxy).
Without a proxy it works perfectly, using the following command:
wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror

But I run into problems when I try it with a proxy. I set up a wget.ini file as follows:
=========
use_proxy = on
passive_ftp = on
http_proxy = http://webcache.blueyonder.co.uk:8080/
ftp_proxy = http://webcache.blueyonder.co.uk:8080/
===========

When I run the same command, I get:
======
C:\PROGRA~1\WGET>wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror
--12:40:41--  ftp://ftp.nai.com/virusdefs/4.x/
           => `ftp.nai.com/virusdefs/4.x/index.html'
Resolving webcache.blueyonder.co.uk... 195.188.152.6
Connecting to webcache.blueyonder.co.uk[195.188.152.6]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,516 [text/html]

100%[====================================>] 2,516         --.--K/s

Last-modified header missing -- time-stamps turned off.
12:40:41 (2.40 MB/s) - `ftp.nai.com/virusdefs/4.x/index.html' saved [2516/2516]


FINISHED --12:40:41--
Downloaded: 2,516 bytes in 1 file
=============

And it downloads the directory listing for the folder as index.html. Is there anyway of making it follow those links in that document and download the files? I've also tried:
wget ftp://ftp.nai.com/virusdefs/4.x/ --recursive

thanks,
0
oBdACommented:
Since the request is coming from the web proxy, the server delivers the index.html file (like when you access it in a browser).
To work around that, you can retrieve update.ini to check for the current version, then download the superdat and the zipped dat file.
Here's a sample; this will download update.ini, sdatxxxx.exe and dat-xxxx.zip , and put them into the target directory defined at the beginning.

@echo off
setlocal
set TargetPath=D:\Test\Antivirus

if exist "%TargetPath%\update.ini" del "%TargetPath%\update.ini"
:: *** Retrieve update.ini with version information:
wget ftp://ftp.nai.com/virusdefs/4.x/update.ini -O "%TargetPath%\update.ini"
:: *** Find the version to download:
for /f "tokens=2 delims==" %%a in ('type "%TargetPath%\update.ini" ^| find /i "datversion="') do set DatVersion=%%a
:: *** Download the superdat and the zipped dat files:
wget ftp://ftp.nai.com/virusdefs/4.x/sdat%DatVersion%.exe -O "%TargetPath%\sdat%DatVersion%.exe"
wget ftp://ftp.nai.com/virusdefs/4.x/dat-%DatVersion%.zip -O "%TargetPath%\dat-%DatVersion%.zip"
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ian_Stewart_UKAuthor Commented:
Thankyou very much, excellent solution! Just incase anyone else has the same problem, I also needed to make it download the the incremental files, as listed in delta.ini; so here's the finished script:

setlocal
set TargetPath=C:\Test\

del %targetpath%\*.* /q

if exist "%TargetPath%\update.ini" del "%TargetPath%\update.ini"
:: *** Retrieve update.ini with version information:
wget ftp://ftp.nai.com/virusdefs/4.x/update.ini -O "%TargetPath%\update.ini"
:: *** Find the version to download:
for /f "tokens=2 delims==" %%a in ('type "%TargetPath%\update.ini" ^| find /i "datversion="') do set DatVersion=%%a
:: *** Download the superdat and the zipped dat files:
wget ftp://ftp.nai.com/virusdefs/4.x/sdat%DatVersion%.exe -O "%TargetPath%\sdat%DatVersion%.exe"
wget ftp://ftp.nai.com/virusdefs/4.x/dat-%DatVersion%.zip -O "%TargetPath%\dat-%DatVersion%.zip"

if exist "%TargetPath%\delta.ini" del "%TargetPath%\delta.ini"

:: *** Retrieve delta.ini with list of update files:
wget ftp://ftp.nai.com/virusdefs/4.x/delta.ini -O "%TargetPath%\delta.ini"
::Extract the setion of the delta.ini that contains the list for .upd files and download each one.
FOR /F "tokens=2 delims==" %%G IN ('findstr /E "upd" %TargetPath%\delta.ini') DO wget ftp://ftp.nai.com/virusdefs/4.x/%%G -O "%TargetPath%\%%G"
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Networking

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.