asked on

Automated FTP download through HTTP proxy (for mirroring McAfee DAT site)

I would like to mirror the mcafee ftp dat site (ftp://ftp.nai.com/pub/antivirus/datfiles/4.x) so that the 2000 server (running netshield 4.5) and xp clients (running vshield 4.5.1 sp1) can update themselves without connecting to the net. We are a school and behind several firewalls.
On the server, the autoupdate program will connect, but fails when it gets the the last file. Also, this version can't mirror the ftp site (therefore, not much use for the clients).
The xp client does have a usefull mirror utiltiy, which works through the firewall. However, only works on XP, and i want it all to come from the server.
The answer should be simple: use a scheduled ftp program (e.g. ws_ftp, scriptftp) to connect to ftp.nai.com and download the right folder then set the virus scan do update from my server. I've tried all sorts of configuration settings to setup the proxy, but with no luck. It must be possible, because the xp version of mcafee's software can do it. I can also access mcafee's site through internet explorer (but that can't be scripted - can it?).
thanks, ian.

p.s. ideally, the solution is free(!)

oBdA

wget will allow you to make an ftp download through a proxy.
Note that you can define the environment variable "wgetrc" to point to an ini file, where you can pre-define some standard settings tha wget should use (keeps the command line a bit less complicated).
For example:

====8<----[wget.ini]----
use_proxy = on
passive_ftp = on
http_proxy = http://your.proxy.server:8080/
ftp_proxy = http://your.proxy.server:8080/
proxy_user=SomeUser
proxy_passwd=SomePassword
====8<----[wget.ini]----

GNU wget
http://www.gnu.org/software/wget/wget.html

Ian_Stewart_UK

ASKER

thanks, i will try this on when i get back to work on monday.

Ian_Stewart_UK

ASKER

I've been trying this at home (where i'm not forced to use a proxy).
Without a proxy it works perfectly, using the following command:
wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror

But I run into problems when I try it with a proxy. I set up a wget.ini file as follows:
=========
use_proxy = on
passive_ftp = on
http_proxy = http://webcache.blueyonder.co.uk:8080/
ftp_proxy = http://webcache.blueyonder.co.uk:8080/
===========

When I run the same command, I get:
======
C:\PROGRA~1\WGET>wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror
--12:40:41-- ftp://ftp.nai.com/virusdefs/4.x/
=> `ftp.nai.com/virusdefs/4.x/index.html'
Resolving webcache.blueyonder.co.uk... 195.188.152.6
Connecting to webcache.blueyonder.co.uk[195.188.152.6]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,516 [text/html]

100%[====================================>] 2,516 --.--K/s

Last-modified header missing -- time-stamps turned off.
12:40:41 (2.40 MB/s) - `ftp.nai.com/virusdefs/4.x/index.html' saved [2516/2516]

FINISHED --12:40:41--
Downloaded: 2,516 bytes in 1 file
=============

And it downloads the directory listing for the folder as index.html. Is there anyway of making it follow those links in that document and download the files? I've also tried:
wget ftp://ftp.nai.com/virusdefs/4.x/ --recursive

thanks,

ASKER CERTIFIED SOLUTION

oBdA

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Ian_Stewart_UK

ASKER

Thankyou very much, excellent solution! Just incase anyone else has the same problem, I also needed to make it download the the incremental files, as listed in delta.ini; so here's the finished script:

setlocal
set TargetPath=C:\Test\

del %targetpath%\*.* /q

if exist "%TargetPath%\update.ini" del "%TargetPath%\update.ini"
:: *** Retrieve update.ini with version information:
wget ftp://ftp.nai.com/virusdefs/4.x/update.ini -O "%TargetPath%\update.ini"
:: *** Find the version to download:
for /f "tokens=2 delims==" %%a in ('type "%TargetPath%\update.ini" ^| find /i "datversion="') do set DatVersion=%%a
:: *** Download the superdat and the zipped dat files:
wget ftp://ftp.nai.com/virusdefs/4.x/sdat%DatVersion%.exe -O "%TargetPath%\sdat%DatVersion%.exe"
wget ftp://ftp.nai.com/virusdefs/4.x/dat-%DatVersion%.zip -O "%TargetPath%\dat-%DatVersion%.zip"

if exist "%TargetPath%\delta.ini" del "%TargetPath%\delta.ini"

:: *** Retrieve delta.ini with list of update files:
wget ftp://ftp.nai.com/virusdefs/4.x/delta.ini -O "%TargetPath%\delta.ini"
::Extract the setion of the delta.ini that contains the list for .upd files and download each one.
FOR /F "tokens=2 delims==" %%G IN ('findstr /E "upd" %TargetPath%\delta.ini') DO wget ftp://ftp.nai.com/virusdefs/4.x/%%G -O "%TargetPath%\%%G"