?
Solved

Automated FTP download through HTTP proxy (for mirroring McAfee DAT site)

Posted on 2004-11-12
5
Medium Priority
?
4,235 Views
Last Modified: 2012-06-21
I would like to mirror the mcafee ftp dat site (ftp://ftp.nai.com/pub/antivirus/datfiles/4.x) so that the 2000 server (running netshield 4.5) and xp clients (running vshield 4.5.1 sp1) can update themselves without connecting to the net. We are a school and behind several firewalls.
On the server, the autoupdate program will connect, but fails when it gets the the last file. Also, this version can't mirror the ftp site (therefore, not much use for the clients).
The xp client does have a usefull mirror utiltiy, which works through the firewall. However, only works on XP, and i want it all to come from the server.
The answer should be simple: use a scheduled ftp program (e.g. ws_ftp, scriptftp) to connect to ftp.nai.com and download the right folder then set the virus scan do update from my server. I've tried all sorts of configuration settings to setup the proxy, but with no luck. It must be possible, because the xp version of mcafee's software can do it. I can also access mcafee's site through internet explorer (but that can't be scripted - can it?).
thanks, ian.

p.s. ideally, the solution is free(!)
0
Comment
Question by:Ian_Stewart_UK
  • 3
  • 2
5 Comments
 
LVL 85

Expert Comment

by:oBdA
ID: 12571648
wget will allow you to make an ftp download through a proxy.
Note that you can define the environment variable "wgetrc" to point to an ini file, where you can pre-define some standard settings tha wget should use (keeps the command line a bit less complicated).
For example:

====8<----[wget.ini]----
use_proxy = on
passive_ftp = on
http_proxy = http://your.proxy.server:8080/
ftp_proxy = http://your.proxy.server:8080/
proxy_user=SomeUser
proxy_passwd=SomePassword
====8<----[wget.ini]----

GNU wget
http://www.gnu.org/software/wget/wget.html
0
 
LVL 2

Author Comment

by:Ian_Stewart_UK
ID: 12573044
thanks, i will try this on when i get back to work on monday.
0
 
LVL 2

Author Comment

by:Ian_Stewart_UK
ID: 12573642
I've been trying this at home (where i'm not forced to use a proxy).
Without a proxy it works perfectly, using the following command:
wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror

But I run into problems when I try it with a proxy. I set up a wget.ini file as follows:
=========
use_proxy = on
passive_ftp = on
http_proxy = http://webcache.blueyonder.co.uk:8080/
ftp_proxy = http://webcache.blueyonder.co.uk:8080/
===========

When I run the same command, I get:
======
C:\PROGRA~1\WGET>wget ftp://ftp.nai.com/virusdefs/4.x/ --passive-ftp --mirror
--12:40:41--  ftp://ftp.nai.com/virusdefs/4.x/
           => `ftp.nai.com/virusdefs/4.x/index.html'
Resolving webcache.blueyonder.co.uk... 195.188.152.6
Connecting to webcache.blueyonder.co.uk[195.188.152.6]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,516 [text/html]

100%[====================================>] 2,516         --.--K/s

Last-modified header missing -- time-stamps turned off.
12:40:41 (2.40 MB/s) - `ftp.nai.com/virusdefs/4.x/index.html' saved [2516/2516]


FINISHED --12:40:41--
Downloaded: 2,516 bytes in 1 file
=============

And it downloads the directory listing for the folder as index.html. Is there anyway of making it follow those links in that document and download the files? I've also tried:
wget ftp://ftp.nai.com/virusdefs/4.x/ --recursive

thanks,
0
 
LVL 85

Accepted Solution

by:
oBdA earned 2000 total points
ID: 12574176
Since the request is coming from the web proxy, the server delivers the index.html file (like when you access it in a browser).
To work around that, you can retrieve update.ini to check for the current version, then download the superdat and the zipped dat file.
Here's a sample; this will download update.ini, sdatxxxx.exe and dat-xxxx.zip , and put them into the target directory defined at the beginning.

@echo off
setlocal
set TargetPath=D:\Test\Antivirus

if exist "%TargetPath%\update.ini" del "%TargetPath%\update.ini"
:: *** Retrieve update.ini with version information:
wget ftp://ftp.nai.com/virusdefs/4.x/update.ini -O "%TargetPath%\update.ini"
:: *** Find the version to download:
for /f "tokens=2 delims==" %%a in ('type "%TargetPath%\update.ini" ^| find /i "datversion="') do set DatVersion=%%a
:: *** Download the superdat and the zipped dat files:
wget ftp://ftp.nai.com/virusdefs/4.x/sdat%DatVersion%.exe -O "%TargetPath%\sdat%DatVersion%.exe"
wget ftp://ftp.nai.com/virusdefs/4.x/dat-%DatVersion%.zip -O "%TargetPath%\dat-%DatVersion%.zip"
0
 
LVL 2

Author Comment

by:Ian_Stewart_UK
ID: 12587105
Thankyou very much, excellent solution! Just incase anyone else has the same problem, I also needed to make it download the the incremental files, as listed in delta.ini; so here's the finished script:

setlocal
set TargetPath=C:\Test\

del %targetpath%\*.* /q

if exist "%TargetPath%\update.ini" del "%TargetPath%\update.ini"
:: *** Retrieve update.ini with version information:
wget ftp://ftp.nai.com/virusdefs/4.x/update.ini -O "%TargetPath%\update.ini"
:: *** Find the version to download:
for /f "tokens=2 delims==" %%a in ('type "%TargetPath%\update.ini" ^| find /i "datversion="') do set DatVersion=%%a
:: *** Download the superdat and the zipped dat files:
wget ftp://ftp.nai.com/virusdefs/4.x/sdat%DatVersion%.exe -O "%TargetPath%\sdat%DatVersion%.exe"
wget ftp://ftp.nai.com/virusdefs/4.x/dat-%DatVersion%.zip -O "%TargetPath%\dat-%DatVersion%.zip"

if exist "%TargetPath%\delta.ini" del "%TargetPath%\delta.ini"

:: *** Retrieve delta.ini with list of update files:
wget ftp://ftp.nai.com/virusdefs/4.x/delta.ini -O "%TargetPath%\delta.ini"
::Extract the setion of the delta.ini that contains the list for .upd files and download each one.
FOR /F "tokens=2 delims==" %%G IN ('findstr /E "upd" %TargetPath%\delta.ini') DO wget ftp://ftp.nai.com/virusdefs/4.x/%%G -O "%TargetPath%\%%G"
0

Featured Post

Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Downtime reduced, data recovered by utilizing an Experts Exchange Business Account Challenge The United States Marine Corps employs more than 200,000 active-duty Marines with operations in four continents, all requiring complex networking system…
Many of us in IT utilize a combination of roaming profiles and folder redirection to ensure user information carries over from one workstation to another; in my environment, it was to enable virtualization without needing a separate desktop for each…
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question