Solved

Wget dowload web site log files and delete file

Posted on 2006-11-10
15
342 Views
Last Modified: 2008-02-01
Hi

The log files are delivered to a folder on web server /purpleski/www.purpleski.com/logs.  Can the following be done with wget and if so can anybody give me the necessary instructions to do it!?

When run the wget command fetches any files in the directory /purpleski/www.purpleski.com/logs downloads these to my local computer to E:\Websites\purpleski\logs and then deletes any files in /purpleski/www.purpleski.com/logs.

This means that each day when it is run there will only be one file to download.

If the above won't work does anybody know of another solution to achieve the above?

Thanks.

purpleperson
0
Comment
Question by:purpleski
  • 8
  • 7
15 Comments
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914349
I don't think wget would be able to delete the log on the server.  wget simply retrives files via an httpclient session.  Does this website have ftp access?  If so, this could be done with ftp fairly easily.
0
 

Author Comment

by:purpleski
ID: 17914785
Yes I think it has ftp access. How could I tell?

Thanks
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914810
Attempt to ftp to the site:
From a dos prompt:
c:\ftp myhostname.com
0
 

Author Comment

by:purpleski
ID: 17915261
I get

Connected to purpleski.com.
220 pop3.gradwell.net NcFTPd Se
User (purpleski.com:(none)):

so yes I think!
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17915528
Assuming you know your username and password, you could create a batch job that would ftp for you.  I recommend you test it on some uselss files first.
Copy and paste EVERYTHING below this line into a file called get.bat:
@echo off
>script.ftp echo open ftp.somesite.com
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916143
Thanks

tried the following (obviously changed password for this post!)

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.
ftp> purpleski
Invalid command.
ftp> PASSWORD
Invalid command.
ftp> prompt
Interactive mode Off .
ftp> mget *
Not connected.
ftp> del ex060809.log
Not connected.
ftp> quit

What am I doing wrong
C:\>
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916196
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.

Your host doesn't exist or doesn't have an ftp  client running.  Are you sure that is where you need to ftp to?
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:purpleski
ID: 17916508
When I use CuteFTP to upload/download files I enter

Host Address
hathor.xcalibre.co.uk

Username
purpleski

password
************

any advice?

purpleperson
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916535
ftp.hathor.xcalibre.co.uk isn't the same as hathor.xcalibre.co.uk
@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916941
Getting there. I now get

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

Looks like

>>script.ftp cd/home/11550/purpleski/www.purpleski.com/logs

I have been told that  Physical filename on the Windows Server is

E:\home\11550\purpleski\www.purpleski.com


What am I doing wrong any thoughts?

Thanks for all your help on this one
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17917098
For one thing, there should be a space:
cd /home/11550/purpleski/www.purpleski.com/logs
For another, it depends on how the ftp server is set up.  Try log in yourself and type the command:
cd /home/11550/purpleski/www.purpleski.com/logs
and see where it takes you.

0
 

Author Comment

by:purpleski
ID: 17921258
Ok we are nearly there hope you can still help me

I can log on manually at a command prompt and do it all manually

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>ftp
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)): purpleski
331 Password required for purpleski.
Password:
230 User purpleski logged in.
ftp> cd www.purpleski.com/logs
250 CWD command successful.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
200 Type set to A.
200 PORT command successful.
150 Opening ASCII mode data connection for xxxx.log(3766362 bytes).
226 Transfer complete.
ftp: 3766362 bytes received in 16.70Seconds 225.49Kbytes/sec.
ftp> del xxxx.log
250 DELE command successful.
ftp> quit
221

C:\>

but if I run the script you gave me:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget xxxx.log
>>script.ftp echo del xxxx.log
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

The output is


C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

It looks like the cd instruction is being ignored.

Upped the points as this is obviously more difficult than I thought!!

purpleperson

0
 

Author Comment

by:purpleski
ID: 17921553
Hi

Did some googling which led me to change the script to

open hathor.xcalibre.co.uk
USERNAME
PASSWORD
cd www.purpleski.com/logs
prompt
mget *
del *
quit
ftp -s:script.ftp
del script.ftp

then at the command line I type

ftp -s:SCRIPT.TXT

This works.

Any thoughts?
0
 
LVL 18

Accepted Solution

by:
Morcalavin earned 250 total points
ID: 17922680
Ahh.  I see the problem now:
>>script.ftp cd www.purpleski.com/logs

This should be :
>>script.ftp echo cd www.purpleski.com/logs
0
 

Author Comment

by:purpleski
ID: 17928908
Thanks for all that it works perfectly as follows:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp echo cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

Going to use it properley as a scheduled task tonight at 00.01 GMT let us hope it works!

Points are yours
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
"In order to have an organized way for empathy mapping, we rely on a psychological model and trying to model it in a simple way, so we will split the board to three section for each persona and a scenario and try to see what those personas would Do,…
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn how to count occurrences of each item in an array.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now