Solved

Wget dowload web site log files and delete file

Posted on 2006-11-10
15
350 Views
Last Modified: 2008-02-01
Hi

The log files are delivered to a folder on web server /purpleski/www.purpleski.com/logs.  Can the following be done with wget and if so can anybody give me the necessary instructions to do it!?

When run the wget command fetches any files in the directory /purpleski/www.purpleski.com/logs downloads these to my local computer to E:\Websites\purpleski\logs and then deletes any files in /purpleski/www.purpleski.com/logs.

This means that each day when it is run there will only be one file to download.

If the above won't work does anybody know of another solution to achieve the above?

Thanks.

purpleperson
0
Comment
Question by:purpleski
  • 8
  • 7
15 Comments
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914349
I don't think wget would be able to delete the log on the server.  wget simply retrives files via an httpclient session.  Does this website have ftp access?  If so, this could be done with ftp fairly easily.
0
 

Author Comment

by:purpleski
ID: 17914785
Yes I think it has ftp access. How could I tell?

Thanks
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914810
Attempt to ftp to the site:
From a dos prompt:
c:\ftp myhostname.com
0
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

 

Author Comment

by:purpleski
ID: 17915261
I get

Connected to purpleski.com.
220 pop3.gradwell.net NcFTPd Se
User (purpleski.com:(none)):

so yes I think!
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17915528
Assuming you know your username and password, you could create a batch job that would ftp for you.  I recommend you test it on some uselss files first.
Copy and paste EVERYTHING below this line into a file called get.bat:
@echo off
>script.ftp echo open ftp.somesite.com
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916143
Thanks

tried the following (obviously changed password for this post!)

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.
ftp> purpleski
Invalid command.
ftp> PASSWORD
Invalid command.
ftp> prompt
Interactive mode Off .
ftp> mget *
Not connected.
ftp> del ex060809.log
Not connected.
ftp> quit

What am I doing wrong
C:\>
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916196
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.

Your host doesn't exist or doesn't have an ftp  client running.  Are you sure that is where you need to ftp to?
0
 

Author Comment

by:purpleski
ID: 17916508
When I use CuteFTP to upload/download files I enter

Host Address
hathor.xcalibre.co.uk

Username
purpleski

password
************

any advice?

purpleperson
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916535
ftp.hathor.xcalibre.co.uk isn't the same as hathor.xcalibre.co.uk
@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916941
Getting there. I now get

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

Looks like

>>script.ftp cd/home/11550/purpleski/www.purpleski.com/logs

I have been told that  Physical filename on the Windows Server is

E:\home\11550\purpleski\www.purpleski.com


What am I doing wrong any thoughts?

Thanks for all your help on this one
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17917098
For one thing, there should be a space:
cd /home/11550/purpleski/www.purpleski.com/logs
For another, it depends on how the ftp server is set up.  Try log in yourself and type the command:
cd /home/11550/purpleski/www.purpleski.com/logs
and see where it takes you.

0
 

Author Comment

by:purpleski
ID: 17921258
Ok we are nearly there hope you can still help me

I can log on manually at a command prompt and do it all manually

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>ftp
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)): purpleski
331 Password required for purpleski.
Password:
230 User purpleski logged in.
ftp> cd www.purpleski.com/logs
250 CWD command successful.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
200 Type set to A.
200 PORT command successful.
150 Opening ASCII mode data connection for xxxx.log(3766362 bytes).
226 Transfer complete.
ftp: 3766362 bytes received in 16.70Seconds 225.49Kbytes/sec.
ftp> del xxxx.log
250 DELE command successful.
ftp> quit
221

C:\>

but if I run the script you gave me:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget xxxx.log
>>script.ftp echo del xxxx.log
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

The output is


C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

It looks like the cd instruction is being ignored.

Upped the points as this is obviously more difficult than I thought!!

purpleperson

0
 

Author Comment

by:purpleski
ID: 17921553
Hi

Did some googling which led me to change the script to

open hathor.xcalibre.co.uk
USERNAME
PASSWORD
cd www.purpleski.com/logs
prompt
mget *
del *
quit
ftp -s:script.ftp
del script.ftp

then at the command line I type

ftp -s:SCRIPT.TXT

This works.

Any thoughts?
0
 
LVL 18

Accepted Solution

by:
Morcalavin earned 250 total points
ID: 17922680
Ahh.  I see the problem now:
>>script.ftp cd www.purpleski.com/logs

This should be :
>>script.ftp echo cd www.purpleski.com/logs
0
 

Author Comment

by:purpleski
ID: 17928908
Thanks for all that it works perfectly as follows:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp echo cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

Going to use it properley as a scheduled task tonight at 00.01 GMT let us hope it works!

Points are yours
0

Featured Post

Webinar: Aligning, Automating, Winning

Join Dan Russo, Senior Manager of Operations Intelligence, for an in-depth discussion on how Dealertrack, leading provider of integrated digital solutions for the automotive industry, transformed their DevOps processes to increase collaboration and move with greater velocity.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
Because your company can’t afford for you to make SEO mistakes, you’ll want to ensure you’re taking the right steps each and every time you post a new piece of content. This list of optimization do’s and don’ts can help you become an SEO wizard.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question