Solved

Wget dowload web site log files and delete file

Posted on 2006-11-10
15
359 Views
Last Modified: 2008-02-01
Hi

The log files are delivered to a folder on web server /purpleski/www.purpleski.com/logs.  Can the following be done with wget and if so can anybody give me the necessary instructions to do it!?

When run the wget command fetches any files in the directory /purpleski/www.purpleski.com/logs downloads these to my local computer to E:\Websites\purpleski\logs and then deletes any files in /purpleski/www.purpleski.com/logs.

This means that each day when it is run there will only be one file to download.

If the above won't work does anybody know of another solution to achieve the above?

Thanks.

purpleperson
0
Comment
Question by:purpleski
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 7
15 Comments
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914349
I don't think wget would be able to delete the log on the server.  wget simply retrives files via an httpclient session.  Does this website have ftp access?  If so, this could be done with ftp fairly easily.
0
 

Author Comment

by:purpleski
ID: 17914785
Yes I think it has ftp access. How could I tell?

Thanks
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17914810
Attempt to ftp to the site:
From a dos prompt:
c:\ftp myhostname.com
0
Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

 

Author Comment

by:purpleski
ID: 17915261
I get

Connected to purpleski.com.
220 pop3.gradwell.net NcFTPd Se
User (purpleski.com:(none)):

so yes I think!
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17915528
Assuming you know your username and password, you could create a batch job that would ftp for you.  I recommend you test it on some uselss files first.
Copy and paste EVERYTHING below this line into a file called get.bat:
@echo off
>script.ftp echo open ftp.somesite.com
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916143
Thanks

tried the following (obviously changed password for this post!)

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.
ftp> purpleski
Invalid command.
ftp> PASSWORD
Invalid command.
ftp> prompt
Interactive mode Off .
ftp> mget *
Not connected.
ftp> del ex060809.log
Not connected.
ftp> quit

What am I doing wrong
C:\>
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916196
ftp> open ftp.hathor.xcalibre.co.uk
Unknown host ftp.hathor.xcalibre.co.uk.

Your host doesn't exist or doesn't have an ftp  client running.  Are you sure that is where you need to ftp to?
0
 

Author Comment

by:purpleski
ID: 17916508
When I use CuteFTP to upload/download files I enter

Host Address
hathor.xcalibre.co.uk

Username
purpleski

password
************

any advice?

purpleperson
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17916535
ftp.hathor.xcalibre.co.uk isn't the same as hathor.xcalibre.co.uk
@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo user
>>script.ftp echo pass
>>script.ftp cd /your/path/to/log/files
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp
0
 

Author Comment

by:purpleski
ID: 17916941
Getting there. I now get

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

Looks like

>>script.ftp cd/home/11550/purpleski/www.purpleski.com/logs

I have been told that  Physical filename on the Windows Server is

E:\home\11550\purpleski\www.purpleski.com


What am I doing wrong any thoughts?

Thanks for all your help on this one
0
 
LVL 18

Expert Comment

by:Morcalavin
ID: 17917098
For one thing, there should be a space:
cd /home/11550/purpleski/www.purpleski.com/logs
For another, it depends on how the ftp server is set up.  Try log in yourself and type the command:
cd /home/11550/purpleski/www.purpleski.com/logs
and see where it takes you.

0
 

Author Comment

by:purpleski
ID: 17921258
Ok we are nearly there hope you can still help me

I can log on manually at a command prompt and do it all manually

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>cd\

C:\>ftp
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)): purpleski
331 Password required for purpleski.
Password:
230 User purpleski logged in.
ftp> cd www.purpleski.com/logs
250 CWD command successful.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
200 Type set to A.
200 PORT command successful.
150 Opening ASCII mode data connection for xxxx.log(3766362 bytes).
226 Transfer complete.
ftp: 3766362 bytes received in 16.70Seconds 225.49Kbytes/sec.
ftp> del xxxx.log
250 DELE command successful.
ftp> quit
221

C:\>

but if I run the script you gave me:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget xxxx.log
>>script.ftp echo del xxxx.log
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

The output is


C:\>get
The system cannot find the path specified.
ftp> open hathor.xcalibre.co.uk
Connected to hathor.xcalibre.co.uk.
220 Microsoft FTP Service
User (hathor.xcalibre.co.uk:(none)):
331 Password required for purpleski.

230 User purpleski logged in.
ftp> prompt
Interactive mode Off .
ftp> mget xxxx.log
xxxx.log: The system cannot find the file specified.
200 Type set to A.
ftp> del xxxx.log
550 xxxx.log: The system cannot find the file specified.
ftp> quit
221

C:\>

It looks like the cd instruction is being ignored.

Upped the points as this is obviously more difficult than I thought!!

purpleperson

0
 

Author Comment

by:purpleski
ID: 17921553
Hi

Did some googling which led me to change the script to

open hathor.xcalibre.co.uk
USERNAME
PASSWORD
cd www.purpleski.com/logs
prompt
mget *
del *
quit
ftp -s:script.ftp
del script.ftp

then at the command line I type

ftp -s:SCRIPT.TXT

This works.

Any thoughts?
0
 
LVL 18

Accepted Solution

by:
Morcalavin earned 250 total points
ID: 17922680
Ahh.  I see the problem now:
>>script.ftp cd www.purpleski.com/logs

This should be :
>>script.ftp echo cd www.purpleski.com/logs
0
 

Author Comment

by:purpleski
ID: 17928908
Thanks for all that it works perfectly as follows:

@echo off
>script.ftp echo open hathor.xcalibre.co.uk
>>script.ftp echo purpleski
>>script.ftp echo mRgyzUpF
>>script.ftp echo cd www.purpleski.com/logs
>>script.ftp echo prompt
>>script.ftp echo mget *
>>script.ftp echo del *
>>script.ftp echo quit
ftp -s:script.ftp
del script.ftp

Going to use it properley as a scheduled task tonight at 00.01 GMT let us hope it works!

Points are yours
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When it comes to write a Context Sensitive Help (an online help that is obtained from a specific point in state of software to provide help with that state) ,  first we need to make the file that contains all topics, which are given exclusive IDs. …
An enjoyable and seamless user experience can go a long way on an eCommerce site. While a cohesive layout and engaging copy play roles in creating a positive user experience, some sites neglect aspects that seem marginal but in actuality prove very …
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
The is a quite short video tutorial. In this video, I'm going to show you how to create self-host WordPress blog with free hosting service.

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question