?
Solved

FTP Script help

Posted on 2006-06-29
4
Medium Priority
?
199 Views
Last Modified: 2013-12-16
Hi I need a script that can be run in cron to connect to a FTP server and download the most recently modified .gz file starting with the filename starting access. in a directory to a folder naming it file.gz

thanks for you help
0
Comment
Question by:loppyrabit
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 22

Expert Comment

by:pjedmond
ID: 17008555
This will depend on the ftp server concerned, and how it presents the index of the files, but something along the lines of:

----------8X------------
#!/bin/bash

wget ftp://myserver.com/path/ -O index.html     #There may be a .listing (or similar index file)
FILENAME=`cat index.html | grep "^access.*\.gz$" | sed q`
rm -Rf $FILENAME                                            # Delete file if it already exists
wget ftp://myserver.com/path/$FILENAME .        # Get the new one
----------8X------------

The difficult bit is deciding what selection criteria you should use for deciding which file to copy.

Of course, you could just copy the whole folder, using the -N timestamping option to ensure that only newer files are downloaded. (or wget the .listing file if it exists and use that to decide which files to download>.

A good look at man wget is suggested

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

is recommended reading, along with:

http://www.softpanorama.org/Tools/Awk/awk_one_liners.shtml

in case the 'analysis' of the index requires a fair amount of work to get the correct file for downloading.

(   (()
(`-' _\
 ''  ''

is recommended to give some ideas as

0
 

Author Comment

by:loppyrabit
ID: 17008984
Hi Thanks for your help

this is the script based on yours i am using

wget ftp://sight.com/logs/ -O index.html
FILENAME=`cat index.html | grep "access.*.gz" | sed q`
echo "$FILENAME"
#rm -Rf $FILENAME
wget $FILENAME

whne it echos out finame it gives me the complete html line ie:
2006 May 08 01:45  File        <a href="ftp://sight.com:21/logs/access.log.18.gz">access.log.18.gz</a>  (2,675,536 bytes)

and then it attempts to wget 2006 not access.log.18.gz

but access.log.18.gz is the oldist file in the dir not the newist

Please help
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 780 total points
ID: 17010241
I do not know what ftpserver you have, or indeed how the data is presented in the .listing. Hence me providing links to sed and awk for you to finish it off.

Anyway, to extract the filename out of that line change to:

FILENAME=`cat index.html | grep "access.*.gz" | sed q | cut -d '"' -f2`

You need to work out how the dates are presented in the list, so that (if necessary), you can use sort or, after uploading the tar.gz file rename it, or whatever approach you decide on.

Oops - didn't add the 'sed' one liners:

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

(   (()
(`-' _\
 ''  ''



0
 
LVL 22

Expert Comment

by:pjedmond
ID: 17010254
Top Tip - I recommend that you use cut and paste on that line - After the cut it is a single quote each side of a double quote. with " being used as the seperator.

man cut for more info.

(   (()
(`-' _\
 ''  ''
0

Featured Post

WordPress Tutorial 1: Installation & Setup

WordPress is a very popular option for running your web site and can be used to get your content online quickly for the world to see. This guide will walk you through installing the WordPress server software and the initial setup process.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month11 days, 21 hours left to enroll

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question