Solved

FTP Script help

Posted on 2006-06-29
4
194 Views
Last Modified: 2013-12-16
Hi I need a script that can be run in cron to connect to a FTP server and download the most recently modified .gz file starting with the filename starting access. in a directory to a folder naming it file.gz

thanks for you help
0
Comment
Question by:loppyrabit
  • 3
4 Comments
 
LVL 22

Expert Comment

by:pjedmond
ID: 17008555
This will depend on the ftp server concerned, and how it presents the index of the files, but something along the lines of:

----------8X------------
#!/bin/bash

wget ftp://myserver.com/path/ -O index.html     #There may be a .listing (or similar index file)
FILENAME=`cat index.html | grep "^access.*\.gz$" | sed q`
rm -Rf $FILENAME                                            # Delete file if it already exists
wget ftp://myserver.com/path/$FILENAME .        # Get the new one
----------8X------------

The difficult bit is deciding what selection criteria you should use for deciding which file to copy.

Of course, you could just copy the whole folder, using the -N timestamping option to ensure that only newer files are downloaded. (or wget the .listing file if it exists and use that to decide which files to download>.

A good look at man wget is suggested

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

is recommended reading, along with:

http://www.softpanorama.org/Tools/Awk/awk_one_liners.shtml

in case the 'analysis' of the index requires a fair amount of work to get the correct file for downloading.

(   (()
(`-' _\
 ''  ''

is recommended to give some ideas as

0
 

Author Comment

by:loppyrabit
ID: 17008984
Hi Thanks for your help

this is the script based on yours i am using

wget ftp://sight.com/logs/ -O index.html
FILENAME=`cat index.html | grep "access.*.gz" | sed q`
echo "$FILENAME"
#rm -Rf $FILENAME
wget $FILENAME

whne it echos out finame it gives me the complete html line ie:
2006 May 08 01:45  File        <a href="ftp://sight.com:21/logs/access.log.18.gz">access.log.18.gz</a>  (2,675,536 bytes)

and then it attempts to wget 2006 not access.log.18.gz

but access.log.18.gz is the oldist file in the dir not the newist

Please help
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 195 total points
ID: 17010241
I do not know what ftpserver you have, or indeed how the data is presented in the .listing. Hence me providing links to sed and awk for you to finish it off.

Anyway, to extract the filename out of that line change to:

FILENAME=`cat index.html | grep "access.*.gz" | sed q | cut -d '"' -f2`

You need to work out how the dates are presented in the list, so that (if necessary), you can use sort or, after uploading the tar.gz file rename it, or whatever approach you decide on.

Oops - didn't add the 'sed' one liners:

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

(   (()
(`-' _\
 ''  ''



0
 
LVL 22

Expert Comment

by:pjedmond
ID: 17010254
Top Tip - I recommend that you use cut and paste on that line - After the cut it is a single quote each side of a double quote. with " being used as the seperator.

man cut for more info.

(   (()
(`-' _\
 ''  ''
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
Using 'screen' for session sharing, The Simple Edition Step 1: user starts session with command: screen Step 2: other user (logged in with same user account) connects with command: screen -x Done. Both users are connected to the same CLI sessio…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now