Solved

FTP Script help

Posted on 2006-06-29
4
193 Views
Last Modified: 2013-12-16
Hi I need a script that can be run in cron to connect to a FTP server and download the most recently modified .gz file starting with the filename starting access. in a directory to a folder naming it file.gz

thanks for you help
0
Comment
Question by:loppyrabit
  • 3
4 Comments
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
This will depend on the ftp server concerned, and how it presents the index of the files, but something along the lines of:

----------8X------------
#!/bin/bash

wget ftp://myserver.com/path/ -O index.html     #There may be a .listing (or similar index file)
FILENAME=`cat index.html | grep "^access.*\.gz$" | sed q`
rm -Rf $FILENAME                                            # Delete file if it already exists
wget ftp://myserver.com/path/$FILENAME .        # Get the new one
----------8X------------

The difficult bit is deciding what selection criteria you should use for deciding which file to copy.

Of course, you could just copy the whole folder, using the -N timestamping option to ensure that only newer files are downloaded. (or wget the .listing file if it exists and use that to decide which files to download>.

A good look at man wget is suggested

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

is recommended reading, along with:

http://www.softpanorama.org/Tools/Awk/awk_one_liners.shtml

in case the 'analysis' of the index requires a fair amount of work to get the correct file for downloading.

(   (()
(`-' _\
 ''  ''

is recommended to give some ideas as

0
 

Author Comment

by:loppyrabit
Comment Utility
Hi Thanks for your help

this is the script based on yours i am using

wget ftp://sight.com/logs/ -O index.html
FILENAME=`cat index.html | grep "access.*.gz" | sed q`
echo "$FILENAME"
#rm -Rf $FILENAME
wget $FILENAME

whne it echos out finame it gives me the complete html line ie:
2006 May 08 01:45  File        <a href="ftp://sight.com:21/logs/access.log.18.gz">access.log.18.gz</a>  (2,675,536 bytes)

and then it attempts to wget 2006 not access.log.18.gz

but access.log.18.gz is the oldist file in the dir not the newist

Please help
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 195 total points
Comment Utility
I do not know what ftpserver you have, or indeed how the data is presented in the .listing. Hence me providing links to sed and awk for you to finish it off.

Anyway, to extract the filename out of that line change to:

FILENAME=`cat index.html | grep "access.*.gz" | sed q | cut -d '"' -f2`

You need to work out how the dates are presented in the list, so that (if necessary), you can use sort or, after uploading the tar.gz file rename it, or whatever approach you decide on.

Oops - didn't add the 'sed' one liners:

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

(   (()
(`-' _\
 ''  ''



0
 
LVL 22

Expert Comment

by:pjedmond
Comment Utility
Top Tip - I recommend that you use cut and paste on that line - After the cut it is a single quote each side of a double quote. with " being used as the seperator.

man cut for more info.

(   (()
(`-' _\
 ''  ''
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This is the error message I got (CODE) Error caused by incompatible libmp3lame 3.98-2 with ffmpeg I've googled this error message and found out sometimes it attaches this note "can be treated with downgrade libmp3lame to version 3.97 or 3.98" …
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now