[Last Call] Learn about multicloud storage options and how to improve your company's cloud strategy. Register Now

x
?
Solved

FTP Script help

Posted on 2006-06-29
4
Medium Priority
?
200 Views
Last Modified: 2013-12-16
Hi I need a script that can be run in cron to connect to a FTP server and download the most recently modified .gz file starting with the filename starting access. in a directory to a folder naming it file.gz

thanks for you help
0
Comment
Question by:loppyrabit
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 22

Expert Comment

by:pjedmond
ID: 17008555
This will depend on the ftp server concerned, and how it presents the index of the files, but something along the lines of:

----------8X------------
#!/bin/bash

wget ftp://myserver.com/path/ -O index.html     #There may be a .listing (or similar index file)
FILENAME=`cat index.html | grep "^access.*\.gz$" | sed q`
rm -Rf $FILENAME                                            # Delete file if it already exists
wget ftp://myserver.com/path/$FILENAME .        # Get the new one
----------8X------------

The difficult bit is deciding what selection criteria you should use for deciding which file to copy.

Of course, you could just copy the whole folder, using the -N timestamping option to ensure that only newer files are downloaded. (or wget the .listing file if it exists and use that to decide which files to download>.

A good look at man wget is suggested

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

is recommended reading, along with:

http://www.softpanorama.org/Tools/Awk/awk_one_liners.shtml

in case the 'analysis' of the index requires a fair amount of work to get the correct file for downloading.

(   (()
(`-' _\
 ''  ''

is recommended to give some ideas as

0
 

Author Comment

by:loppyrabit
ID: 17008984
Hi Thanks for your help

this is the script based on yours i am using

wget ftp://sight.com/logs/ -O index.html
FILENAME=`cat index.html | grep "access.*.gz" | sed q`
echo "$FILENAME"
#rm -Rf $FILENAME
wget $FILENAME

whne it echos out finame it gives me the complete html line ie:
2006 May 08 01:45  File        <a href="ftp://sight.com:21/logs/access.log.18.gz">access.log.18.gz</a>  (2,675,536 bytes)

and then it attempts to wget 2006 not access.log.18.gz

but access.log.18.gz is the oldist file in the dir not the newist

Please help
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 780 total points
ID: 17010241
I do not know what ftpserver you have, or indeed how the data is presented in the .listing. Hence me providing links to sed and awk for you to finish it off.

Anyway, to extract the filename out of that line change to:

FILENAME=`cat index.html | grep "access.*.gz" | sed q | cut -d '"' -f2`

You need to work out how the dates are presented in the list, so that (if necessary), you can use sort or, after uploading the tar.gz file rename it, or whatever approach you decide on.

Oops - didn't add the 'sed' one liners:

http://www.eng.cam.ac.uk/help/tpl/unix/sed.html

(   (()
(`-' _\
 ''  ''



0
 
LVL 22

Expert Comment

by:pjedmond
ID: 17010254
Top Tip - I recommend that you use cut and paste on that line - After the cut it is a single quote each side of a double quote. with " being used as the seperator.

man cut for more info.

(   (()
(`-' _\
 ''  ''
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

650 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question