shell script pattern matching / ftping?

I think I've written maybe one shell script my entire life, and I'm not even sure if it's possible to do this, but I'm trying to write a script that will ftp the contents of a directory, one at a time. That is, it'll ftp one and then close the connection, then ftp the second, and close that etc. This is because there may be up to five files in a directory all of which are a minimum of 2GB each. FTPing them all at once always results in a reset connection. I thought that if I could match by partial filename, then perhaps that will help, as they are all named the same way.

So, in a directory, it'll have:

SampleFileA_20100322_1.txt
SampleFileA_20100322_2.txt
SampleFileB_20100322_1.txt
SampleFileC_20100322_1.txt

I'd like to ftp SampleFileA_xxxx_1 first, then SampleFileA_xxxx_2, etc. This is the current ftp script, which tries to download everything all at once...
#!/bin/bash
REMOTE='ftp.EXAMPLE.com'
USER='USERNAME'
PASSWORD='PASSWORD'
FTPLOG='/tmp/ftplog'
date >> $FTPLOG

ftp -in $REMOTE <<EOF
_FTP>>$FTPLOG
quote USER $USER
quote PASS $PASSWORD
bin
cd download
mget *
quit
_FTP
:wq!

Open in new window

n00b0101Asked:
Who is Participating?
 
woolmilkporcConnect With a Mentor Commented:
OK,
that's not a big change in our script, although I doubt that it will have much effect.

#!/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp.$$"
FTPLOG="/tmp/ftplog"

REMOTEDIR="download"
PATTERN="SampleFile*.txt"

date > $FTPLOG

rm $CMDFILE 2>/dev/null

LISTING=$(ftp -in $REMOTE <<EOF
user $USER $PASSWORD
cd $REMOTEDIR
ls $PATTERN
quit
EOF )

for FILE in $LISTING
 do
  echo "open $REMOTE" >> $CMDFILE
  echo "user $USER $PASSWORD" >> $CMDFILE
  echo "verbose" >> $CMDFILE
  echo "bin" >> $CMDFILE
  echo "cd $REMOTEDIR" >> $CMDFILE
  echo "get $FILE" >> $CMDFILE
  echo "verbose" >> $CMDFILE
  echo "close" >> $CMDFILE
 done
echo "quit" >> $CMDFILE

ftp -in < $CMDFILE >> $FTPLOG 2>&1

#rm $CMDFILE

Please note that I introduced a new variable REMOTEDIR, to make changes easier.

wmp



0
 
zkrrzkCommented:
if you know the filenames then you can call this script in a loop inside shell script.
hope this helps.
0
 
n00b0101Author Commented:
I only know part of it, thus my question regarding pattern matching.
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
woolmilkporcCommented:
Hi,
you could achieve what you desire with an intermediate command script, generated "on the fly" - like the following.
Note that the script would transfer the matching files in the current local directory one by one, but not close the connection, which seems a bit unnecessary to me.
If this is desired nonetheless, please let me know.

#!/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"

date >> $FTPLOG
echo "open $REMOTE" > $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
ls SampleFile*.txt | while read FILE
 do
  echo "get $FILE" >> $CMDFILE
 done
echo "quit" >> $CMDFILE

ftp -in < $CMDFILE >> $FTPLOG 2>&1
rm $CMDFILE
wmp
 
 
0
 
n00b0101Author Commented:
Well, I am trying this, and it looks like it's doing something, but I don't see any files actually being downloaded...  

At the prompt, there's just this:
macbook-pro-175:admin$ ./receive.sh

And doing more /tmp/rc.ftp:

open ftp.example.com  (changed from real domain)
user username password   (changed from real username/password)
verbose
bin
cd download

And doing more /tmp/ftplog:

Tue Mar 23 06:30:49 PDT 2010
Tue Mar 23 06:31:13 PDT 2010
Tue Mar 23 06:32:15 PDT 2010
Tue Mar 23 06:38:47 PDT 2010



So, I'm not sure if it's actually doing anything?  Is this expected behavior?
0
 
woolmilkporcCommented:
No, it isn't.
I intentionally used "verbose" to see what's going on, and there is obviously nothing being done.
It just looks like as if there were no files matching the pattern in the current directory.
I used  SampleFile*.txt as you did in in your question.
Did you replace this string with your actual value? Do you run the script from the directory the files are in?
wmp
0
 
woolmilkporcCommented:
Sorry,
at the moment I'm not sure what we're actually trying to do.
Are we talking about download or upload? I designed the script for upload, but now I see that I used "get" in the script, because there is also "mget" in your question (copy and paste ...).
If it's indeed for download, I'll have to redesign the whole thing.
wmp
0
 
n00b0101Author Commented:
Yes, it is for download, not upload.  
0
 
woolmilkporcCommented:
OK,
in this case we will have to do an intermediate step to get the remote listing first -

#!/usr/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"
PATTERN="SampleFile*.txt"
date > $FTPLOG
rm $CMDFILE 2>/dev/null
LISTING=$(ftp -in $REMOTE <<EOF
user $USER $PASSWORD
cd download
ls $PATTERN
quit
EOF )
echo "open $REMOTE" >> $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
 for FILE in $LISTING
   do
    echo "get $FILE" >> $CMDFILE
   done
echo "quit" >> $CMDFILE
  ftp -in < $CMDFILE >> $FTPLOG 2>&1
#rm $CMDFILE
Please note that I put the file pattern in a variable PATTERN to make it easier to modify.
For a complete download use PATTERN="*" or omit PATTERN=... as a whole.
wmp
0
 
n00b0101Author Commented:
Well, it kind of worked, but the same thing happens as in the original script I posted:

125 Data connection already open; Transfer starting.
421 Service not available, remote server timed out. Connection closed
243357154 bytes received in 15:41 (252.31 KB/s)
ftp: No control connection for command.


I'm trying to avoid the timeout, which is why I thought I should close and reopen the connection?
0
 
woolmilkporcCommented:
... and the CMDFILE now carries the PID as a suffix, to make it unique.
It will now be called something like /tmp/rc.ftp.12345
0
 
n00b0101Author Commented:
Thanks!
0
All Courses

From novice to tech pro — start learning today.