n00b0101
asked on
shell script pattern matching / ftping?
I think I've written maybe one shell script my entire life, and I'm not even sure if it's possible to do this, but I'm trying to write a script that will ftp the contents of a directory, one at a time. That is, it'll ftp one and then close the connection, then ftp the second, and close that etc. This is because there may be up to five files in a directory all of which are a minimum of 2GB each. FTPing them all at once always results in a reset connection. I thought that if I could match by partial filename, then perhaps that will help, as they are all named the same way.
So, in a directory, it'll have:
SampleFileA_20100322_1.txt
SampleFileA_20100322_2.txt
SampleFileB_20100322_1.txt
SampleFileC_20100322_1.txt
I'd like to ftp SampleFileA_xxxx_1 first, then SampleFileA_xxxx_2, etc. This is the current ftp script, which tries to download everything all at once...
So, in a directory, it'll have:
SampleFileA_20100322_1.txt
SampleFileA_20100322_2.txt
SampleFileB_20100322_1.txt
SampleFileC_20100322_1.txt
I'd like to ftp SampleFileA_xxxx_1 first, then SampleFileA_xxxx_2, etc. This is the current ftp script, which tries to download everything all at once...
#!/bin/bash
REMOTE='ftp.EXAMPLE.com'
USER='USERNAME'
PASSWORD='PASSWORD'
FTPLOG='/tmp/ftplog'
date >> $FTPLOG
ftp -in $REMOTE <<EOF
_FTP>>$FTPLOG
quote USER $USER
quote PASS $PASSWORD
bin
cd download
mget *
quit
_FTP
:wq!
ASKER
I only know part of it, thus my question regarding pattern matching.
Hi,
you could achieve what you desire with an intermediate command script, generated "on the fly" - like the following.
Note that the script would transfer the matching files in the current local directory one by one, but not close the connection, which seems a bit unnecessary to me.
If this is desired nonetheless, please let me know.
#!/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"
date >> $FTPLOG
echo "open $REMOTE" > $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
ls SampleFile*.txt | while read FILE
do
echo "get $FILE" >> $CMDFILE
done
echo "quit" >> $CMDFILE
ftp -in < $CMDFILE >> $FTPLOG 2>&1
rm $CMDFILE
wmp
you could achieve what you desire with an intermediate command script, generated "on the fly" - like the following.
Note that the script would transfer the matching files in the current local directory one by one, but not close the connection, which seems a bit unnecessary to me.
If this is desired nonetheless, please let me know.
#!/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"
date >> $FTPLOG
echo "open $REMOTE" > $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
ls SampleFile*.txt | while read FILE
do
echo "get $FILE" >> $CMDFILE
done
echo "quit" >> $CMDFILE
ftp -in < $CMDFILE >> $FTPLOG 2>&1
rm $CMDFILE
wmp
ASKER
Well, I am trying this, and it looks like it's doing something, but I don't see any files actually being downloaded...
At the prompt, there's just this:
macbook-pro-175:admin$ ./receive.sh
And doing more /tmp/rc.ftp:
open ftp.example.com (changed from real domain)
user username password (changed from real username/password)
verbose
bin
cd download
And doing more /tmp/ftplog:
Tue Mar 23 06:30:49 PDT 2010
Tue Mar 23 06:31:13 PDT 2010
Tue Mar 23 06:32:15 PDT 2010
Tue Mar 23 06:38:47 PDT 2010
So, I'm not sure if it's actually doing anything? Is this expected behavior?
At the prompt, there's just this:
macbook-pro-175:admin$ ./receive.sh
And doing more /tmp/rc.ftp:
open ftp.example.com (changed from real domain)
user username password (changed from real username/password)
verbose
bin
cd download
And doing more /tmp/ftplog:
Tue Mar 23 06:30:49 PDT 2010
Tue Mar 23 06:31:13 PDT 2010
Tue Mar 23 06:32:15 PDT 2010
Tue Mar 23 06:38:47 PDT 2010
So, I'm not sure if it's actually doing anything? Is this expected behavior?
No, it isn't.
I intentionally used "verbose" to see what's going on, and there is obviously nothing being done.
It just looks like as if there were no files matching the pattern in the current directory.
I used SampleFile*.txt as you did in in your question.
Did you replace this string with your actual value? Do you run the script from the directory the files are in?
wmp
I intentionally used "verbose" to see what's going on, and there is obviously nothing being done.
It just looks like as if there were no files matching the pattern in the current directory.
I used SampleFile*.txt as you did in in your question.
Did you replace this string with your actual value? Do you run the script from the directory the files are in?
wmp
Sorry,
at the moment I'm not sure what we're actually trying to do.
Are we talking about download or upload? I designed the script for upload, but now I see that I used "get" in the script, because there is also "mget" in your question (copy and paste ...).
If it's indeed for download, I'll have to redesign the whole thing.
wmp
at the moment I'm not sure what we're actually trying to do.
Are we talking about download or upload? I designed the script for upload, but now I see that I used "get" in the script, because there is also "mget" in your question (copy and paste ...).
If it's indeed for download, I'll have to redesign the whole thing.
wmp
ASKER
Yes, it is for download, not upload.
OK,
in this case we will have to do an intermediate step to get the remote listing first -
#!/usr/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"
PATTERN="SampleFile*.txt"
date > $FTPLOG
rm $CMDFILE 2>/dev/null
LISTING=$(ftp -in $REMOTE <<EOF
user $USER $PASSWORD
cd download
ls $PATTERN
quit
EOF )
echo "open $REMOTE" >> $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
for FILE in $LISTING
do
echo "get $FILE" >> $CMDFILE
done
echo "quit" >> $CMDFILE
ftp -in < $CMDFILE >> $FTPLOG 2>&1
#rm $CMDFILE
Please note that I put the file pattern in a variable PATTERN to make it easier to modify.
For a complete download use PATTERN="*" or omit PATTERN=... as a whole.
wmp
in this case we will have to do an intermediate step to get the remote listing first -
#!/usr/bin/bash
REMOTE="ftp.EXAMPLE.com"
USER="USERNAME"
PASSWORD="PASSWORD"
CMDFILE="/tmp/rc.ftp"
FTPLOG="/tmp/ftplog"
PATTERN="SampleFile*.txt"
date > $FTPLOG
rm $CMDFILE 2>/dev/null
LISTING=$(ftp -in $REMOTE <<EOF
user $USER $PASSWORD
cd download
ls $PATTERN
quit
EOF )
echo "open $REMOTE" >> $CMDFILE
echo "user $USER $PASSWORD" >> $CMDFILE
echo "verbose" >> $CMDFILE
echo "bin" >> $CMDFILE
echo "cd download" >> $CMDFILE
for FILE in $LISTING
do
echo "get $FILE" >> $CMDFILE
done
echo "quit" >> $CMDFILE
ftp -in < $CMDFILE >> $FTPLOG 2>&1
#rm $CMDFILE
Please note that I put the file pattern in a variable PATTERN to make it easier to modify.
For a complete download use PATTERN="*" or omit PATTERN=... as a whole.
wmp
ASKER
Well, it kind of worked, but the same thing happens as in the original script I posted:
125 Data connection already open; Transfer starting.
421 Service not available, remote server timed out. Connection closed
243357154 bytes received in 15:41 (252.31 KB/s)
ftp: No control connection for command.
I'm trying to avoid the timeout, which is why I thought I should close and reopen the connection?
125 Data connection already open; Transfer starting.
421 Service not available, remote server timed out. Connection closed
243357154 bytes received in 15:41 (252.31 KB/s)
ftp: No control connection for command.
I'm trying to avoid the timeout, which is why I thought I should close and reopen the connection?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
... and the CMDFILE now carries the PID as a suffix, to make it unique.
It will now be called something like /tmp/rc.ftp.12345
It will now be called something like /tmp/rc.ftp.12345
ASKER
Thanks!
hope this helps.