asked on

In over my head - scripting in bash for FTP

I'm throwing all the points I have into this question.

I am writing a script that will automate an FTP process that is currently a manual process at the moment.

Currently:
A directory sits out on the file system. An administrator goes in, creates the directory structure on the remote server to match the local directory structure, then copies all the files locally to the remote server via ftp.

I have written a script that can automatically login and create a directory structure based upon a predefined set of rules.
This same script can copy an entire directory over to the remote server so long as the directory structure already exists.

However, I do not know how to implement into the script the results of a find command that gives me all the files that have been modified in the last 24 hours (find -ctime -1). I have a script that runs the find command and dumps the data into a file called changed.txt. I also do not know how to dynamically create the directory structure on the remote machine. When I try to do a copy without creating it, I get an error that the file cannot be created.

Now, all paths are relative to a predefined path (/mnt/raid/JPG). Within this JPG directory it's broken down further (07/PB/01_Hardware/Filename_to_copy.jpg - as an example). On the remote directory, relative from login root (/) I need it to create a directory 07, a directory 07/PB, a directory 07/PB/01_hardware, then copy the file over. And I need to do this from a list that is in my changed.txt file.

I know I can feed data into a script by using <<. However, I don't know what that looks like to the script. I don't even know how to test what that looks like to the script.

Please help!!!

Tintin

Do you have to use FTP? Sounds like a much better job for rsync.

Ziggie013

ASKER

Yes, unfortunately. The remote server belongs to a customer and this is the only access they will allow.

But I agree with you about rsync.

Mysidia

There is a tool called wput which may be of some help
http://wput.sourceforge.net/

amit_g

Make a small file TestDataFile with about 10 entries of your find result and run this on it

awk -F/ '{
for (i = 1 ; i < NF ; i++)
{
print "mkdir " $i
print "cd " $i
}
print "put " $0
for (i = 1 ; i < NF ; i++)
{
print "cd .."
}
}' TestDataFile

Save the result and insert into your automated ftp script as commands after the login is successful. Make sure to cd and lcd to starting directories before thes estatements. Does it work? If so we can enhance it to do the complete job automatically.

Ziggie013

ASKER

This doesn't work based on this script, and I think I know why....

When I ran the awk command I was given a long list of things which I pushed into my script. The script failed to copy any files, but did create the directory structure. Adding a 'print " cd /"' before 'print "put " $0' rectified the situation. Because the find command returned a full path relative to the starting directory, when it tried to put the file in while in the most recently created directory it was trying to follow the path out from where it was (did I just confuse everyone along with myself? I think, also, I could eliminate the second for loop since I'm selecting the root directory again?

At the very least, the script does work when I hard code all the lines, including the cd /, into the ftp script. So now the next step is automation...

Thanks for all your help.

amit_g

Please post the modified awk script. Also post the script that you used to ftp. Make sure to mask the servername/username/password if present in that script.

Ziggie013

ASKER

The two scripts are below. However, I just found out a couple things. One, I don't need the 07 information at all. I don't want to create the Year directory, as they remove the files after 60 days anyway. Second, the files need to be put into folders based on the month. The Sale Code (PA, PB, PC, etc) determines the month by using the second letter: A - January, B-February, C-March, etc. So, on top of all of this I have to include some logic to determine the month based on the file I'm looking at.

I also found a program called ncftpput which allows, from the command line, the username/pw information and will recursively upload a directory. From what I can tell, it will even take input from a text file, if I can figure out how to do that. It may be the easier way to go here.

The ftp script below was originally very simple. But using the echo for every line grew cumbersome.

Thank you thank you thank you thank you for all your help!

modified awk script:

awk -F/ '{
for (i=1 ; i<NF ; i++)
{
print "mkdir " $i
print "cd " $i
}
print "cd /"
print "put " $0
}' TestDataFile

Here is the ftp script:

awk -F/ '{
for (i=1 ; i<NF ; i++)
{
print "echo "mkdir " $i
print "cd " $i
}
print "cd /"
print "echo "put " $0
}' TestDataFile

#!/bin/sh

temp="/home/user/ftpoutput.txt"
remote="SERVER"
basedir="/mnt/raid/press/JPG/Sale_Pages"
rm $temp
echo "quote user USERNAME" >> $temp
echo "quote pass PASSWORD" >> $temp

echo "lcd $basedir" >> $temp

echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg" >> $temp
echo "echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating_600.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating.pdf" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior_150.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior_600.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior.pdf" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/771_Interior_with_5_Gallons_150.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/771_Interior_with_5_Gallons_600.jpg" >> $temp

echo "quit" >> $temp

ftp -in $remote < $temp

exit 0

Ziggie013

ASKER

Oops, got overexcited iwth copy/paste.

Here is the ftp script:

#!/bin/sh

temp="/home/user/ftpoutput.txt"
remote="SERVER"
basedir="/mnt/raid/press/JPG/Sale_Pages"
rm $temp
echo "quote user USERNAME" >> $temp
echo "quote pass PASSWORD" >> $temp

echo "lcd $basedir" >> $temp

echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating_600.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Hardware" >> $temp
echo "cd 01_Hardware" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Hardware/006_Electrical_and_Heating.pdf" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior_150.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior_600.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/770_Interior.pdf" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/771_Interior_with_5_Gallons_150.jpg" >> $temp
echo "mkdir ." >> $temp
echo "cd ." >> $temp
echo "mkdir 07" >> $temp
echo "cd 07" >> $temp
echo "mkdir PA" >> $temp
echo "cd PA" >> $temp
echo "mkdir 01_Paint" >> $temp
echo "cd 01_Paint" >> $temp
echo "cd /" >> $temp
echo "put ./07/PA/01_Paint/771_Interior_with_5_Gallons_600.jpg" >> $temp

echo "quit" >> $temp

ftp -in $remote < $temp

exit 0

amit_g

Make a new file say SyncViaFtp.sh and put this in it

awk -F/ '
BEGIN {
print "temp=\"/home/user/ftpoutput.txt\""
print "remote=\"SERVER\""
print "basedir=\"/mnt/raid/press/JPG/Sale_Pages\""
print "rm $temp"
print "ftp -in $remote <<EOF"
print "quote user USERNAME"
print "quote pass PASSWORD"
print "lcd $basedir"
}

{
for (i=1 ; i<NF ; i++)
{
print "mkdir " $i
print "cd " $i
}
print "cd /"
print "put " $0
}
END {
print "quit"
print "EOF"
}
' $1

Now execute it by commandline

SyncViaFtp.sh YourDataFile

Do this for a small datafile. You should see a script on the screen. Verify that it looks good. If so execute the command

SyncViaFtp.sh YourDataFile | sh

and it should do ftp. Obviously you will need to do some tweaking in it but the basic idea is to generate the script dynamically based on datafile and execute that script using sh. Let us know how it went.

Ziggie013

ASKER

That worked, and you're awesome!

Any thoughts on how to evaluate each line to determine where each file should go?

On the local file system they are stored in 07/PA/01_Hardware/Filename_to_copy.jpg. On the Remote system, it would go here:

/January/PA/01_hardware/Filename_to_copy.jpg

The A in January tells me I need to put the file in the January folder.

In Visual Basic I would do a Case Statement:

Select Case left($filename, 5, 1)
Case A
Case B
Case C
etc.

I would then do a replace. Replacing ./07/ with ./January/ for A ./February for B, etc.

Is there a way to do that with Bash?

Or rather, how would I do that with Bash?

I can do a Case statement on a variable, but my lack of knowledge regarding file inputs is hurting me in this case.

I also don't know how to select the nth character from the variable I am looking at, or do a replace, once I have the information I need.

Please and thank you.

amit_g

How does your input file look?

07/PA/01_Hardware/Filename_to_copy.jpg

or

./07/PA/01_Hardware/Filename_to_copy.jpg

Also could you please post the output that you got by running this script. Just do it with 2-3 lines of input to keep it small.

Where does the file is put currently in the remote system by this script?

Ziggie013

ASKER

The input file displays like this:

./PA/01_Hardware/006_Electrical_and_Heating_150.jpg
./PA/01_Hardware/006_Electrical_and_Heating_600.jpg
./PA/01_Hardware/006_Electrical_and_Heating.pdf

The output of the script is:

temp="/home/user/ftpoutput.txt"
remote="SERVER"
basedir="/mnt/raid/press/JPG/Sale_Pages"
rm $temp
ftp -in $remote <<EOF
quote user USER
quote pass PASS
lcd $basedir
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_600.jpg
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating.pdf
quit
EOF

Currently, the temp="/home/chad/ftpoutput.txt"
remote="dibacdc"
basedir="/mnt/raid/press/JPG/Sale_Pages"
rm $temp
ftp -in $remote <<EOF
quote user ftptest
quote pass print
lcd $basedir
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_600.jpg
mkdir .
cd .
mkdir 07
cd 07
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating.pdf
quit
EOF

Currently, the file 07/PA/01_Hardware/Filename_to_copy.jpg would be put in (on the remote server): /January/PA/01_Hardware/Filename_to_copy.jpg. The script, as is, puts it in 07/PA/01_Hardware/Filename_to_copy.jpg (which I realize is what I initially said I needed. I just finally got access to the remote system today and found out they hadn't told me a few things).

ASKER CERTIFIED SOLUTION

amit_g

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Ziggie013

ASKER

The output (included below) is still using the /07/ path for the put operation, which is trying to, on the remote server, put it in an /07/PA directory, versus the /January/PA

I think rather than building this into this script, maybe it needs to be another script that modifies the list of file locations, replacing ./07/ with the correct iteration of the Month (based on the second character of the sale code [PB, PA]).

Thanks.

temp="/home/user/ftpoutput.txt"
remote="SERVER"
basedir="/mnt/raid/press/JPG/Sale_Pages"
rm $temp
ftp -in $remote <<EOF
quote user USER
quote pass PASSWORD
lcd $basedir
mkdir January
cd January
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg
mkdir January
cd January
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating_600.jpg
mkdir January
cd January
mkdir PA
cd PA
mkdir 01_Hardware
cd 01_Hardware
cd /
put ./07/PA/01_Hardware/006_Electrical_and_Heating.pdf
quit
EOF

Ziggie013

ASKER

OK, so this script:

year=`date '+%y'`
OriginalFile="/home/user/TestDataFile"
NewFile="/home/user/NewerDataFile"

case $year in
00) year1=01;;
01) year1=02;;
02) year1=03;;
03) year1=04;;
04) year1=05;;
05) year1=06;;
06) year1=07;;
07) year1=08;;
08) year1=09;;
*) year1=`exp $year + 1`;;
esac
`sed -e "s|.*|& &|" -e "s|./$year|./January|2" -e "s|./$year1|./January|2"<$OriginalFile >$NewFile`
exit 0

will take the year, add one to it (I kept getting single digit years without the case statement), and use sed to duplicate the lines (so I have ./07/PA.... ./07/PA....) and then to look through this year and next years and replace the 2nd ./07 with ./January.

I can use a string expression:
stringZ="./07/PA/01_Hardware/006_Electrical_and_Heating_150.jpg"
echo `expr substr $stringZ 7 1`

To return the A in the sale code (PA) and use a case statement from that to choose the correct month based on the sale code, before I run the sed commands.

However, I can't figure out how to get the string expression to read from the file. Every thing I've tried keeps giving me syntax errors.

Any suggestions?

Please and thank you.

Ziggie013

ASKER

For the curious, here is my final script:

#!/bin/bash

#set variables
remote="SERVER"
username="USER"
password="PASSWORD"
year=`date '+%y'`
OriginalFile="/tmp/DataFile"
WorkingFile="/tmp/NewerDataFile"
FinalFile="/tmp/FinalFile"
base="/mnt/raid/press/JPG/Sale_Pages/"
ftpscript="/root/scripts/FTP/FTP.sh"

#Remove Files from previous runs.
`rm -f $FinalFile`

#Create the list of changed files.
cd $base
`find -ctime -2 > $OriginalFile`

#Duplicate the lines of code for put operation
`sed "s|.*|& &|"<$OriginalFile >$WorkingFile`

#Get the total number of lines in the list
line_count=`wc $WorkingFile | awk '{ print $1 }'`

#Determine this year and next year.
case $year in
00) year1=01;;
01) year1=02;;
02) year1=03;;
03) year1=04;;
04) year1=05;;
05) year1=06;;
06) year1=07;;
07) year1=08;;
08) year1=09;;
*) year1=`exp $year + 1`;;
esac

#Determine the Month from the sale code
for name in `seq $line_count`; do
read name
monthcode=`expr substr "$name" 7 1`
salecode=`expr substr "$name" 6 2`

case $monthcode in
A) month=January;;
B) month=February;;
C) month=March;;
D) month=April;;
E) month=May;;
F) month=June;;
G) month=July;;
H) month=August;;
I) month=September;;
J) month=October;;
K) month=November;;
L) month=December;;
esac

#Replace the year in the filename with the month in the filename.
`sed -n -e "s|./$year/$salecode|./$month/$salecode|2p" -e "s|./$year1/$salecode|./$month/$salecode|2p"<$WorkingFile >>$FinalFile`

done < $WorkingFile

#run the FTP Script based on the file that was created
`$ftpscript $remote $username $password $base $FinalFile|sh >> /home/chad/ftpscript.log`

exit 0

The ftpscript calls:

#!/bin/bash
remote=$1
username=$2
password=$3
basedir=$4
FinalFile=$5

awk -F/ '
BEGIN {
Months["A"] = "January"
Months["B"] = "February"
Months["C"] = "March"
                  Months["D"] = "April"
                  Months["E"] = "May"
                  Months["F"] = "June"
                  Months["G"] = "July"
                  Months["H"] = "August"
                  Months["I"] = "September"
                  Months["J"] = "October"
                  Months["K"] = "November"
                  Months["L"] = "December"

print "ftp -in '$remote' <<EOF"
print "quote user '$username'"
print "quote pass '$password'"
print "lcd '$basedir'"
}
{
MonthIndex = substr($3, 2, 1)
print "cd " Months[MonthIndex]
for (i=3 ; i<5 ; i++)
{
print "mkdir " $i
print "cd " $i
}
print "cd /"
print "put " $0 $1
}
END {
print "quit"
print "EOF"
}
' $FinalFile
exit 0

Thanks for all your help.

amit_g

Sorry for not being able to get back to this earlier. I am glad that you got it resolved in the mean time.

Ziggie013

ASKER

No worries. You definitely got me headed in the right direction. It was good to get this figured out on my own in some way.

Any suggestions for improvements are welcome, but not necessary as I have a working script!

Thanks again!