?
Solved

FTP Upload Script

Posted on 2005-04-11
16
Medium Priority
?
16,433 Views
Last Modified: 2013-11-17
I've been given the task of creating an FTP script that will copy files (created by our ERP app) from one directory to an archive directory, send them to a remote FTP server and then delete them from the original directory. Since the files are named based on the time created - the name changes and there will be more than one that needs to be processed. I will need to "wildcard" the PUT command for the contents of the directory.

This process has to be scheduled so I plan on creating a cron job to do that but I'm not all that versed in scripting in a UNIX environment. We are running AIX 5.1. Any suggesting would be VERY appreciated.
0
Comment
Question by:kaesm
  • 7
  • 3
  • 2
  • +2
14 Comments
 
LVL 20

Expert Comment

by:cpc2004
ID: 13751594
./ftp_sample.sh hostname ftp.cmd target_path source_path user password

##ftp_sample.sh
echo "open $1\nuser $5 $6\n" > $2
echo "user " >> $2
echo "bin" >> $2
echo "cd $3" >> $2
echo "lcd $4" >> $2
echo "prompt" >> $2
echo "mget *" >> $2
echo "dir\nquit" >> $2
chmod 744 $2
ftp -n  < $2
rm $2
0
 
LVL 20

Expert Comment

by:cpc2004
ID: 13751607
Sample script to ftp an entire directory from another unix system. Make sure that the userid has the read access authority at the target unix system.

./ftp_sample.sh hostname ftp.cmd target_path source_path user password

##ftp_sample.sh
echo "open $1\nuser $5 $6\n" > $2
echo "cd $3" >> $2
echo "lcd $4" >> $2
echo "prompt" >> $2
echo "mget *" >> $2
echo "dir\nquit" >> $2
chmod 744 $2
ftp -n  < $2
rm $2

0
 
LVL 3

Expert Comment

by:esanchezvela
ID: 13769152

the solution given by cpc2004 is almost complete, he/she overlooked the fact that the script has to delete the files from the original location, I would not do a wild  "mdel * " in the ftp script but maybe a "mdel <suffix>* "  right after the "mget *"

regards,
esv.
ps. also, instead of using "mget *" I would have use " mget <suffix>*" to avoid getting stuff not intended for me.

0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:kaesm
ID: 13769995
I do not think I posed my query very well so here it goes again -

Our system generates picking and invoice files (basic text file) that need to be uploaded to our transport provider on a scheduled basis. The basic function requirements are:

The files are created and placed in the "out" directory by our ERP application
An FTP session is then established with the external provider from this UNIX server and the files are uploaded to their receiving directory (Windows FTP server)
The files are deleted from the original upload (out) directory and moved to an "out_save" directory for archive purposes.
What I've done thus far is to create a kermit script as follows:

#!/usr/local/bin/kermit +

 ; Open the connection and log in
 ;
  ftp open ftp.remoteftpserver.com /user:"UserName" /password:"Password"
  if fail exit 1 Connection failed
  if not \v(ftp_loggedin) exit 1 Login failed

 ; Change to local orders upload directory
 ;
  lcd /local ftp directory/
 
 ;  OK to send - cd to server's working directory.
 ;
  ftp cd /upload directory
  if fail exit 1 ftp cd working: \v(ftp_message)

 ;  Now we upload the files, moving each
 ;  uploaded copy when successful.
 ;
  ftp mput * /DELETE /MOVE-TO:../out_save/
  if fail exit 1 Upload Failed

  bye
  exit 0

The upload succeeds but the files are not moved. As much as I hate to - the password has to be hard coded, but that's something to address later since I don't believe they are running secure FTP. For now I just need it to work. Sorry for the confusion but I hope this makes more sense..
0
 
LVL 3

Expert Comment

by:esanchezvela
ID: 13770877

 Has this ever worked?  if you delete the files from its original location, how do u expect to be able to move them later to somewhere else, just move them.

or you want to move the files on the receiving ftp server?

regards,
esv.
0
 

Author Comment

by:kaesm
ID: 13771201
Missed that bit - should have been this but the answer is no - it doesn't, even corrected as show below...

The files are transfered correctly but they are not moved to the out_save folder.

#!/usr/local/bin/kermit +

 ; Open the connection and log in
 ;
  ftp open ftp.remoteftpserver.com /user:"UserName" /password:"Password"
  if fail exit 1 Connection failed
  if not \v(ftp_loggedin) exit 1 Login failed

 ; Change to local orders upload directory
 ;
  lcd /local ftp directory/
 
 ;  OK to send - cd to server's working directory.
 ;
  ftp cd /upload directory
  if fail exit 1 ftp cd working: \v(ftp_message)

 ;  Now we upload the files, moving each
 ;  uploaded copy when successful.
 ;
  ftp mput * /MOVE-TO:../out_save/
  if fail exit 1 Upload Failed

  bye
  exit 0
0
 

Author Comment

by:kaesm
ID: 13771226
One other thing - I run the script using this command line

# /tmp/script_name | tee /tmp/script_outputfile_name.out

This works  as well but I want to also see the files copied in the log.
0
 
LVL 3

Expert Comment

by:esanchezvela
ID: 13771456
probably you need to redirect stderr as well.

 using bsh, ksh and bash style shells ....

        /tmp/script_name 2>&1 | tee /tmp/.....

regards,
esv
0
 

Author Comment

by:kaesm
ID: 13777704
That doesn't correcty the fact that the files are not being moved and seems to work either way.
0
 

Author Comment

by:kaesm
ID: 13778867
I found the issue with the move - the * was in the wrong position in the script. What I can't get to work is a check of the local upload directory. I want to be able to query it to see if there are any files in it. If not it needs to quit the script. The line I am using is:

ftp check /upload_directory/*
if fail exit 1 Transfers directory is empty!

The result of this (even when ran at the command line) is:

The system cannot find the path specified

The way this problem is progressing I am wondering if I should repost it in the programming area rather than here...
0
 

Author Comment

by:kaesm
ID: 13779032
This is what I ended up with and I can see the transfers but not the file names that are being sent. It really doesn't matter if the upload directory is empty or not  - the script stops regardless.

#!/usr/local/bin/kermit +

set ftp verbose-mode on

; Open the connection and log in
;
  ftp open ftp.remotesite.com /user:username /password:password
  if fail exit 1 Connection failed!
  if not \v(ftp_loggedin) exit 1 Login failed!
 
; Change to local orders upload directory on Local Server
;
  lcd /localdirectory
  if fail exit 1 lcd /localdirectory: \v(errstring)

; Check to see if the local upload directory is empty and quit if it is
;
; ftp check /localdirectory/*
; if fail goto transfer

; OK to send - cd to Remote upload directory.
;
  ftp cd /UploadDirectory
  if fail exit 1 ftp cd //UploadDirectory: \v(ftp_message)

; Now we upload the files, moving each uploaded copy when successful.
;
  ftp mput /MOVE-TO:/savedirectory/ *
  if fail exit 1 ftp mput \v(filename): \v(ftp_message)
 
  bye
  exit 0
0
 

Expert Comment

by:pinball22
ID: 13827329
Here's a shell script to do the entire thing -- hopefully I've understood all your requirements correctly.

#!/bin/ksh

# This script assumes that the files you want to transfer are in a
# directory called /mypath/mydirectory, the directory you want to archive
# them to is /mypath/myarchivedir, the remote machine you want to connect
# to is called remotehost, your login to that machine is username
# with password password, and you want to put the files in a directory
# called /remotepath/remotedirectory -- you should replace these with
# the appropriate information, of course.  It's also assumed that you
# want to transfer/remove ALL files in the source directory.

# All log information for this process is directed to the file
# log$$.log in the /mypath/myarchivedir directory -- $$ is the shell
# variable for the process ID of the current process, so the file will
# be named something like log12345.log.  This is a simple way to create
# a fairly unique filename for the log.

# Check to see if the directory is empty by checking to see if the
# result of an ls of the directory is null.  If so, write a message
# out to the log and exit.

if [[ -z `ls /mypath/mydirectory` ]]; then
   echo "Directory empty, exiting." > /mypath/myarchivedir/log$$.log
   exit
fi

# Write the contents of the directory out to the log.
ls -lt /mypath/mydirectory >> /mypath/myarchivedir/log$$.log

# Copy the contents of the source directory to the archive
# directory, using the -p option to preserve file modification times.
cp -p /mypath/mydirectory/* /mypath/myarchivedir

# Connect to the machine remotehost via ftp using these flags:
# -i   Turns off prompting so that multiple files can be transferred
#      without user intervention.
# -n   Turns off checking for automatic logins set up in .netrc.
# -v   Turns on verbose mode.
#
# Standard output from the ftp command is appended to the log file;
# standard error is also sent to the log file using 2>&1.
#
# The << symbols after the command indicate the beginning of a "here
# document" -- a section of text that is to be sent to output exactly as
# written.  The END after << signifies that the end of the here document
# will be at the next appearance of END.
#
# Within the here document are the commands to be sent to the ftp server:
#
# user username password    Logs in to the server as user username
#                                         with password password.
# lcd /mypath/mydirectory   Changes to the local directory containing
#                                       the files to be uploaded.
# cd /remotepath/remotedir  Changes to the remote directory the files
#                                        are to be transferred to.
# mput *                    Transfers all files in the current local directory.
# close                       Closes the connection to the server.
# bye                         Exits the ftp process.
#

ftp -inv remotehost >> /mypath/myarchivedir/log$$.log 2>&1 << END
user username password
lcd /mypath/mydirectory
cd /remotepath/remotedir
mput *
close
bye
END

# Remove all files in /mypath/mydirectory once they've been transferred.
rm /mypath/mydirectory/*
0
 

Author Comment

by:kaesm
ID: 14523708
Here is what I ultimately came up with using Kermit as the command interpreter. Works like a charm!

#!/usr/local/bin/kermit +

set ftp verbose-mode on
date

; Open the connection and log in
;
  ftp open ftp.kagan.com.au /user:ftpusername /password:password
  if fail exit 1 Connection failed!
  if not \v(ftp_loggedin) exit 1
 
; Change to local orders upload directory on Newpronto
;
  lcd /pro/pronto/edi/out/berger/orders
  if fail exit 1 lcd /pro/pronto/edi/out/berger/orders: \v(errstring)

; OK to send - cd to Kagan upload directory.
;
  ftp cd /u/KAGDAT/EDIORDERIN
  if fail exit 1 ftp cd /u/KAGDAT/EDIORDERIN: \v(ftp_message)

; Now we upload the files, moving each uploaded copy when successful.
;
  ftp mput /MOVE-TO:/pro/pronto/edi/out_save/berger/orders/ *
  if fail exit 1 ftp mput \v(filename): \v(ftp_message)

; List contents of upload directory to verify receipt
 
 ls
 
  bye
  exit 0

0
 

Accepted Solution

by:
CetusMOD earned 0 total points
ID: 14586877
PAQ'd, 500 points refunded.
CetusMOD
Community Support Moderator
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question