[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 657
  • Last Modified:

UNIX Shell script to ftp logs on daily basis


I need to ftp logs from 2 webservers to an ftp server on daily basis.

Assume the webservers are Web1 & Web4 while the ftp server's
hostname (defined in the webservers' /etc/hosts) is ftpsvr

Web1 server:
==========
at 0600hrs daily,
a) ftp to ftpsvr using the login id ftpusr, password is ftpasswd
b) mkdir YYYYMMMDD_web1
c) cd YYYYMMMDD_web1
d) put (not mput) in Ascii mode from Web1's directory /etc/httpd/w1_log
      access_YYYYMMMDD.log
e) put (not mput) in Ascii mode from Web1's directory /etc/httpd/w1_log
      ssl_access_YYYYMMMDD.log files
f) quit
g) on the web1, do a "gzip -9 ..." of the above 2 files that were ftp'ed
    across to the ftpsvr


Web4 server:
==========
at 0630hrs daily,
a) ftp to ftpsvr using the login id ftpusr, password is ftpasswd
b) mkdir YYYYMMMDD_web4
c) cd YYYYMMMDD_web4
d) put (not mput) in Ascii mode from Web4's directory /etc/httpd/w4_log
      access_YYYYMMMDD.log
e) put (not mput) in Ascii mode from Web4's directory /etc/httpd/w4_log
      ssl_access_YYYYMMMDD.log files
f) quit

g) on web4, do a "gzip -9 ..." of the above 2 files that were ftp'ed
    across to the ftpsvr


Note that YYYYMMMDD is current day's date in numeric eg:
20110501  is first of May, year 2011

0
sunhux
Asked:
sunhux
  • 6
  • 6
5 Solutions
 
a1jCommented:
#!/bin/sh

stamp=`date "+%Y%m%d`

ftp -n ftpsvr << EOT
quote user ftpuser
quote pass ftpass
passive
ascii
prompt
mkdir ${stamp}_web1
cd ${stamp}_web1
lcd /etc/httpd/
put w1_logaccess_${stamp}.log
put w4_logssl_access_${stamp}.log
quit
EOT
#
exit $?
=============================
Install this script into cron via crontab -e -u <username>

crontab -e -u root
MAILTO=wheretomailerrors@domain.com
0 6 * * * /path/ftpscript.sh >/dev/null

==================================


Write second script similar to this one and install it into cron for 30 6 * * *
0
 
a1jCommented:
But ideally you should be not using ftp for this because it transmits clear text passwords.

The ideal solution for you would be to set up ssh keys and use rsync over ssh:
rsync --timeout 900 -r -t -e 'ssh -oBatchmode=yes'  --safe-links  <src> <dst>

Rsync is much more reliable than ftp and will not leave you with partially incomplete files.
0
 
sunhuxAuthor Commented:

Thanks v much a1j.  rsync is a good idea, just that I need to transfer files
in ASCII mode which I believe rsync does not support.  Is there anyway
to circumvent (say using rsync with redirection, pipe & unix2dos?

Kindly provide me the exact syntax of rsync with |, >, unix2dos if possible
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
a1jCommented:
You can run conversion program after syncing files
Lets say find /directory -type f -name "*.txt" -exec sed -i -e's/.$//' {} \;
will convert all files from DOS to unix (sed edit string is taken from sed 1-liners) - http://sed.sourceforge.net/sed1line.txt
0
 
sunhuxAuthor Commented:


> convert all files from DOS to unix

I'm putting files from Linux to a Windows server, so will need to convert from
UNIX to DOS format; would you be able to revise the syntax of the command
& incorporate rsync as part of the command as well?

Thanks vm
0
 
a1jCommented:
Try this if you are under BASH
find /directory -type f -name "*.txt" -exec sed -i -e's/$'"/`echo \\\r`/" {} \;

Or use unix2dos converters under windows (google up).

0
 
sunhuxAuthor Commented:

Let me make & attempt & you correct me if I got it wrong.

To put files in ASCII from my Linux box to the remote Windows server,
  rsync --timeout 900 -r -t -e 'ssh -oBatchmode=yes'  --safe-links
     `cat srcfile | sed -i -e's/$'"/`echo \\\r`` <dst>

To get files in ASCII from the remote Windows server to my Linux box,
  rsync --timeout 900 -r -t -e 'ssh -oBatchmode=yes'  --safe-links
     `type srcfile_in_Windows | sed -i -e's/\r'"/`echo \$`` dstfile_in_Linux
0
 
sunhuxAuthor Commented:

One other thing : would rsync retain the ownership / permissioning
(eg rw-r--r-- of the destination files) ?
0
 
a1jCommented:
Rsync will retain ownership/permissions and even data stamps if you specify -a flag. Also rsync will transfer delta only, it will not transfer whatever it is already there. So it is useful to make any editing or conversion before launching rsync.  The commands your posted probably wont work (rsync has no -exec switch similar to find).
Also rsync windows port is somewhat unreliable and you need to jump through the hoops to get it work with putty so maybe ftp script posted above is better solution (it is not clear from your question the platform you are using).
If you transfer files between UNIX machines the rsync is the best way of doing so.
0
 
sunhuxAuthor Commented:

My ftp client that initiates both puts & gets is a Linux box while
the ftp server is a Windows 2003 server
0
 
a1jCommented:
Yeh so use shell script then.
0
 
sunhuxAuthor Commented:
excellent
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 6
  • 6
Tackle projects and never again get stuck behind a technical roadblock.
Join Now