Solved

sftp logs

Posted on 2014-11-11
13
187 Views
Last Modified: 2014-11-20
Hi there,
Need a little assistance with setting up some basic logging in this script.

As an example below, I need the script below to log to a file for 24 hours,  then start a new log file each day for the full calendar month, then overwrite the calendar starting on the 1st day of each month.
example belot does that:
vmstat |awk 'NR == 3 {now=strftime("%Y-%m-%d %T "); print now $0}' >> /home/user/logs/vmstat.log.`date +%d`


#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
lcd /directory1
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory1/*.xml /directory1/archive/

lcd /directory2
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml                                                             [send the files]
!mv /directory2/*.xml /directory2/archive/      [move the files to archive]

lcd /directory3                                                        [move to the next directory and repeat]
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory3/*.xml /directory3/archive/

exit
EOF
0
Comment
Question by:Peter Kuczynski
  • 7
  • 5
13 Comments
 
LVL 29

Expert Comment

by:fibo
ID: 40442182
Mhhh.. Your script presently does keeps each daily log for a full month.
What is missing to achieve your goal?

Side remark: you might consider creating a loop in your script, since the code for each of the 3 directories is "the same"
0
 
LVL 21

Expert Comment

by:Mazdajai
ID: 40442515
What distro are you using?  Are you trying to rotate the log on a monthly basis?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40443187
Thank you fibo for your comments. I agree, a loop would be better. I just updated this yesterday, adding the logging, but would like to know how to incorporate a loop.

Mazdajai; The distro is Amazon linux kernel v 3.10.53-56.140.amzn1.x86_64 I am trying to rotate on a monthly basis yes.
0
Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

 
LVL 29

Accepted Solution

by:
fibo earned 500 total points
ID: 40443292
For the loop, and the test for non-empty directory,  it would be something like:
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5

for mydir in /directory1 /directory2 /directory3
do
   lcd $mydir
   if [ "$(ls -A $mydir)" ]; then
     !pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
     !ls  >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
     mput *.xml
     !mv /$mydir/*.xml /$mydir/archive/
   fi
done

exit
EOF 

Open in new window

(the elements to navigate thru in the for loop are enumerated with spaces).
Note: I have added spaces for readability, you might need to remove them.
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40443932
Thanks fibo, I think I understand this .
If I have different file types in the different directories, can I just append that to the script?
I added .ppt to the script below

     mput *.xml
     !mv /$mydir/*.xml *.ppt /$mydir/archive/
   fi
0
 
LVL 29

Expert Comment

by:fibo
ID: 40444227
Différent file types: your mod seems fine, but of course test and check.
You don't want to mputDifférent file types: currently the loop and its body are oriented to directories ppt, right?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40444578
>currently the loop and its body are oriented to directories ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat


Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?

Today, the customers server is down:
./sftp.sh
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer





.
0
 
LVL 29

Expert Comment

by:fibo
ID: 40444621
>currently the loop and its body are oriented to directories [...] ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat
If in each directory you want to act on xml, csv and dat file, the program will be easy (and presumably also if in a given directory one of the categories is missing)

Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?
This depends not just on the shell, but also on your sftp client and on your sftp server. The simplest case would be if the sftp returns an error code, then you can handle this within your script. BUT since you get the log, you would find the error messages there

In a similar case, I send by mail a daily confirmation. If no error, then the mail is just "OK". If any error, I tell "errors" AND send the log as an attchment.

You could probably achieve that by adding any error message to an error.log, sending it by mail if not empty, then emptying it (so that it is ready for next day)
0
 
LVL 29

Expert Comment

by:fibo
ID: 40445627
I have about 27 directories
If by chance they are all in the same dir (say /home/userdata ) and if this dir holds no other directory, the code would become
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
home="/myhome" #the directory containing the directories to manage

for mydir in $home/*
do
   if [ -d $mydir ]; then
	   if [ "$(ls -A $mydir)" ]; then
	     !echo $mydir >> /var/log/sftp/sftp_$(date '+%d').log   #[to log what directory im in]
	     !ls  $mydir >> /var/log/sftp/sftp_$(date '+%d').log    #[to log any files ready to be sent in that directory]
	     mput $mydir/*.xml
	     !mv $mydir/*.xml $mydir/archive/
	   fi
   fi
done

exit
EOF 

Open in new window

0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40445819
Where I'm confused is where does the script lcd [change directories] to the different sub-directories ?
 
Then it still needs to upload a different file type,
meaning, /directory1 may have *.dat files ready to upload
directory2 may have *.xml files ready to upload


Regarding error handling, all I get from the sftp server I am trying to send to is:
sftp username@192.168.234.123
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer
This is sent to the console only and not logged as an error.
I suppose I will need to add a if else statement at the top something like if connection is OK, run the script, if not, send an email.
So if the script runs, do nothing, just run the script, but if somehow the script dosnt run, for whatever reason, send a script failed alert?
0
 
LVL 29

Expert Comment

by:fibo
ID: 40445949
1 - oops, lcd should still be there

2 - If you (might) have 3 different filetypes, simply try to upload all of them (if that is correct). [Depending on your problem: if files of several types might be present in a directory but you do want to upload just one filetype... then the script would be slightly more complicated and would probably use an external config file giving the directories and each of the types which should be archived from this directory]

3 - sftp connection error: you seemed to imply that THIS is working. Has it been working or is this connection not tested yet?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40455935
thank you fibo, I ran out of time to fully test this, I will come back to this in the near future, for now I will close this case, and thanks!
0
 
LVL 29

Expert Comment

by:fibo
ID: 40456099
B-) Glad to help.
Thx for the grade and points.
0

Featured Post

Networking for the Cloud Era

Join Microsoft and Riverbed for a discussion and demonstration of enhancements to SteelConnect:
-One-click orchestration and cloud connectivity in Azure environments
-Tight integration of SD-WAN and WAN optimization capabilities
-Scalability and resiliency equal to a data center

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
Fine Tune your automatic Updates for Ubuntu / Debian
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

790 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question