Solved

sftp logs

Posted on 2014-11-11
13
183 Views
Last Modified: 2014-11-20
Hi there,
Need a little assistance with setting up some basic logging in this script.

As an example below, I need the script below to log to a file for 24 hours,  then start a new log file each day for the full calendar month, then overwrite the calendar starting on the 1st day of each month.
example belot does that:
vmstat |awk 'NR == 3 {now=strftime("%Y-%m-%d %T "); print now $0}' >> /home/user/logs/vmstat.log.`date +%d`


#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
lcd /directory1
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory1/*.xml /directory1/archive/

lcd /directory2
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml                                                             [send the files]
!mv /directory2/*.xml /directory2/archive/      [move the files to archive]

lcd /directory3                                                        [move to the next directory and repeat]
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory3/*.xml /directory3/archive/

exit
EOF
0
Comment
Question by:Peter Kuczynski
  • 7
  • 5
13 Comments
 
LVL 29

Expert Comment

by:fibo
Comment Utility
Mhhh.. Your script presently does keeps each daily log for a full month.
What is missing to achieve your goal?

Side remark: you might consider creating a loop in your script, since the code for each of the 3 directories is "the same"
0
 
LVL 21

Expert Comment

by:Mazdajai
Comment Utility
What distro are you using?  Are you trying to rotate the log on a monthly basis?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
Comment Utility
Thank you fibo for your comments. I agree, a loop would be better. I just updated this yesterday, adding the logging, but would like to know how to incorporate a loop.

Mazdajai; The distro is Amazon linux kernel v 3.10.53-56.140.amzn1.x86_64 I am trying to rotate on a monthly basis yes.
0
 
LVL 29

Accepted Solution

by:
fibo earned 500 total points
Comment Utility
For the loop, and the test for non-empty directory,  it would be something like:
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5

for mydir in /directory1 /directory2 /directory3
do
   lcd $mydir
   if [ "$(ls -A $mydir)" ]; then
     !pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
     !ls  >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
     mput *.xml
     !mv /$mydir/*.xml /$mydir/archive/
   fi
done

exit
EOF 

Open in new window

(the elements to navigate thru in the for loop are enumerated with spaces).
Note: I have added spaces for readability, you might need to remove them.
0
 
LVL 1

Author Comment

by:Peter Kuczynski
Comment Utility
Thanks fibo, I think I understand this .
If I have different file types in the different directories, can I just append that to the script?
I added .ppt to the script below

     mput *.xml
     !mv /$mydir/*.xml *.ppt /$mydir/archive/
   fi
0
 
LVL 29

Expert Comment

by:fibo
Comment Utility
Différent file types: your mod seems fine, but of course test and check.
You don't want to mputDifférent file types: currently the loop and its body are oriented to directories ppt, right?
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 
LVL 1

Author Comment

by:Peter Kuczynski
Comment Utility
>currently the loop and its body are oriented to directories ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat


Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?

Today, the customers server is down:
./sftp.sh
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer





.
0
 
LVL 29

Expert Comment

by:fibo
Comment Utility
>currently the loop and its body are oriented to directories [...] ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat
If in each directory you want to act on xml, csv and dat file, the program will be easy (and presumably also if in a given directory one of the categories is missing)

Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?
This depends not just on the shell, but also on your sftp client and on your sftp server. The simplest case would be if the sftp returns an error code, then you can handle this within your script. BUT since you get the log, you would find the error messages there

In a similar case, I send by mail a daily confirmation. If no error, then the mail is just "OK". If any error, I tell "errors" AND send the log as an attchment.

You could probably achieve that by adding any error message to an error.log, sending it by mail if not empty, then emptying it (so that it is ready for next day)
0
 
LVL 29

Expert Comment

by:fibo
Comment Utility
I have about 27 directories
If by chance they are all in the same dir (say /home/userdata ) and if this dir holds no other directory, the code would become
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
home="/myhome" #the directory containing the directories to manage

for mydir in $home/*
do
   if [ -d $mydir ]; then
	   if [ "$(ls -A $mydir)" ]; then
	     !echo $mydir >> /var/log/sftp/sftp_$(date '+%d').log   #[to log what directory im in]
	     !ls  $mydir >> /var/log/sftp/sftp_$(date '+%d').log    #[to log any files ready to be sent in that directory]
	     mput $mydir/*.xml
	     !mv $mydir/*.xml $mydir/archive/
	   fi
   fi
done

exit
EOF 

Open in new window

0
 
LVL 1

Author Comment

by:Peter Kuczynski
Comment Utility
Where I'm confused is where does the script lcd [change directories] to the different sub-directories ?
 
Then it still needs to upload a different file type,
meaning, /directory1 may have *.dat files ready to upload
directory2 may have *.xml files ready to upload


Regarding error handling, all I get from the sftp server I am trying to send to is:
sftp username@192.168.234.123
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer
This is sent to the console only and not logged as an error.
I suppose I will need to add a if else statement at the top something like if connection is OK, run the script, if not, send an email.
So if the script runs, do nothing, just run the script, but if somehow the script dosnt run, for whatever reason, send a script failed alert?
0
 
LVL 29

Expert Comment

by:fibo
Comment Utility
1 - oops, lcd should still be there

2 - If you (might) have 3 different filetypes, simply try to upload all of them (if that is correct). [Depending on your problem: if files of several types might be present in a directory but you do want to upload just one filetype... then the script would be slightly more complicated and would probably use an external config file giving the directories and each of the types which should be archived from this directory]

3 - sftp connection error: you seemed to imply that THIS is working. Has it been working or is this connection not tested yet?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
Comment Utility
thank you fibo, I ran out of time to fully test this, I will come back to this in the near future, for now I will close this case, and thanks!
0
 
LVL 29

Expert Comment

by:fibo
Comment Utility
B-) Glad to help.
Thx for the grade and points.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Displaying an arrayList in a listView using the default adapter is rarely the best solution. To get full control of your display data, and to be able to refresh it after editing, requires the use of a custom adapter.
A short article about problems I had with the new location API and permissions in Marshmallow
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now