Solved

sftp logs

Posted on 2014-11-11
13
189 Views
Last Modified: 2014-11-20
Hi there,
Need a little assistance with setting up some basic logging in this script.

As an example below, I need the script below to log to a file for 24 hours,  then start a new log file each day for the full calendar month, then overwrite the calendar starting on the 1st day of each month.
example belot does that:
vmstat |awk 'NR == 3 {now=strftime("%Y-%m-%d %T "); print now $0}' >> /home/user/logs/vmstat.log.`date +%d`


#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
lcd /directory1
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory1/*.xml /directory1/archive/

lcd /directory2
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml                                                             [send the files]
!mv /directory2/*.xml /directory2/archive/      [move the files to archive]

lcd /directory3                                                        [move to the next directory and repeat]
!pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
!ls     >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
mput *.xml
!mv /directory3/*.xml /directory3/archive/

exit
EOF
0
Comment
Question by:Peter Kuczynski
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 5
13 Comments
 
LVL 29

Expert Comment

by:fibo
ID: 40442182
Mhhh.. Your script presently does keeps each daily log for a full month.
What is missing to achieve your goal?

Side remark: you might consider creating a loop in your script, since the code for each of the 3 directories is "the same"
0
 
LVL 21

Expert Comment

by:Mazdajai
ID: 40442515
What distro are you using?  Are you trying to rotate the log on a monthly basis?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40443187
Thank you fibo for your comments. I agree, a loop would be better. I just updated this yesterday, adding the logging, but would like to know how to incorporate a loop.

Mazdajai; The distro is Amazon linux kernel v 3.10.53-56.140.amzn1.x86_64 I am trying to rotate on a monthly basis yes.
0
Transaction Monitoring Vs. Real User Monitoring

Synthetic Transaction Monitoring Vs. Real User Monitoring: When To Use Each Approach? In this article, we will discuss two major monitoring approaches: Synthetic Transaction and Real User Monitoring.

 
LVL 29

Accepted Solution

by:
fibo earned 500 total points
ID: 40443292
For the loop, and the test for non-empty directory,  it would be something like:
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5

for mydir in /directory1 /directory2 /directory3
do
   lcd $mydir
   if [ "$(ls -A $mydir)" ]; then
     !pwd >> /var/log/sftp/sftp_$(date '+%d').log    [to log what directory im in]
     !ls  >> /var/log/sftp/sftp_$(date '+%d').log     [to log any files ready to be sent in that directory]
     mput *.xml
     !mv /$mydir/*.xml /$mydir/archive/
   fi
done

exit
EOF 

Open in new window

(the elements to navigate thru in the for loop are enumerated with spaces).
Note: I have added spaces for readability, you might need to remove them.
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40443932
Thanks fibo, I think I understand this .
If I have different file types in the different directories, can I just append that to the script?
I added .ppt to the script below

     mput *.xml
     !mv /$mydir/*.xml *.ppt /$mydir/archive/
   fi
0
 
LVL 29

Expert Comment

by:fibo
ID: 40444227
Différent file types: your mod seems fine, but of course test and check.
You don't want to mputDifférent file types: currently the loop and its body are oriented to directories ppt, right?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40444578
>currently the loop and its body are oriented to directories ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat


Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?

Today, the customers server is down:
./sftp.sh
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer





.
0
 
LVL 29

Expert Comment

by:fibo
ID: 40444621
>currently the loop and its body are oriented to directories [...] ppt, right?

I have about 27 directories with 3 different file types in the real world script. .xml .csv and .dat
If in each directory you want to act on xml, csv and dat file, the program will be easy (and presumably also if in a given directory one of the categories is missing)

Also is it possible to send an email alert if for some reason the sftp session wont connect to the server it's designed to connect to?
This depends not just on the shell, but also on your sftp client and on your sftp server. The simplest case would be if the sftp returns an error code, then you can handle this within your script. BUT since you get the log, you would find the error messages there

In a similar case, I send by mail a daily confirmation. If no error, then the mail is just "OK". If any error, I tell "errors" AND send the log as an attchment.

You could probably achieve that by adding any error message to an error.log, sending it by mail if not empty, then emptying it (so that it is ready for next day)
0
 
LVL 29

Expert Comment

by:fibo
ID: 40445627
I have about 27 directories
If by chance they are all in the same dir (say /home/userdata ) and if this dir holds no other directory, the code would become
#!/bin/sh

sftp username@192.168.123.123 <<EOF
sleep 5
home="/myhome" #the directory containing the directories to manage

for mydir in $home/*
do
   if [ -d $mydir ]; then
	   if [ "$(ls -A $mydir)" ]; then
	     !echo $mydir >> /var/log/sftp/sftp_$(date '+%d').log   #[to log what directory im in]
	     !ls  $mydir >> /var/log/sftp/sftp_$(date '+%d').log    #[to log any files ready to be sent in that directory]
	     mput $mydir/*.xml
	     !mv $mydir/*.xml $mydir/archive/
	   fi
   fi
done

exit
EOF 

Open in new window

0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40445819
Where I'm confused is where does the script lcd [change directories] to the different sub-directories ?
 
Then it still needs to upload a different file type,
meaning, /directory1 may have *.dat files ready to upload
directory2 may have *.xml files ready to upload


Regarding error handling, all I get from the sftp server I am trying to send to is:
sftp username@192.168.234.123
ssh_exchange_identification: read: Connection reset by peer
Couldn't read packet: Connection reset by peer
This is sent to the console only and not logged as an error.
I suppose I will need to add a if else statement at the top something like if connection is OK, run the script, if not, send an email.
So if the script runs, do nothing, just run the script, but if somehow the script dosnt run, for whatever reason, send a script failed alert?
0
 
LVL 29

Expert Comment

by:fibo
ID: 40445949
1 - oops, lcd should still be there

2 - If you (might) have 3 different filetypes, simply try to upload all of them (if that is correct). [Depending on your problem: if files of several types might be present in a directory but you do want to upload just one filetype... then the script would be slightly more complicated and would probably use an external config file giving the directories and each of the types which should be archived from this directory]

3 - sftp connection error: you seemed to imply that THIS is working. Has it been working or is this connection not tested yet?
0
 
LVL 1

Author Comment

by:Peter Kuczynski
ID: 40455935
thank you fibo, I ran out of time to fully test this, I will come back to this in the near future, for now I will close this case, and thanks!
0
 
LVL 29

Expert Comment

by:fibo
ID: 40456099
B-) Glad to help.
Thx for the grade and points.
0

Featured Post

Webinar: Aligning, Automating, Winning

Join Dan Russo, Senior Manager of Operations Intelligence, for an in-depth discussion on how Dealertrack, leading provider of integrated digital solutions for the automotive industry, transformed their DevOps processes to increase collaboration and move with greater velocity.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Utilizing an array to gracefully append to a list of EmailAddresses
Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
Progress

729 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question