Solved

Unix shell scripting

Posted on 2004-08-09
27
434 Views
Last Modified: 2010-04-21
Hi,

I'm in the process of writing a UNIX script that queries our daily backup log to verify if it was successful and then and email sent to my email address. The issue here is that all the daily backup logs are appended to one single logfile and since I dont have root privilages I cannot change this. So I have written a script that only writes the day before's backup (search by date) to another file and then I grep for errors in that file. The only problem is that backup starts on daily at 17:00 and finishes on day2 at 03:00, so when I get the logs for the day after (search by date range) it also get bits of day2 from the previous days (03:00). My script is a follows:

#!/bin/sh
mailadmin='my email address'
zerror=`cat tmp.output1|grep -i error`

y=`date +20%y%m%d`
x=`expr $y - 1`

cat /usr/local/tsg/log/oracle.log|grep $x> tmp.output1
cat /usr/local/tsg/log/oracle.log|grep $y>>tmp.output1

sleep 30

if [ -z "$zerror" ];
then
mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <tmp.output1 $mailadmin
else
mailx -s "BACKUP OF BWP FAILED" <tmp.output1 $mailadmin
fi

This is a copy of my log file:

20040809-03:00:04:ORABCK_NOTICE:BWP NOTICE ### START OF BACKUP RUN ###
20040809-03:00:04:ORABCK_NOTICE:BWP NOTICE Saving detailed log files for this run
20040809-03:00:06:ORABCK_NOTICE:BWP NOTICE SQL version is 8.1 - Oracle version is 8.1, 9.X or higher
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE Oracle major version is : 8.1
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE Generating config file only
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE ### END OF BACKUP RUN ###
20040809-17:00:02:ORABCK_NOTICE:BWP NOTICE ### START OF BACKUP RUN ###
20040809-17:00:02:ORABCK_NOTICE:BWP NOTICE Log Directory /usr/local/tsg/log/BWP/HOT200408091700 does not exist ... creating it now
20040809-17:00:04:ORABCK_NOTICE:BWP NOTICE SQL version is 8.1 - Oracle version is 8.1, 9.X or higher
20040809-17:00:10:ORABCK_NOTICE:BWP NOTICE Oracle major version is : 8.1
20040809-17:00:10:ORABCK_NOTICE:BWP NOTICE Starting HOT backup
20040809-17:00:10:ORABCK_NOTICE:BWP NOTICE Spawning backup of Tablespace:PSAPBTABD Instance:BWP
20040809-17:00:20:ORABCK_NOTICE:BWP NOTICE BWP:PSAPBTABD:/oracle/BWP/sapdata3/btabd_1/btabd.data1:Save starting
...............

See how the log file also contains entries from the prvious days backup at 03:00. Is it possible just to get everything after the following line only:

"20040809-17:00:02:ORABCK_NOTICE:BWP NOTICE ### START OF BACKUP RUN ###" (this is when the backup starts)

If so please advise what I have to ajdust in my script. Help would be appreciated.

thanks
Bud.
0
Comment
Question by:budrito12
  • 10
  • 9
  • 8
27 Comments
 
LVL 48

Expert Comment

by:Tintin
ID: 11759313
You have a novel introduction of a Y3K bug ;-)

Anyway, this is how I would write it:

#!/bin/sh
mailadmin='some@address'
log=/usr/local/tsg/log/oracle.log
temp=/tmp/$$
today=`date +%Y%m%d`
yesterday=`TZ=$TZ+24 date +%Y%m%d`

egrep "$today|$yesterday" $log | awk '/START OF BACKUP/,/END OF BACKUP/ {print}' | grep -i error >$temp

if [ -z $temp ]
then
   mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <$temp $mailadmin
else
   mailx -s "BACKUP OF BWP FAILED" <$temp $mailadmin
fi

rm -f $temp
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11759347
Not sure if you want to check the backup for the last 2 days, or only for the current day.

Here's how the script looks like:

#!/bin/sh
mailadmin='my email address'
logfile="/usr/local/tsg/log/oracle.log"

today=`date +20%y%m%d`
yesterday=``TZ=PST+24 date +20%y%m%d` # if for 1 day coment out

grep ^$today $logfile >/tmp/tmp.output1
grep ^$yesterday $logfile >>/tmp/tmp.output1

if grep -i error /tmp/tmp.output1 > /dev/null 2>&1 ; then
    mailx -s "BACKUP OF BWP FAILED"  $mailadmin </tmp/tmp.output1
else
    mailx -s "BACKUP OF BWP WAS SUCCESSFUL"  $mailadmin </tmp/tmp.output1
fi

exit
#End of script





0
 
LVL 38

Expert Comment

by:yuzh
ID: 11759351
Tintin,
     you type faster than me!
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11759357
But:
   mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <$temp $mailadmin
   will not work!
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11759360
yuhz.

Your version doesn't take into account that it will pick up part of the previous backup logs.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11759366
yuhz.

Explain why you think

mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <$temp $mailadmin

will not work.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11759387
yuhz.

I see what you are saying now.  Here's the fixed script

#!/bin/sh
mailadmin='some@address'
log=/usr/local/tsg/log/oracle.log
temp=/tmp/$$
today=`date +%Y%m%d`
yesterday=`TZ=$TZ+24 date +%Y%m%d`

egrep "$today|$yesterday" $log | awk '/START OF BACKUP/,/END OF BACKUP/ {print}'  >$temp

if grep -i error $temp >/dev/null 2>&1
then
   mailx -s "BACKUP OF BWP FAILED" <$temp $mailadmin
else
   mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <$temp $mailadmin
fi

rm -f $temp
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11759416
>>>"Your version doesn't take into account that it will pick up part of the previous backup logs."

I assume that Bud want to look at for two days, see: http:#11759347

I also tell Bud for 1 day need to comment out yesterday:
yesterday=``TZ=PST+24 date +20%y%m%d` # if for 1 day coment out
grep ^$yesterday $logfile >>/tmp/tmp.output1

>>>"mailx -s "BACKUP OF BWP WAS SUCCESSFUL" <$temp $mailadmin"

systax error for some OS version, missing email add:

you can do:
cat $temp | mailx -s "BACKUP OF BWP WAS SUCCESSFUL" $mailadmin
Or:
mailx -s "BACKUP OF BWP WAS SUCCESSFUL" $mailadmin < $temp

man mailx


0
 
LVL 48

Expert Comment

by:Tintin
ID: 11759496
Sometimes I can't see the obvious errors.  You're correct about the mailx parameters.
0
 

Author Comment

by:budrito12
ID: 11759891
Nice comments guys and thanks for your replies Yuzh and Tintin. I just want to check the backup for 1 day NOT 2 days. BUT the backup spans across two days. Example the backup starts at 17:00 today and ends at 03:00 the next day. So the date in the common log file would be for 2 days, hope you get what I mean.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11760517
Hi budrito12 ,

   You can use Tintin's script (http:#11759387) just rmember to fix the "mailx" syntax to:

mailx -s "BACKUP OF BWP WAS SUCCESSFUL" $mailadmin < $temp

0
 

Author Comment

by:budrito12
ID: 11768825
Guyz,

Your variable for yesterday's date is not right. I'm currently testing the script but for variable 'yesterday=`TZ=$TZ+24 date +%Y%m%d`
 I get today's (current) date. Try and echo variable yesterday and see what you get.

cheers
Bud.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11768867
What version of OS are you using, it works for me in Solaris
 
# yesterday=`TZ=$TZ+24 date +%Y%m%d`  
# echo $yesterday
20040810
# date  +%Y%m%d
20040811
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 38

Expert Comment

by:yuzh
ID: 11769028
TZ might not work on some OS version, in this case, have a look at the following
scripts (get yesterday, you can define it as a yesterday() function):

http:Q_20486669.html
0
 

Author Comment

by:budrito12
ID: 11769034
Guys,

The script needs to be alittle bit more smarter. O.K regarding your AWK statement, looking at the backup log file that I have posted there is also a ##START OF BACKUP RUN## at 03:00 for the previous days backup. Your script when I run it also picks this up. This is the problem that I have.

The backup log that I want picked up is run on the same day but starts at 17:00 and completes the next day at 03:00. So the log file should only contain entries after 17:00.

Hope this helps and also thanks for both your help in sorting this out.
0
 

Author Comment

by:budrito12
ID: 11769037
I'm on Compaq TRU 64 version 4.0f.

thanks
Bud.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11769076
In your original statement, you said that the backup starts at 17:00 and finishes at 03:00.  If that's the case, how is it that the log sample you show has the first backup *starting* at 03:00?
0
 

Author Comment

by:budrito12
ID: 11769574
Tintin,

The backup at 03:00:00 only backups the oracle config files as it only runs for 15secs. This is a separate cron jobs scheduled at 03:00 but appends the output to the one log file. I still want to check for erros in this backup as well. The actual backup starts at 17:00 and ends at 00:30 and then this backup kicks off at 03:00:04 and ends at 03:00:19 but they write to the same log file.

Sorry for the confusion if I may have caused any.

cheers
Bud.

0
 
LVL 48

Accepted Solution

by:
Tintin earned 125 total points
ID: 11769695
Try this version:

#!/bin/sh
mailadmin='some@address'
log=/usr/local/tsg/log/oracle.log
temp=/tmp/$$
today=`date +%Y%m%d`
yesterday=`TZ=$TZ+24 date +%Y%m%d`

egrep "$today|$yesterday" $log | sed 1d |  awk '/START OF BACKUP/,/END OF BACKUP/ {print}' | grep -i error >$temp

if [ -z $temp ]
then
   mailx -s "BACKUP OF BWP WAS SUCCESSFUL"  $mailadmin <$temp
else
   mailx -s "BACKUP OF BWP FAILED" $mailadmin <$temp

rm -f $temp
0
 

Author Comment

by:budrito12
ID: 11769805
Hi Tintin,

The log file is now not being emailed any reason why???. Also what does the variable:

temp=/tmp/$$ - what does this do

cheers
Bud.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11769863
because "....|grep -i error >$temp" only contains the errors!

change the script to:

#!/bin/sh
mailadmin='some@address'
log=/usr/local/tsg/log/oracle.log
temp=/tmp/$$
today=`date +%Y%m%d`
yesterday=`TZ=$TZ+24 date +%Y%m%d`

egrep "$today|$yesterday" $log | sed 1d |  awk '/START OF BACKUP/,/END OF BACKUP/ {print}' >$temp

if grep -i error $temp > /dev/null 2>&1 ; then
  mailx -s "BACKUP OF BWP FAILED"  $mailadmin <$temp
else
    mailx -s "BACKUP OF BWP WAS SUCCESSFUL"  $mailadmin <$temp
fi

 
0
 

Author Comment

by:budrito12
ID: 11769928
Yeap got the log file BUT it still contains the entries from 03:00:00.

20040809-03:00:04:ORABCK_NOTICE:BWP NOTICE ### START OF BACKUP RUN ###
20040809-03:00:04:ORABCK_NOTICE:BWP NOTICE Saving detailed log files for this run
20040809-03:00:06:ORABCK_NOTICE:BWP NOTICE SQL version is 8.1 - Oracle version is 8.1, 9.X or higher
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE Oracle major version is : 8.1
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE Generating config file only
20040809-03:00:19:ORABCK_NOTICE:BWP NOTICE ### END OF BACKUP RUN ###
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11769946
You can get rid of all the 03:00 by change:

egrep "$today|$yesterday" $log | sed 1d |  awk '/START OF BACKUP/,/END OF BACKUP/ {print}' >$temp

to

egrep "$today|$yesterday" $log | sed 1d | grep -v "03:00" | awk '/START OF BACKUP/,/END OF BACKUP/ {print}' >$temp
0
 

Author Comment

by:budrito12
ID: 11770200
You guys are legends in the making and scripting guru's. Many thanks for your help and Yuzh that grep -v "03:00" did the trick. Tintin and Yuzh thanks for the UNIX scripts as well and solving this.

I'II be splitting the points among both of you.

cheers
Bud

P.S - What does temp=/tmp/$$  mean.
0
 
LVL 38

Expert Comment

by:yuzh
ID: 11770236
"What does temp=/tmp/$$" is a tmp file in /tmp and use the current processID as the
file name!

$$ -- current PID

cheers!
0
 
LVL 48

Expert Comment

by:Tintin
ID: 11778759
Generally, it is best to create temporary files with a semi-unique name.  Using the Process ID gives you a reasonably unique name (for most purposes).
0
 

Author Comment

by:budrito12
ID: 11779945
Once again thanks guys.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now