Start Free Trial

asked on

improve / fine-tune a Shell script ( tail command?) so that it consumes less CPU

When I run a Shell script (codes are given below) for 10 seconds (timed
using my watch), it consumes the following amount of CPU resource :

/usr/local/bin > time ./chkXXXXcount.sh
real 6.8
user 1.1
sys 2.6

I suspect it's the tail command in the script but I'm not sure.
Need someone to fine-tune my codes below so that it consumes
less CPU. Running the script using "nohup nice ./chkXXXXcount.sh &"
is not an option. The codes of the script follows :

c2=5
c3=1

while [ 1 ]
do
c1=`tail -1 /app/bea/yyy/eservices/XXXX/countlog/Xppt2activecount_ManagedXpptServer.log | sed 's/=/ /' | awk '{print ($3)}'`
# echo $c1
# compare c1 and c2 so tt if same as last value, we dont alert again
if [ "$c1" -ge "250" ] && [ "$c1" -ne "$c2" ]
then
cd /app/bea/yyy/eservices/XXXX/countlog
tail -5 Xppt2activecount_ManagedXpptServer.log |sed s/ConcurrSessCnt/Cnt/ | ux2dos > /home/myuser/Xppt.txt
echo $c1 > /home/myuser/Xpptcnt.dat # this file is for SMS & Xppt.txt above is for email

if [ "$c3" -eq "1" ]
then
DT1=$(date +%d%H%M)
fi

if [ "$c3" -eq "5" ]
then
DT5=$(date +%d%H%M)
fi

chmod 777 /home/myuser/Xpptcnt.dat
chmod 777 /home/myuser/Xppt.txt
/var/tmp/Xppt/sendmlcnt.pl

sleep 9 # about 3 times the frequency of job in emsyyy01
rm -f /home/myuser/Xppt.txt
rm -f /home/myuser/Xpptcnt.dat

# if clauses below r to ensure we dont get too many emails/SMS ie
# pause for 2 mins if get 3 consecutive alerts within approx 1 min or so
if [ "$c3" -ge "3" ]
then
if [ "$DT1" -eq "$DT5" ]
then
sleep 120
c3=0
fi
fi

c3=$((c3 +1))

fi

c2=c1

done

I'm running on HP-UX B11.11

is Xppt2activecount_ManagedXpptServer.log a very large file?

ASKER

Generally about 50kbytes to 85kbytes with about 950 to 1980 lines

ASKER

I'll need to run the script continuously using nohup in background
to check the last line of that Xppt2activecount_ManagedXpptServer.log
file to get the count & if it exceeds certain value, I have to alert.

After going thru the historical data, I can see that this
Xppt2activecount_ManagedXpptServer.log can be updated
every 8 seconds or even 60 seconds

ASKER

Extracted below a few lines from that count log file:

2011-03-09 23:55:40,886;ConcurrSessCnt=37
2011-03-09 23:56:08,498;ConcurrSessCnt=37
2011-03-09 23:57:54,480;ConcurrSessCnt=36
2011-03-09 23:57:59,166;ConcurrSessCnt=37
2011-03-09 23:58:02,435;ConcurrSessCnt=37
2011-03-09 23:58:09,584;ConcurrSessCnt=37
2011-03-09 23:58:40,996;ConcurrSessCnt=38
2011-03-09 23:59:30,815;ConcurrSessCnt=34
2011-03-09 23:59:39,885;ConcurrSessCnt=35

The time figures you posted aren't possible for the code you posted as the code has an infinite loop.

What conditions are you using the time the script?

BTW, easier to read with proper indentation and use the "Code" formatting in EE, eg:

ASKER

c2=5
c3=1

while [ 1 ]
do
  c1=`tail -1 /app/bea/yyy/eservices/XXXX/countlog/Xppt2activecount_ManagedXpptServer.log | sed 's/=/ /' | awk '{print ($3)}'`
  # echo $c1
   # compare c1 and c2 so tt if same as last value, we dont alert again
   if [ "$c1" -ge "250" ] && [ "$c1" -ne "$c2" ]
   then
     cd /app/bea/yyy/eservices/XXXX/countlog
     tail -5 Xppt2activecount_ManagedXpptServer.log |sed s/ConcurrSessCnt/Cnt/ | ux2dos > /home/myuser/Xppt.txt
     echo $c1 > /home/myuser/Xpptcnt.dat  # this file is for SMS & Xppt.txt above is for email

   if [ "$c3" -eq "1" ]
   then
     DT1=$(date +%d%H%M)
   fi

   if [ "$c3" -eq "5" ]
   then
     DT5=$(date +%d%H%M)
   fi

   chmod 777 /home/myuser/Xpptcnt.dat
   chmod 777 /home/myuser/Xppt.txt
   /var/tmp/Xppt/sendmlcnt.pl

   sleep 9  # about 3 times the frequency of job in emsyyy01
   rm -f /home/myuser/Xppt.txt
   rm -f /home/myuser/Xpptcnt.dat

   # if  clauses below r to ensure we dont get too many emails/SMS ie
   #  pause for 2 mins if get 3 consecutive alerts within approx 1 min or so
     if [ "$c3" -ge "3" ]
       then
          if [ "$DT1" -eq "$DT5" ]
          then
             sleep 120
             c3=0
          fi
     fi

   c3=$((c3 +1))

   fi

   c2=c1

done

Open in new window

Inserted the script using EE "Code" format.

> What conditions are you using the time the script?
I login as root when the server's CPU utilization is about
50-80%.

Not sure if I've replied to what you wanted

What I meant was that your script has an infinite loop, so it would only finish if you interrupted it, so I have no idea how you got the original timings.

Additionally, your script has various conditions which will affect the timings and the amount of CPU consumed, so without knowing what data you ran your test with, it's hard to say what can be improved.

ASKER CERTIFIED SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Hi Tintin,

>What I meant was that your script has an infinite loop, so it would only finish
>if you interrupted it, so I have no idea how you got the original timings.
Yes, I used a watched to time it for 10 secs & then press Ctrl-C to interrupt it

Hi tfewster,

So your code would be

while [ 1 ]
do
c1=`tail -1 /app/bea/yyy/eservices/XXXX/countlog/Xppt2activecount_ManagedXpptServer.log | sed 's/=/ /' | awk '{print ($3)}'`

# "if" statement taken out, as it's only for when thing go wrong ;-)

c2=c1; sleep 1

done

Hi Tomunique,

I'll test out your code. One clarification on your codes :
Wouldn't the "-f" cause tail to read the end of the file
continuously ?
tail -1 -f Server.log |awk -F= '{print $2}'| while read c1

Hi sunhux - Yes, that's what I meant [I just removed the "if" statements to make it easier to read the logic of the script], but Tomunique's suggestion of `tail -f` is far better, even if you can't use "-1" as well.

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Thanks for clarifying tfewster & Tommunique.

Will try out Tommunique's method tomorrow morning.

ASKER

Hi Tommunique,

I ran 2 sessions at the same time & when the value changes (as
shown in Session 1, value changes fr 47 to 48), the codes in
Session 2 (which closely match your codes) did not execute :

Session 1:
========
cd /app/bea/yyy/eservices/XXXX/countlog
tail -1 -f xxxServer.log
2011-04-01 10:52:21,058;ConcurrSessCnt=47
2011-04-01 10:53:00,026;ConcurrSessCnt=48

Session 2:
========
cd /app/bea/yyy/eservices/XXXX/countlog
tail -1 -f xxxServer.log |awk -F= '{print $2}'| while read c1
do
echo $c1
done

What did I miss?

ASKER

To clarify myself, for Session 2,
the line "echo $c1 "
did not echo out the output

ASKER

& I noticed Session 2 can't be interrupted by Ctrl-C while Session 1 can be interrupted.

Also, the following command would regularly output something to screen, so
I guess the " | while read c1 " portion has an issue

# tail -1 -f ippt2activecount_ManagedIpptServer.log | awk -F= '{print $2}'
34
36
38

Hmm, we have an 11.11 system I can test on, maybe there's something about hpux's implementation. Let me look

ASKER

Hi Tomunique,

Have you got any update on testing it on your HP-UX B11.11 ?

It may be an HP-UX implementation issue with `tail`; You could try
tail -n -1 -f
or just
tail -f # The code will process the last few lines of the file correctly, one by one

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

correction to comment above...
first sentence should read:
the problem would NOT be tail if his testing shows that it does return data when updated.

ASKER

> Are you saying it didn't echo ANYTHING?
It didn't echo anything.

On my HP-UX, the following code echoed the appended :
# tail -1 -f ippt2activecount_ManagedIpptServer.log | awk -F= '{print $2}'

but the following code did not echo anything & just 'hanged' there (& can't
be interrupted with Ctrl-C ) :
tail -1 -f ippt2activecount_ManagedIpptServer.log | awk -F= '{print $2}' | while read c1
do
echo $c1
done

I'll check for hidden characters with "cat -ev" (or rather if the TERM variable is set
correctly) but the above codes are all typed on the terminal directly, not run from
a script. I'll run from a script tomorrow to see if it makes any difference

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Thanks very much.

The app is written by a remote programmer, sort of weblogic related.

I'll test it tomorrow as I don't have remote/VPN access to the HP-UX server.

ASKER

I've checked with "ls -i -l xxxServer.log" & noted that the
first column (inode) did not change despite that I appened
additional line of data to it.

I tried with another input log file (apend.txt), just in case the app created
'weird' problem with that logfile but I still get the same issue
of "hanged" & nothing was echoed :

# ls -i -l /tmp/apend.txt
329 -rw-r--r-- 1 wlsuser wlsgroup 42 Apr 4 10:31 /tmp/apend.txt
# tail -1 xxxServer.log >> /tmp/apend.txt
# ls -i -l /tmp/apend.txt
329 -rw-r--r-- 1 wlsuser wlsgroup 84 Apr 4 11:29 /tmp/apend.txt

Appears to hv to do with "while" but strange that using
cat with while is Ok :

cat apend.txt | while read c1
do
echo $c1
sleep 5
done

echoes the output below:
2011-04-04 10:31:03,710;ConcurrSessCnt=57
2011-04-04 11:28:46,792;ConcurrSessCnt=75
2011-04-04 11:31:13,334;ConcurrSessCnt=71
2011-04-04 11:34:44,137;ConcurrSessCnt=75
2011-04-04 11:34:44,137;ConcurrSessCnt=75
2011-04-04 11:34:44,137;ConcurrSessCnt=75

but additional lines that I appended to apend.txt
are not echoed

Wow..

tail -f {file} works

appending lines to that file from another session shows up in the tail -f output

cat testfile to while read loop, echoing records .... Works...

But, combining tail -f, with the while read loop produces no output, and hangs

I'm sorry.... I'm at a loss

Any chance in the world you have HP Support?
Call it in as a bug..

ASKER

Yes, sounds like a HP-UX issue. I don't have OS/software support with HP,
only hardware.

Btw, is there any way we can recode
tail -1 -f xxxServer.log | awk -F= '{print $2}' | while read c1
say using cat & while ?

ASKER

Or perhaps replace while or for loop? I'm not good at this

ASKER

I've recoded using for loop, I can interrupt (by Ctrl-C) the loop
but still no echo :

for c1 in `tail -1 -f ape.tx | awk -F= '{print $2}'`
do
echo $c1
date
done

I've tried using Bourne & Korn shells but still the same results.
C shell won't accept the syntax. I tried on another HP-UX box
(B11.23) & got the same result. So B11.11 or B11.23 same thing

What's your version of HP-UX?

#!/usr/bin/perl
use File::Tail;
my $ref=tie *FH,"File::Tail",(name=>"xxxServer.log");
$\=$/;
while( <FH> ){
print /\s(\S+)/;
}

ASKER

/tmp/rd.pl gave:

Can't locate File/Tail.pm in @INC (@INC contains: /opt/perl/lib/5.8.2/PA-RISC1.1-thread-multi /opt/perl/lib/5.8.2 /opt/perl/lib/site_perl/5.8.2/PA-RISC1.1-thread-multi /opt/perl/lib/site_perl/5.8.2 /opt/perl/lib/site_perl .) at ./rd.pl line 2.
BEGIN failed--compilation aborted at ./rd.pl line 2.

I remember there's something missing with my HP-UX Perl library.

cat /tmp/rd.pl
#!/usr/bin/perl
use File::Tail;
my $ref=tie *FH,"File::Tail",(name=>"/tmp/apend.tx");
$\=$/;
while( <FH> ){
print /\s(\S+)/;
}

ASKER

I put a "sleep 2" in my original Shell script as tfewster suggested &
now my script consumes only about 0.10% of CPU

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

With this script below :

tail -1 -f ippt2activecount_ManagedIpptServer.log | awk -F= '{print $2}' |tee yourlog| while read c1
do
echo $c1
done

"cat 1_line.txt >> yourlog" to append 1 line into the log will result in
yourlog being overwritten with that 1 line & nothing was output to
screen by the codes above

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

excellent