?
Solved

AIX 5.3 - Something is clearing my error logs....

Posted on 2009-02-17
19
Medium Priority
?
1,161 Views
Last Modified: 2013-11-17
Something is clearing my error logs.  I know it's something that my vendor did, but I would like to see for myself where this can be found and how I can undo it.  When I run:  errpt |pg   I get nothing.  When I run:  /usr/lib/errdemon   it says it's already running.  Can someone help me find where it could be scripted to clear the error logs?  I am still learning AIX so I haven't done a lot of advanced things with it.  Thanks!
0
Comment
Question by:cansib
  • 9
  • 8
  • 2
19 Comments
 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 200 total points
ID: 23661138
run crontab -l and see if there is a job that is clearing your logs
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 23661147
Hi,
the only 'allowed' method to clear the errorlog is using the 'errclear' program.
Basically, you tell errclear how many days of data to leave in the log and which types of records to delete, which means,
errclear 0
will delete everything. Use man errclear to see more.
Seems your log gets cleared very frequently, so I would have a look at root's crontab -
crontab -l | grep errclear  (as user root, because only root is allowed to use errclear)
You should find the standard AIX entries, which normally read
0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90
which means 'clear Software and errlOgger-generated errors older than 30 days, clear Hardware errors older than 90 days.
If you find other values, especially for the retention days settings, or additional errclear entries, you have found it.
-----------
To test whether your logging is working at all, use
errlogger "This is a test"
then use
errpt
to see if it's there.
 
wmp
 
 

 
 
0
 

Author Comment

by:cansib
ID: 23661423
This is crazy.  In the crontab, I only found the standard entries:

0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90

Then, when I ran errlogger "This is a test" and then used errpt, there was still nothing.  Is my error logging corrupted?  Can I rebuild it?  Thanks!

Mark
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 68

Expert Comment

by:woolmilkporc
ID: 23661568
1)
ps -ef | grep errdemon
do you find a running process /usr/lib/errdemon ?
Issue
/usr/lib/errstop
then
/usr/lib/errdemon
and test anew.
Have some meetings now, will be back in ca. 2 hrs.
wmp
 
0
 

Author Comment

by:cansib
ID: 23661634
Here's the output from the first command:

idxhost:root:/ =>ps -ef | grep errdemon
    root  6994     1   0   Feb 14      -  0:00 /usr/lib/errdemon
    root 30738 26460   0 08:49:48 pts/23  0:00 grep errdemon

and here's what happened with the next 2 commands:

idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>/usr/lib/errdemon
idxhost:root:/ =>errlogger "This is a test"
idxhost:root:/ =>errpt
idxhost:root:/ =>

Strange, huh?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 23662241
So, please repeat the errstop, then look with ps if errdemon is running nevertheless.
If yes, terminate it with kill -9 [pid] and see if it vanishes.
If yes, issue the /usr/lib/errdemon again and test.

Look at /var/adm/ras for the files errlog and errtmplt.
errlog must be writeable for user root and group system
errtmplt must be writeable for root and have a minimum size of 250-300 K.

If errlog is not there, do
touch /var/adm/ras/errlog, chown root:system /var/adm/ras/errlog, chmod 664 /var/adm/ras/errlog

I'll do some research in the meantime.

wmp





0
 

Author Comment

by:cansib
ID: 23662596
Thank you!  I have to make a run offsite real quick, but I will post back my results.  Thank you so much for helping!  I really appreciate it.

Mark
0
 

Author Comment

by:cansib
ID: 23665484
I ran the errstop, then the ps command, here is the output from that:

idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>ps -ef | grep errdemon
    root 19650 34522   0 15:32:57  pts/9  0:00 grep errdemon

Does that mean it's still running?  Thanks!
0
 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 200 total points
ID: 23666435
no, it is better if you do it like this:

/usr/lib/errstop
ps -ef | grep -v grep | grep errdemon

the line you see is the grep itself
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 23668508
OK, omarfarid is right.
errdemon is not running anymore. Now start it using /usr/lib/errdemon and test.
If it still doesn't work, please examine /var/adm/ras as I suggested above.
wmp
 
 
0
 

Author Comment

by:cansib
ID: 23671155
I tested and it's still not logging anything.

Here is what I found on the 2 log files:

-rw-rw-r--   1 root     system       104218 Feb 18 07:10 errlog
-rw-r--r--   1 root     system       241805 Mar 07 2007  errtmplt

Thanks!
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 23671868
Some sort of 'hard' method -
1) /usr/lib/errstop
2) rm /var/adm/ras/errlog
3) /usr/lib/errdemon
4) errpt
You should see
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
9DBCFDEE   0218172709 T O errdemon       ERROR LOGGING TURNED ON
If not, I fear I will be out of ideas in a while ...
 
0
 

Author Comment

by:cansib
ID: 23673361
I tried that and still no luck.  Is it possible that the error log entries are somehow being redirected?
0
 
LVL 68

Assisted Solution

by:woolmilkporc
woolmilkporc earned 1800 total points
ID: 23673883
OK,
some more things to test:

-  After having issued

1) /usr/lib/errstop2) rm /var/adm/ras/errlog3) /usr/lib/errdemon

/var/adm/ras/errlog must be exactly 8192 bytes in size. Please verify by issuing ls -l /var/adm/ras/errlog

-  With errdemon running, issue  fuser /dev/error . You should see

/dev/error:   [some_process-id]

Now issue ps -ef | grep [some_process-id] with the process-id from fuser.
Do you see /usr/lib/errdemon as the command? If not, what is it?
Or does fuser give more than one pid? If yes, test the other pids with  ps , too.
What do you see?

-  Issue  alias errpt  in your rootshell
You should see something like  errpt: ksh alias not found
If not, what do you see?

- Issue ls -l /usr/lib/errdemon , ls -l /usr/lib/errstop
You should see regular files, and not links ( a  '->' followed by a path after the filename)
If you see a link, where does it point to?
errdemon should be roundabout 100K in size, errstop 12 K. Yes?

All this sounds like paranoia, but who knows ...

wmp




0
 
LVL 68

Assisted Solution

by:woolmilkporc
woolmilkporc earned 1800 total points
ID: 23673911
... additionally: fuser /dev/errorctl
Same pid as with  fuser /dev/error ?
0
 
LVL 68

Assisted Solution

by:woolmilkporc
woolmilkporc earned 1800 total points
ID: 23678920
... and finally -
do a 'which errpt'
Is it really /usr/bin/errpt ?
0
 

Author Comment

by:cansib
ID: 24006102
Someone from our vendor support remoted in and fixed this without my knowledge.  In fact, I don't even know for sure if it was them, but I'm thinking, who else could it be.  I did an "esc + k" and saw a back log of commands that I didn't run that all were related to the errdemon and errpt.  So it's working now, I just don't know who fixed it.  Thanks for the help though and sorry for the delay in getting back to this issue.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 24006203
Hi again,
glad to hear that it works now. But too bad that we don't know why! Is there really no chance to ask someone from your vendor's support people what they did? The answer might help other people, too!
wmp
P.S. What commands did you see with esc-k?
0
 

Accepted Solution

by:
cansib earned 0 total points
ID: 24117775
Here are the commands.  It appears that the actual file "errpt" was corrupted.


idxhost:root:/ =>ps -ef | grep -v grep | grep errdemon
idxhost:root:/ =>errpt
idxhost:root:/ =>cd var
idxhost:root:/ =>cd adm
idxhost:root:/ =>cd ras
idxhost:root:/ =>ls -l |pg
idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>rm /var/adm/ras/errlog
idxhost:root:/ =>/usr/lib/errdemon
idxhost:root:/ =>errpt
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>ls |pg
idxhost:root:/ =>TERM=vt100;export TERM
idxhost:root:/ =>errlogger "Test"
idxhost:root:/ =>errpt
idxhost:root:/ =>TERM=vt100;export TERM
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>l errlog
idxhost:root:/ =>chmod 0644 errlog
idxhost:root:/ =>l /usr/bin/errpt
idxhost:root:/ =>chmod 04555 /usr/bin/errpt
idxhost:root:/ =>errpt -a
idxhost:root:/ =>l errlog
idxhost:root:/ =>ps -ef|grep errd
idxhost:root:/ =>errpt -t
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>l er*
idxhost:root:/ =>pg devinst.log






0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month14 days, 7 hours left to enroll

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question