We help IT Professionals succeed at work.

We've partnered with Certified Experts, Carl Webster and Richard Faulkner, to bring you a podcast all about Citrix Workspace, moving to the cloud, and analytics & intelligence. Episode 2 coming soon!Listen Now

x

AIX 5.3 - Something is clearing my error logs....

cansib
cansib asked
on
Medium Priority
1,313 Views
Last Modified: 2013-11-17
Something is clearing my error logs.  I know it's something that my vendor did, but I would like to see for myself where this can be found and how I can undo it.  When I run:  errpt |pg   I get nothing.  When I run:  /usr/lib/errdemon   it says it's already running.  Can someone help me find where it could be scripted to clear the error logs?  I am still learning AIX so I haven't done a lot of advanced things with it.  Thanks!
Comment
Watch Question

CERTIFIED EXPERT
Top Expert 2007
Commented:
run crontab -l and see if there is a job that is clearing your logs

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
Hi,
the only 'allowed' method to clear the errorlog is using the 'errclear' program.
Basically, you tell errclear how many days of data to leave in the log and which types of records to delete, which means,
errclear 0
will delete everything. Use man errclear to see more.
Seems your log gets cleared very frequently, so I would have a look at root's crontab -
crontab -l | grep errclear  (as user root, because only root is allowed to use errclear)
You should find the standard AIX entries, which normally read
0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90
which means 'clear Software and errlOgger-generated errors older than 30 days, clear Hardware errors older than 90 days.
If you find other values, especially for the retention days settings, or additional errclear entries, you have found it.
-----------
To test whether your logging is working at all, use
errlogger "This is a test"
then use
errpt
to see if it's there.
 
wmp
 
 

 
 

Author

Commented:
This is crazy.  In the crontab, I only found the standard entries:

0 11 * * * /usr/bin/errclear -d S,O 30
0 12 * * * /usr/bin/errclear -d H 90

Then, when I ran errlogger "This is a test" and then used errpt, there was still nothing.  Is my error logging corrupted?  Can I rebuild it?  Thanks!

Mark
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
1)
ps -ef | grep errdemon
do you find a running process /usr/lib/errdemon ?
Issue
/usr/lib/errstop
then
/usr/lib/errdemon
and test anew.
Have some meetings now, will be back in ca. 2 hrs.
wmp
 

Author

Commented:
Here's the output from the first command:

idxhost:root:/ =>ps -ef | grep errdemon
    root  6994     1   0   Feb 14      -  0:00 /usr/lib/errdemon
    root 30738 26460   0 08:49:48 pts/23  0:00 grep errdemon

and here's what happened with the next 2 commands:

idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>/usr/lib/errdemon
idxhost:root:/ =>errlogger "This is a test"
idxhost:root:/ =>errpt
idxhost:root:/ =>

Strange, huh?
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
So, please repeat the errstop, then look with ps if errdemon is running nevertheless.
If yes, terminate it with kill -9 [pid] and see if it vanishes.
If yes, issue the /usr/lib/errdemon again and test.

Look at /var/adm/ras for the files errlog and errtmplt.
errlog must be writeable for user root and group system
errtmplt must be writeable for root and have a minimum size of 250-300 K.

If errlog is not there, do
touch /var/adm/ras/errlog, chown root:system /var/adm/ras/errlog, chmod 664 /var/adm/ras/errlog

I'll do some research in the meantime.

wmp





Author

Commented:
Thank you!  I have to make a run offsite real quick, but I will post back my results.  Thank you so much for helping!  I really appreciate it.

Mark

Author

Commented:
I ran the errstop, then the ps command, here is the output from that:

idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>ps -ef | grep errdemon
    root 19650 34522   0 15:32:57  pts/9  0:00 grep errdemon

Does that mean it's still running?  Thanks!
CERTIFIED EXPERT
Top Expert 2007
Commented:
no, it is better if you do it like this:

/usr/lib/errstop
ps -ef | grep -v grep | grep errdemon

the line you see is the grep itself
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
OK, omarfarid is right.
errdemon is not running anymore. Now start it using /usr/lib/errdemon and test.
If it still doesn't work, please examine /var/adm/ras as I suggested above.
wmp
 
 

Author

Commented:
I tested and it's still not logging anything.

Here is what I found on the 2 log files:

-rw-rw-r--   1 root     system       104218 Feb 18 07:10 errlog
-rw-r--r--   1 root     system       241805 Mar 07 2007  errtmplt

Thanks!
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
Some sort of 'hard' method -
1) /usr/lib/errstop
2) rm /var/adm/ras/errlog
3) /usr/lib/errdemon
4) errpt
You should see
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
9DBCFDEE   0218172709 T O errdemon       ERROR LOGGING TURNED ON
If not, I fear I will be out of ideas in a while ...
 

Author

Commented:
I tried that and still no luck.  Is it possible that the error log entries are somehow being redirected?
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013
Commented:
OK,
some more things to test:

-  After having issued

1) /usr/lib/errstop2) rm /var/adm/ras/errlog3) /usr/lib/errdemon

/var/adm/ras/errlog must be exactly 8192 bytes in size. Please verify by issuing ls -l /var/adm/ras/errlog

-  With errdemon running, issue  fuser /dev/error . You should see

/dev/error:   [some_process-id]

Now issue ps -ef | grep [some_process-id] with the process-id from fuser.
Do you see /usr/lib/errdemon as the command? If not, what is it?
Or does fuser give more than one pid? If yes, test the other pids with  ps , too.
What do you see?

-  Issue  alias errpt  in your rootshell
You should see something like  errpt: ksh alias not found
If not, what do you see?

- Issue ls -l /usr/lib/errdemon , ls -l /usr/lib/errstop
You should see regular files, and not links ( a  '->' followed by a path after the filename)
If you see a link, where does it point to?
errdemon should be roundabout 100K in size, errstop 12 K. Yes?

All this sounds like paranoia, but who knows ...

wmp




CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013
Commented:
... additionally: fuser /dev/errorctl
Same pid as with  fuser /dev/error ?
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013
Commented:
... and finally -
do a 'which errpt'
Is it really /usr/bin/errpt ?

Author

Commented:
Someone from our vendor support remoted in and fixed this without my knowledge.  In fact, I don't even know for sure if it was them, but I'm thinking, who else could it be.  I did an "esc + k" and saw a back log of commands that I didn't run that all were related to the errdemon and errpt.  So it's working now, I just don't know who fixed it.  Thanks for the help though and sorry for the delay in getting back to this issue.
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
Hi again,
glad to hear that it works now. But too bad that we don't know why! Is there really no chance to ask someone from your vendor's support people what they did? The answer might help other people, too!
wmp
P.S. What commands did you see with esc-k?
Commented:
Here are the commands.  It appears that the actual file "errpt" was corrupted.


idxhost:root:/ =>ps -ef | grep -v grep | grep errdemon
idxhost:root:/ =>errpt
idxhost:root:/ =>cd var
idxhost:root:/ =>cd adm
idxhost:root:/ =>cd ras
idxhost:root:/ =>ls -l |pg
idxhost:root:/ =>/usr/lib/errstop
idxhost:root:/ =>rm /var/adm/ras/errlog
idxhost:root:/ =>/usr/lib/errdemon
idxhost:root:/ =>errpt
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>ls |pg
idxhost:root:/ =>TERM=vt100;export TERM
idxhost:root:/ =>errlogger "Test"
idxhost:root:/ =>errpt
idxhost:root:/ =>TERM=vt100;export TERM
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>l errlog
idxhost:root:/ =>chmod 0644 errlog
idxhost:root:/ =>l /usr/bin/errpt
idxhost:root:/ =>chmod 04555 /usr/bin/errpt
idxhost:root:/ =>errpt -a
idxhost:root:/ =>l errlog
idxhost:root:/ =>ps -ef|grep errd
idxhost:root:/ =>errpt -t
idxhost:root:/ =>cd /var/adm/ras
idxhost:root:/ =>l er*
idxhost:root:/ =>pg devinst.log






Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.