Solved

RHEL 5 auditd crashes the machine at roll over

Posted on 2011-09-26
7
600 Views
Last Modified: 2012-06-27
Hello,

I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.

Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls

Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/audit/audit.log

That allows us to start the audit service:
service auditd start


All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.

Thank you for your time,
TLB
0
Comment
Question by:bowmantl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
7 Comments
 
LVL 21

Expert Comment

by:Papertrip
ID: 36602509
Is there a specific reason you are doing this with autofs instead of just making an entry in fstab?

I don't think it's wise to be logging and doing rollovers like that to an autofs mount, there are good chances you will see problems similar to what you are now.  IMO autofs shouldn't be used for things like that, it has it's uses but this should not be one of them.
0
 
LVL 23

Expert Comment

by:savone
ID: 36602573
Have you check syslog to see if there is any explaination as to why the autofs is unmounted?  

Does the log rollover kill the server even if its writing locally?

And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Your mounting it then starting autofs.  If its mounted already then you dont need autofs.  autofs usually mounts automatically, by just changing directory to the mount location.

I suspect you have a flawed automount config.

You should be able to access the share by typing:

cd /loglocation

on the local machine without the mount command.  If you can not your autofs is not configured correctly.
0
 

Author Comment

by:bowmantl
ID: 36646191
Hello,

We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.

The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.

Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.

Thank you for your time,
TLB
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 21

Accepted Solution

by:
Papertrip earned 500 total points
ID: 36650109
I'm not following you on the I/O and bandwidth comment.  Any I/O being done on the machine serving the NFS share is not going to affect performance of the server mounting it, except for possibly I/O wait due to the NFS server being overloaded, but that would happen via fstab as well.  Autofs entries and fstab entries end up doing the same thing in the end -- mounting a disk/share/etc.  How and when they are mounted is the only real difference.

It's not that autofs isn't meant for high traffic or not meant for auditd, it's that it shouldn't be used for what you are trying to do.  In the Unix world there are often several ways to do the same thing -- not all are equal.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36652226
An example for using autofs -

All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).

Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable.  That's when autofs comes into play.
0
 

Author Comment

by:bowmantl
ID: 36895259
Hey All,

So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.

TLB
0
 

Author Closing Comment

by:bowmantl
ID: 36895270
The culmination of the comments helped  reveal the problem.. This is the comment that got me barking up the right tree.
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
​Being a Managed Services Provider (MSP) has presented you  with challenges in the past— and by meeting those challenges you’ve reaped the rewards of success.  In 2014, challenges and rewards remain; but as the Internet and business environment evol…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question