Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 610
  • Last Modified:

RHEL 5 auditd crashes the machine at roll over

Hello,

I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.

Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls

Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/audit/audit.log

That allows us to start the audit service:
service auditd start


All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.

Thank you for your time,
TLB
0
bowmantl
Asked:
bowmantl
  • 3
  • 3
1 Solution
 
PapertripCommented:
Is there a specific reason you are doing this with autofs instead of just making an entry in fstab?

I don't think it's wise to be logging and doing rollovers like that to an autofs mount, there are good chances you will see problems similar to what you are now.  IMO autofs shouldn't be used for things like that, it has it's uses but this should not be one of them.
0
 
savoneCommented:
Have you check syslog to see if there is any explaination as to why the autofs is unmounted?  

Does the log rollover kill the server even if its writing locally?

And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Your mounting it then starting autofs.  If its mounted already then you dont need autofs.  autofs usually mounts automatically, by just changing directory to the mount location.

I suspect you have a flawed automount config.

You should be able to access the share by typing:

cd /loglocation

on the local machine without the mount command.  If you can not your autofs is not configured correctly.
0
 
bowmantlAuthor Commented:
Hello,

We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.

The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.

Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.

Thank you for your time,
TLB
0
Building an Effective Phishing Protection Program

Join Director of Product Management Todd OBoyle on April 26th as he covers the key elements of a phishing protection program. Whether you’re an old hat at phishing education or considering starting a program -- we'll discuss critical components that should be in any program.

 
PapertripCommented:
I'm not following you on the I/O and bandwidth comment.  Any I/O being done on the machine serving the NFS share is not going to affect performance of the server mounting it, except for possibly I/O wait due to the NFS server being overloaded, but that would happen via fstab as well.  Autofs entries and fstab entries end up doing the same thing in the end -- mounting a disk/share/etc.  How and when they are mounted is the only real difference.

It's not that autofs isn't meant for high traffic or not meant for auditd, it's that it shouldn't be used for what you are trying to do.  In the Unix world there are often several ways to do the same thing -- not all are equal.
0
 
PapertripCommented:
An example for using autofs -

All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).

Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable.  That's when autofs comes into play.
0
 
bowmantlAuthor Commented:
Hey All,

So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.

TLB
0
 
bowmantlAuthor Commented:
The culmination of the comments helped  reveal the problem.. This is the comment that got me barking up the right tree.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

  • 3
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now