bowmantl
asked on
RHEL 5 auditd crashes the machine at roll over
Hello,
I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.
Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload
Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls
Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/au
That allows us to start the audit service:
service auditd start
All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.
Thank you for your time,
TLB
I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.
Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload
Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls
Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/au
That allows us to start the audit service:
service auditd start
All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.
Thank you for your time,
TLB
Have you check syslog to see if there is any explaination as to why the autofs is unmounted?
Does the log rollover kill the server even if its writing locally?
And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload
Your mounting it then starting autofs. If its mounted already then you dont need autofs. autofs usually mounts automatically, by just changing directory to the mount location.
I suspect you have a flawed automount config.
You should be able to access the share by typing:
cd /loglocation
on the local machine without the mount command. If you can not your autofs is not configured correctly.
Does the log rollover kill the server even if its writing locally?
And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload
Your mounting it then starting autofs. If its mounted already then you dont need autofs. autofs usually mounts automatically, by just changing directory to the mount location.
I suspect you have a flawed automount config.
You should be able to access the share by typing:
cd /loglocation
on the local machine without the mount command. If you can not your autofs is not configured correctly.
ASKER
Hello,
We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.
The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.
Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.
Thank you for your time,
TLB
We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.
The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.
Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.
Thank you for your time,
TLB
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
An example for using autofs -
All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).
Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable. That's when autofs comes into play.
All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).
Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable. That's when autofs comes into play.
ASKER
Hey All,
So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.
TLB
So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.
TLB
ASKER
The culmination of the comments helped reveal the problem.. This is the comment that got me barking up the right tree.
I don't think it's wise to be logging and doing rollovers like that to an autofs mount, there are good chances you will see problems similar to what you are now. IMO autofs shouldn't be used for things like that, it has it's uses but this should not be one of them.