Solved

RHEL 5 auditd crashes the machine at roll over

Posted on 2011-09-26
7
595 Views
Last Modified: 2012-06-27
Hello,

I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.

Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls

Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/audit/audit.log

That allows us to start the audit service:
service auditd start


All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.

Thank you for your time,
TLB
0
Comment
Question by:bowmantl
  • 3
  • 3
7 Comments
 
LVL 21

Expert Comment

by:Papertrip
Comment Utility
Is there a specific reason you are doing this with autofs instead of just making an entry in fstab?

I don't think it's wise to be logging and doing rollovers like that to an autofs mount, there are good chances you will see problems similar to what you are now.  IMO autofs shouldn't be used for things like that, it has it's uses but this should not be one of them.
0
 
LVL 23

Expert Comment

by:savone
Comment Utility
Have you check syslog to see if there is any explaination as to why the autofs is unmounted?  

Does the log rollover kill the server even if its writing locally?

And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Your mounting it then starting autofs.  If its mounted already then you dont need autofs.  autofs usually mounts automatically, by just changing directory to the mount location.

I suspect you have a flawed automount config.

You should be able to access the share by typing:

cd /loglocation

on the local machine without the mount command.  If you can not your autofs is not configured correctly.
0
 

Author Comment

by:bowmantl
Comment Utility
Hello,

We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.

The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.

Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.

Thank you for your time,
TLB
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 21

Accepted Solution

by:
Papertrip earned 500 total points
Comment Utility
I'm not following you on the I/O and bandwidth comment.  Any I/O being done on the machine serving the NFS share is not going to affect performance of the server mounting it, except for possibly I/O wait due to the NFS server being overloaded, but that would happen via fstab as well.  Autofs entries and fstab entries end up doing the same thing in the end -- mounting a disk/share/etc.  How and when they are mounted is the only real difference.

It's not that autofs isn't meant for high traffic or not meant for auditd, it's that it shouldn't be used for what you are trying to do.  In the Unix world there are often several ways to do the same thing -- not all are equal.
0
 
LVL 21

Expert Comment

by:Papertrip
Comment Utility
An example for using autofs -

All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).

Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable.  That's when autofs comes into play.
0
 

Author Comment

by:bowmantl
Comment Utility
Hey All,

So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.

TLB
0
 

Author Closing Comment

by:bowmantl
Comment Utility
The culmination of the comments helped  reveal the problem.. This is the comment that got me barking up the right tree.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

I am a long time windows user and for me it is normal to have spaces in directory and file names. Changing to Linux I found myself frustrated when I moved my windows data over to my new Linux computer. The problem occurs when at the command line.…
How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now