Solved

RHEL 5 auditd crashes the machine at roll over

Posted on 2011-09-26
7
597 Views
Last Modified: 2012-06-27
Hello,

I have a RHEL 5.4 server that has an autofs mount to a nfs share to store the audit logs. On this server when the logs do roll over it drops the nfs mount and then starts buffering the logs locally. The buffer fills and then the server locks up and is not accessible via ssh or local login.

Rebooting the machine requires the following actions to be completed before auditd can start again:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Then we check to make sure that the loglocation is populated with the folder name of the server
cd /loglocation:
ls

Then we have to change the permissions of the audit.log file to allow for the machine to write to them again:
chmod u+w /loglocation/servername/audit/audit.log

That allows us to start the audit service:
service auditd start


All actions are done as root or sudo, this will allow the logs to work properly until the next roll over. We have not been able to find a solution that keeps the nfs mount active after a roll over.

Thank you for your time,
TLB
0
Comment
Question by:bowmantl
  • 3
  • 3
7 Comments
 
LVL 21

Expert Comment

by:Papertrip
ID: 36602509
Is there a specific reason you are doing this with autofs instead of just making an entry in fstab?

I don't think it's wise to be logging and doing rollovers like that to an autofs mount, there are good chances you will see problems similar to what you are now.  IMO autofs shouldn't be used for things like that, it has it's uses but this should not be one of them.
0
 
LVL 23

Expert Comment

by:savone
ID: 36602573
Have you check syslog to see if there is any explaination as to why the autofs is unmounted?  

Does the log rollover kill the server even if its writing locally?

And this is throwing me off a little:
mount logserver:/loglocation /loglocation
service autofs restart
service autofs reload

Your mounting it then starting autofs.  If its mounted already then you dont need autofs.  autofs usually mounts automatically, by just changing directory to the mount location.

I suspect you have a flawed automount config.

You should be able to access the share by typing:

cd /loglocation

on the local machine without the mount command.  If you can not your autofs is not configured correctly.
0
 

Author Comment

by:bowmantl
ID: 36646191
Hello,

We have 100 machines connecting to the log server and knew that there would be a high I/O and may take a large amount of bandwidth if we had the fstab mounts. We thought that if we went with autofs that we would save on both.

The mount command was being used to verify that the directory was mounting correctly and then doing a service autofs status showed that the service was stopped. That was the reason for doing the restart then reload. Reviewing the autofs configuration we noticed that the configuration was the same as the other autofs mounts that we have on the machines.

Is autofs not meant for this high of traffic or is it that it is not meant for auditd? We can move the audit traffic over its own network if the fstab option is the most reliable. We would just prefer to not run an additional line to each of the machines.

Thank you for your time,
TLB
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 21

Accepted Solution

by:
Papertrip earned 500 total points
ID: 36650109
I'm not following you on the I/O and bandwidth comment.  Any I/O being done on the machine serving the NFS share is not going to affect performance of the server mounting it, except for possibly I/O wait due to the NFS server being overloaded, but that would happen via fstab as well.  Autofs entries and fstab entries end up doing the same thing in the end -- mounting a disk/share/etc.  How and when they are mounted is the only real difference.

It's not that autofs isn't meant for high traffic or not meant for auditd, it's that it shouldn't be used for what you are trying to do.  In the Unix world there are often several ways to do the same thing -- not all are equal.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36652226
An example for using autofs -

All of your homedirs are exported via NFS.
You want any homedir to mount when accessed, like /home/user
You don't need them to be mounted all the time (not all users will be on all boxes at all times).

Think about if you had 1000 users -- dynamically managing fstab for all your servers to statically mount each users homedir is just not scalable.  That's when autofs comes into play.
0
 

Author Comment

by:bowmantl
ID: 36895259
Hey All,

So this was a sneaky problem, but Papertrip got me going in the right direction. The problem ended up being the security settings we have in our environment coupled with the share server being overloaded. We reworked the script to keep the active logs local and push the rolled over logs to the NFS mount which doesn't trigger our Linux crash on audit fail equivalent.

TLB
0
 

Author Closing Comment

by:bowmantl
ID: 36895270
The culmination of the comments helped  reveal the problem.. This is the comment that got me barking up the right tree.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Why VNC doesn't work in Redhat? 11 45
Automating a script for user accounts LINUX 14 70
linux installs 6 49
Exchange 2010 unable to mount DB due to corruption 11 71
Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now