Linux server checklist

WINAIRMVS
WINAIRMVS used Ask the Experts™
on
We are developing some process flow and routine tasks for our linux server environment, and need some insightful advice on what we should include in a daily, weekly, monthly, and quarterly server checklist.  We currently run all Red Hat servers ranging from version 3 to 5.  We have an apache web server, sendmail server, a number of other linux boxes doing backups and pushing configurations, and a number of boxes that are used for monitoring.  Any suggestions are welcome.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Some additional context would be useful.

What is the purpose of this checklist:  Initial configuration?   Security?  Configuration?  Log review and auditing?  Ensure that the machine is not broken?  Maintenance?  General systems-management discipline?
President
Commented:
Hi,

I'll suggest you to use a tool such as Nagios t automate the task of monitoring:

- Critical Services,
- Volume Space Monitoring,
- Number of Tasks Running
- CPU Usage,
- Network Connectivity issues.

- You can start to use SVN for backing up critical System-Configuration info.
- You can set-up a backup solution to backup all your ciritical system data regularly.

This will also help you to create statistics on outages etc. After you've accomplish these tasks it comes to regular audits for the system. During these audits you can monitor:

- If all system patches are applied (through rpm and yum)
- Any old stale filesets are still on disk (rpm)
- If there are any unnecessary stale accounts on system (/etc/root, /etc/group)
- If there are any unnecessary services are running over the system (ps, netstat)
- Check IPTables if it protects adequately all the necessary ports and it does not have any unnecsary ports open.
- Check applications such as Databases for access and the rights for access is up to the need of the running software and not more.
- You can check for any rootkits on the system. (chkrootkit)
- Then you can use a tool such as nessus to determine if your servers are running any vulnerable services.

I guess this will be enough for biannual audits.
 
Cheers,
K.

Author

Commented:
"LukeScharf:
Some additional context would be useful.

What is the purpose of this checklist:  Initial configuration?   Security?  Configuration?  Log review and auditing?  Ensure that the machine is not broken?  Maintenance?  General systems-management discipline?"

Luke,

This is for servers that are already in production and are running the services that I specified.  This checklist would be for the route tasks that would help us in detecting problems, and preventing them.

Author

Commented:
KeremE,

Thanks for the good list, but we are looking for more specific tasks we can do or script for each individual system type, i.e. what tasks would you run on a sendmail server to keep the sendmail process and its dependancies running smoothly, on apache which commands would your run in a script to check consistenency such as httpd -t or httpd -S.
WINAIRMVS,

I second the suggestion of Naggios.  You can have it make sure the services are running every couple of minutes.

Well-written services don't require periodic cleanup, and there isn't a great analog to defragmenting disks or anything like that on Linux.  The main thing is to know if your service (or the box that hosts it) has crashed, to make sure that it hasn't run out of disk space, and is only running what you installed on it.  Of course, there are applications that require a little extra coddling, but they would be much more specialized than Apache or Sendmail.  (Perhaps a PHP application that you're hosting with Apache would require some extra daily maintenance.)

Here are a few suggestions:
  1. Secure the machine appropriately. This means shutting off extra services, iptables firewalls, running ssh on alternate ports, and possibly something like denyhosts are all helpful -- but there are a lot of good guides out there for this.
  2. Patch patch patch!  The process varies depending on your distribution (up2date for RHEL, yum for Centos, apt-get for Debian/Ubuntu, emerge for Gentoo, and so on).  Anything that you can do to make this more regular is usually a win.  I have a script called "patch_everything.sh" that logs in to all of my VMs, servers, and clients, determines which tools are installed (yum or apt-get in my case), and runs the appropriate update.  That way, I can just kick off one command each morning and watch it carefully as it walks through dozens of machines.  Fully automatic updates are good if you're forgetful, but bad if you run highly customized systems.  (My systems automatically download updates and then e-mail me.)
  3. Run logwatch, or something like it.  Logwatch will e-mail you a summary of the syslog activity each day.  Make reading them a part of your morning routine.  The report includes unrecognized log entries, and also a report of the disk space.
  4. An automated script to ensure that this is all set up to spec wouldn't be a bad idea, tough I'd rather integrate it in to Naggios or Big Brother than run it manually.
  5. Run "netstat", "nmap", and "ps" on the machine every once in a while, to make sure that only what you expect to be running is running.
If you want to know about the automatically-configured periodic maintenance,  you can dump the contents root's crontab ("crontab -l -u root")  and also look in the /etc/cron.d/ & /etc/cron.daily/ directories.  This should include things like log rotation and updating the locate database.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial