Server stopped working all of the sudden, possible attack?

Hello All,

We have a server running with RHE 3, Apache, Php, Mysql and Bind. It's our main production server.

Now the problem we faced day before was something very weird. All of the sudden all the services on the server stopped responding, we couldn't ssh into the server, we couldn't see the sites, couldn't fetch mails and so on.

We could ping the server though, ping was going fine, but we just couldn't access the server in any way.

Now the question I would like to ask you guys is,  if such problem occurs again, how to trace it. I tried to check /var/log/messages but all I found were portsentry's messages.

If we were under attack, or if the server was hacked, how do we trace it? Are there any guidelines or links through which we can possibly know what cause our server to respond all of the sudden?


Thanks in Advance.
manojoswalAsked:
Who is Participating?
 
pjedmondConnect With a Mentor Commented:
I recommend having a look at tripwire:

http://sourceforge.net/projects/tripwire/

It enables you to setup up an index of all the important files on your system, and if anyone messes with them you are notified. You can check the integrity of your files at any time that you want. This will give you the confidence to know that you javen't been hacked.

As for tracing the fault. First you need to check whether you can access the PC in anyway at all. Hopefully you should be able  to access at least a login prompt from the console?

If you can, then you need to try:

ps -ef               What processes are running
top                  Is anything using up the processor time


You may find that killing off/restarting certain rogue processes solves the problem, in which case you will know where to look next time.

You've done the obvious thing of looking in the logs. If there is nothing there, then you may need to increase the amount of logging that occurs. Linux software often has a -v (verbose), or -d  (debugging) mode.

It's also woth learning a little about klogd (the kernel logging daemon):

 http://linux.about.com/library/cmd/blcmdl8_klogd.htm

HTH:)
0
 
jlevieCommented:
> but we just couldn't access the server in any way.

Was the console still responsive? From the console could you initiate a network connection out to some other box (e.g., ssh some-local-host)? What was the load average on the box?

Is the server up to date (as in running up2date and allowing it to update the kernel)?
0
 
bestondoaConnect With a Mentor Commented:
Hello,

You should run a check on your systemany "rootkit". Rootkits are a collection of modified program sources or binaries which replace a selection of system binaries. For example, an hacker would enter your system and replace \bin\login with a modified one that has a secret username/password that will allways let him enter as root.

Go to http://www.chkrootkit.org/ and download chkrootkit. chkrootkit localy scans your system binaries for rootkit modification.

Cheers,

Bestondoa

0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

 
manojoswalAuthor Commented:
1. We then got the server rebooted, since then its running fine.
2. We keep a good vigil on it, till the moment it stopped responding, we had adequate memory and CPU free. (its a dual xeon with 1 gb ram, SCSI and RAID) so the process that caused it was a sudden one.
3. There are no signs of DDOS or DOS.
4. We need to know exactly how we could go about finding the cause of this failure. What logs, what we should look for in it.
5. We have trip wire, it shows really nothing serious.

are there some links or guidlines somewhere that will guide us to track activity and find all possible traces of the cause of the problem.

regards

manoj



0
 
manojoswalAuthor Commented:
we need to report to the client the cause of this failure, any guidelines or papers on the net that have clues to foresnic analsysis of an attack.

regards

manoj oswal
0
 
jlevieConnect With a Mentor Commented:
Is the server (and the kernel) up to date w/respect to the RHEL errata?

Is the BIOS on the system current?

What is the typical and peak load average?
0
 
LieutenantLefsaConnect With a Mentor Commented:
Did any of the disks fill up?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.