Solved

Server stopped working all of the sudden, possible attack?

Posted on 2004-08-18
7
314 Views
Last Modified: 2010-04-20
Hello All,

We have a server running with RHE 3, Apache, Php, Mysql and Bind. It's our main production server.

Now the problem we faced day before was something very weird. All of the sudden all the services on the server stopped responding, we couldn't ssh into the server, we couldn't see the sites, couldn't fetch mails and so on.

We could ping the server though, ping was going fine, but we just couldn't access the server in any way.

Now the question I would like to ask you guys is,  if such problem occurs again, how to trace it. I tried to check /var/log/messages but all I found were portsentry's messages.

If we were under attack, or if the server was hacked, how do we trace it? Are there any guidelines or links through which we can possibly know what cause our server to respond all of the sudden?


Thanks in Advance.
0
Comment
Question by:manojoswal
7 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 11832964
> but we just couldn't access the server in any way.

Was the console still responsive? From the console could you initiate a network connection out to some other box (e.g., ssh some-local-host)? What was the load average on the box?

Is the server up to date (as in running up2date and allowing it to update the kernel)?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 150 total points
ID: 11835251
I recommend having a look at tripwire:

http://sourceforge.net/projects/tripwire/

It enables you to setup up an index of all the important files on your system, and if anyone messes with them you are notified. You can check the integrity of your files at any time that you want. This will give you the confidence to know that you javen't been hacked.

As for tracing the fault. First you need to check whether you can access the PC in anyway at all. Hopefully you should be able  to access at least a login prompt from the console?

If you can, then you need to try:

ps -ef               What processes are running
top                  Is anything using up the processor time


You may find that killing off/restarting certain rogue processes solves the problem, in which case you will know where to look next time.

You've done the obvious thing of looking in the logs. If there is nothing there, then you may need to increase the amount of logging that occurs. Linux software often has a -v (verbose), or -d  (debugging) mode.

It's also woth learning a little about klogd (the kernel logging daemon):

 http://linux.about.com/library/cmd/blcmdl8_klogd.htm

HTH:)
0
 
LVL 1

Assisted Solution

by:bestondoa
bestondoa earned 150 total points
ID: 11843279
Hello,

You should run a check on your systemany "rootkit". Rootkits are a collection of modified program sources or binaries which replace a selection of system binaries. For example, an hacker would enter your system and replace \bin\login with a modified one that has a secret username/password that will allways let him enter as root.

Go to http://www.chkrootkit.org/ and download chkrootkit. chkrootkit localy scans your system binaries for rootkit modification.

Cheers,

Bestondoa

0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 

Author Comment

by:manojoswal
ID: 11843478
1. We then got the server rebooted, since then its running fine.
2. We keep a good vigil on it, till the moment it stopped responding, we had adequate memory and CPU free. (its a dual xeon with 1 gb ram, SCSI and RAID) so the process that caused it was a sudden one.
3. There are no signs of DDOS or DOS.
4. We need to know exactly how we could go about finding the cause of this failure. What logs, what we should look for in it.
5. We have trip wire, it shows really nothing serious.

are there some links or guidlines somewhere that will guide us to track activity and find all possible traces of the cause of the problem.

regards

manoj



0
 

Author Comment

by:manojoswal
ID: 11843518
we need to report to the client the cause of this failure, any guidelines or papers on the net that have clues to foresnic analsysis of an attack.

regards

manoj oswal
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 100 total points
ID: 11843704
Is the server (and the kernel) up to date w/respect to the RHEL errata?

Is the BIOS on the system current?

What is the typical and peak load average?
0
 
LVL 1

Assisted Solution

by:LieutenantLefsa
LieutenantLefsa earned 100 total points
ID: 12171255
Did any of the disks fill up?
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question