Solved

Server stopped working all of the sudden, possible attack?

Posted on 2004-08-18
7
313 Views
Last Modified: 2010-04-20
Hello All,

We have a server running with RHE 3, Apache, Php, Mysql and Bind. It's our main production server.

Now the problem we faced day before was something very weird. All of the sudden all the services on the server stopped responding, we couldn't ssh into the server, we couldn't see the sites, couldn't fetch mails and so on.

We could ping the server though, ping was going fine, but we just couldn't access the server in any way.

Now the question I would like to ask you guys is,  if such problem occurs again, how to trace it. I tried to check /var/log/messages but all I found were portsentry's messages.

If we were under attack, or if the server was hacked, how do we trace it? Are there any guidelines or links through which we can possibly know what cause our server to respond all of the sudden?


Thanks in Advance.
0
Comment
Question by:manojoswal
7 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 11832964
> but we just couldn't access the server in any way.

Was the console still responsive? From the console could you initiate a network connection out to some other box (e.g., ssh some-local-host)? What was the load average on the box?

Is the server up to date (as in running up2date and allowing it to update the kernel)?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 150 total points
ID: 11835251
I recommend having a look at tripwire:

http://sourceforge.net/projects/tripwire/

It enables you to setup up an index of all the important files on your system, and if anyone messes with them you are notified. You can check the integrity of your files at any time that you want. This will give you the confidence to know that you javen't been hacked.

As for tracing the fault. First you need to check whether you can access the PC in anyway at all. Hopefully you should be able  to access at least a login prompt from the console?

If you can, then you need to try:

ps -ef               What processes are running
top                  Is anything using up the processor time


You may find that killing off/restarting certain rogue processes solves the problem, in which case you will know where to look next time.

You've done the obvious thing of looking in the logs. If there is nothing there, then you may need to increase the amount of logging that occurs. Linux software often has a -v (verbose), or -d  (debugging) mode.

It's also woth learning a little about klogd (the kernel logging daemon):

 http://linux.about.com/library/cmd/blcmdl8_klogd.htm

HTH:)
0
 
LVL 1

Assisted Solution

by:bestondoa
bestondoa earned 150 total points
ID: 11843279
Hello,

You should run a check on your systemany "rootkit". Rootkits are a collection of modified program sources or binaries which replace a selection of system binaries. For example, an hacker would enter your system and replace \bin\login with a modified one that has a secret username/password that will allways let him enter as root.

Go to http://www.chkrootkit.org/ and download chkrootkit. chkrootkit localy scans your system binaries for rootkit modification.

Cheers,

Bestondoa

0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Author Comment

by:manojoswal
ID: 11843478
1. We then got the server rebooted, since then its running fine.
2. We keep a good vigil on it, till the moment it stopped responding, we had adequate memory and CPU free. (its a dual xeon with 1 gb ram, SCSI and RAID) so the process that caused it was a sudden one.
3. There are no signs of DDOS or DOS.
4. We need to know exactly how we could go about finding the cause of this failure. What logs, what we should look for in it.
5. We have trip wire, it shows really nothing serious.

are there some links or guidlines somewhere that will guide us to track activity and find all possible traces of the cause of the problem.

regards

manoj



0
 

Author Comment

by:manojoswal
ID: 11843518
we need to report to the client the cause of this failure, any guidelines or papers on the net that have clues to foresnic analsysis of an attack.

regards

manoj oswal
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 100 total points
ID: 11843704
Is the server (and the kernel) up to date w/respect to the RHEL errata?

Is the BIOS on the system current?

What is the typical and peak load average?
0
 
LVL 1

Assisted Solution

by:LieutenantLefsa
LieutenantLefsa earned 100 total points
ID: 12171255
Did any of the disks fill up?
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
High Bandwidth Usage 6 75
Run same command on multiple files in Linux 3 59
fedora linux on laptop - setup sendmail - or some kind of email 5 54
I NEED A "BARE" LINUX ... 9 56
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Fine Tune your automatic Updates for Ubuntu / Debian
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question