[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Server stopped working all of the sudden, possible attack?

Posted on 2004-08-18
7
Medium Priority
?
320 Views
Last Modified: 2010-04-20
Hello All,

We have a server running with RHE 3, Apache, Php, Mysql and Bind. It's our main production server.

Now the problem we faced day before was something very weird. All of the sudden all the services on the server stopped responding, we couldn't ssh into the server, we couldn't see the sites, couldn't fetch mails and so on.

We could ping the server though, ping was going fine, but we just couldn't access the server in any way.

Now the question I would like to ask you guys is,  if such problem occurs again, how to trace it. I tried to check /var/log/messages but all I found were portsentry's messages.

If we were under attack, or if the server was hacked, how do we trace it? Are there any guidelines or links through which we can possibly know what cause our server to respond all of the sudden?


Thanks in Advance.
0
Comment
Question by:manojoswal
7 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 11832964
> but we just couldn't access the server in any way.

Was the console still responsive? From the console could you initiate a network connection out to some other box (e.g., ssh some-local-host)? What was the load average on the box?

Is the server up to date (as in running up2date and allowing it to update the kernel)?
0
 
LVL 22

Accepted Solution

by:
pjedmond earned 450 total points
ID: 11835251
I recommend having a look at tripwire:

http://sourceforge.net/projects/tripwire/

It enables you to setup up an index of all the important files on your system, and if anyone messes with them you are notified. You can check the integrity of your files at any time that you want. This will give you the confidence to know that you javen't been hacked.

As for tracing the fault. First you need to check whether you can access the PC in anyway at all. Hopefully you should be able  to access at least a login prompt from the console?

If you can, then you need to try:

ps -ef               What processes are running
top                  Is anything using up the processor time


You may find that killing off/restarting certain rogue processes solves the problem, in which case you will know where to look next time.

You've done the obvious thing of looking in the logs. If there is nothing there, then you may need to increase the amount of logging that occurs. Linux software often has a -v (verbose), or -d  (debugging) mode.

It's also woth learning a little about klogd (the kernel logging daemon):

 http://linux.about.com/library/cmd/blcmdl8_klogd.htm

HTH:)
0
 
LVL 1

Assisted Solution

by:bestondoa
bestondoa earned 450 total points
ID: 11843279
Hello,

You should run a check on your systemany "rootkit". Rootkits are a collection of modified program sources or binaries which replace a selection of system binaries. For example, an hacker would enter your system and replace \bin\login with a modified one that has a secret username/password that will allways let him enter as root.

Go to http://www.chkrootkit.org/ and download chkrootkit. chkrootkit localy scans your system binaries for rootkit modification.

Cheers,

Bestondoa

0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:manojoswal
ID: 11843478
1. We then got the server rebooted, since then its running fine.
2. We keep a good vigil on it, till the moment it stopped responding, we had adequate memory and CPU free. (its a dual xeon with 1 gb ram, SCSI and RAID) so the process that caused it was a sudden one.
3. There are no signs of DDOS or DOS.
4. We need to know exactly how we could go about finding the cause of this failure. What logs, what we should look for in it.
5. We have trip wire, it shows really nothing serious.

are there some links or guidlines somewhere that will guide us to track activity and find all possible traces of the cause of the problem.

regards

manoj



0
 

Author Comment

by:manojoswal
ID: 11843518
we need to report to the client the cause of this failure, any guidelines or papers on the net that have clues to foresnic analsysis of an attack.

regards

manoj oswal
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 300 total points
ID: 11843704
Is the server (and the kernel) up to date w/respect to the RHEL errata?

Is the BIOS on the system current?

What is the typical and peak load average?
0
 
LVL 1

Assisted Solution

by:LieutenantLefsa
LieutenantLefsa earned 300 total points
ID: 12171255
Did any of the disks fill up?
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month18 days, left to enroll

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question