?
Solved

Server Hacked: To Reformat or Not?

Posted on 2003-03-05
8
Medium Priority
?
449 Views
Last Modified: 2010-04-20
Hello all,

  My server was broken into yesterday using the sendmail flaw. Around 7:30 in the morning I received emails designed to cause buffer overflow (the address had a long string of <><><><>...). At the same time the traffic chart shows that there was a lot of traffic out of the server (possibly similar emails sent to other servers). Also later in the day I noticed some processes (groff, troff etc.) running which I have not seen before nor do I believe any of my software should cause to run. Conclusion: Highly likely that the server has been broken into. The OS is Redhat 7.1.

  Question is now what? Should I reformat the server or not? Can I detect and disable all rootkits that may have been installed by the hacker? Is it impossible to detect and get rid of any LKM trojans that may have been installed?

  I would appreciate replies detailing the pros or cons of the two strategies. And an informative answer of whether reformatting is the best solution?

  Thanks in advance,

RtN
0
Comment
Question by:RuletheNet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
8 Comments
 
LVL 38

Accepted Solution

by:
yuzh earned 2000 total points
ID: 8078005
To figure out what files had been replaced by the hacker is very time consume process.

you should put the server off the network first. If you want to figure out what had the hacker done on your system, you keep the HD. Install a new HD on the server and restore from your reliable system backup (If you have one), otherwise, perform a fresh OS install for the server, and make sure that you have all the latest OS patch install, then disable all the unwanted services, configure a firewall if you can, at least you should install the tcp_wrapper 7.6 (tcp_wrappers package).

After you get your server up and running, you can then transfer the applcation data from the old drive (hacked HD). to the new one.

You can then mount the hacked HD, and check it out.

It is likely the hacked replaced the following binary.

netstat, su, passwd, ps,rusers,  ls, ifconfig,...etc

do a "cksum" to find out.

Something the hacked install a rootkit in /dev (or sniffer)

find /dev -type f -print

to find out.

Soem people download "chkrootkit" script and try to detect the rootkit, but it can only delect the OLD ones, it cann't detect the new root kits !!!


#****************************************************
IMPORTANT: after you rebuilt the server, please remember to change the "root" password, and change all the users passwd as well.


Hope the information can help !


0
 
LVL 38

Expert Comment

by:yuzh
ID: 8078013
You might be able to find the hacker had install soemthing in /usr/lib as well.
0
 
LVL 1

Expert Comment

by:LamerSmurf
ID: 8078393
alot can be done, however if you are not better at Linux than your hacker.. i would suggest reinstall.

my 5 cents.

/LamerSmurf
0
Free Backup Tool for VMware and Hyper-V

Restore full virtual machine or individual guest files from 19 common file systems directly from the backup file. Schedule VM backups with PowerShell scripts. Set desired time, lean back and let the script to notify you via email upon completion.  

 
LVL 20

Expert Comment

by:Gns
ID: 8078451
I'm in complete agreement with yuzh. there is to much chance that you will miss something, as well as the exorbitant amount of work involved, that the reinstall->migrate "non-volatile" data is the only viable option.

In the future you would do well to implement tripwire, or some (full-fledged) IDS. They could help you reduce the amount of work involved in a "restoration" to manageable amounts, as well as drastically increase the accuracy thereof.

-- Glenn
0
 

Expert Comment

by:LaMaZ
ID: 8078454
If you don't want to reinstall, i suggest monitoring your server very closely for the next few days

You could also tighten the firewall for a while (month or so) ... If this is possible ofcourse.

i suggest you install IDS (intrusion detection system) to monitor the server.  
I suggest snort as IDS. http://www.snort.org/about.html
It is a bitch to get installed, but it'll be worth it.

You could also scan your system for rootkits using a program like chkrootkit.  http://www.chkrootkit.org/


It seems the hacker that hit you was a sciptkiddie. He prob scanned from your machine, so he will have found more machines with the hole.  This makes him no real treath for you again, you closed the hole, and he lost his access...

If he doesn't have a rootkit, and you check regularly for intrusions, you should be safe without reinstalling.

Stefaan
 
0
 
LVL 2

Expert Comment

by:jimbb
ID: 8086312
Reinstall, unquestionably.

Make sure if you restore anything from the "old" system, that it's non-executable stuff.. e.g., configuration files (after you check them for correctness) and/or app data and so on, as others have suggested.
0
 
LVL 20

Expert Comment

by:Gns
ID: 8086737
Funny thing.
Snort turn out to have a vulnerability, so ISS issued an alert last week.
Talk about contraproductive IDS:-).
Fortunately fixed versions are already out (since monday), and workarounds for those who cannot upgrade (just comment out "# preprocessor rpc_decode" in snort.conf).
Oh well...

-- Glenn
0
 

Author Comment

by:RuletheNet
ID: 8088909
Thanks everybody for the very helpful information. I am giving the points to Yuzh as he or she answered first and in detail, but I appreciate everybody's help.

Regards,

Rtn
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month15 days, 4 hours left to enroll

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question