Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Server Hacked: To Reformat or Not?

Posted on 2003-03-05
Medium Priority
Last Modified: 2010-04-20
Hello all,

  My server was broken into yesterday using the sendmail flaw. Around 7:30 in the morning I received emails designed to cause buffer overflow (the address had a long string of <><><><>...). At the same time the traffic chart shows that there was a lot of traffic out of the server (possibly similar emails sent to other servers). Also later in the day I noticed some processes (groff, troff etc.) running which I have not seen before nor do I believe any of my software should cause to run. Conclusion: Highly likely that the server has been broken into. The OS is Redhat 7.1.

  Question is now what? Should I reformat the server or not? Can I detect and disable all rootkits that may have been installed by the hacker? Is it impossible to detect and get rid of any LKM trojans that may have been installed?

  I would appreciate replies detailing the pros or cons of the two strategies. And an informative answer of whether reformatting is the best solution?

  Thanks in advance,

Question by:RuletheNet
LVL 38

Accepted Solution

yuzh earned 2000 total points
ID: 8078005
To figure out what files had been replaced by the hacker is very time consume process.

you should put the server off the network first. If you want to figure out what had the hacker done on your system, you keep the HD. Install a new HD on the server and restore from your reliable system backup (If you have one), otherwise, perform a fresh OS install for the server, and make sure that you have all the latest OS patch install, then disable all the unwanted services, configure a firewall if you can, at least you should install the tcp_wrapper 7.6 (tcp_wrappers package).

After you get your server up and running, you can then transfer the applcation data from the old drive (hacked HD). to the new one.

You can then mount the hacked HD, and check it out.

It is likely the hacked replaced the following binary.

netstat, su, passwd, ps,rusers,  ls, ifconfig,...etc

do a "cksum" to find out.

Something the hacked install a rootkit in /dev (or sniffer)

find /dev -type f -print

to find out.

Soem people download "chkrootkit" script and try to detect the rootkit, but it can only delect the OLD ones, it cann't detect the new root kits !!!

IMPORTANT: after you rebuilt the server, please remember to change the "root" password, and change all the users passwd as well.

Hope the information can help !

LVL 38

Expert Comment

ID: 8078013
You might be able to find the hacker had install soemthing in /usr/lib as well.

Expert Comment

ID: 8078393
alot can be done, however if you are not better at Linux than your hacker.. i would suggest reinstall.

my 5 cents.

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

LVL 20

Expert Comment

ID: 8078451
I'm in complete agreement with yuzh. there is to much chance that you will miss something, as well as the exorbitant amount of work involved, that the reinstall->migrate "non-volatile" data is the only viable option.

In the future you would do well to implement tripwire, or some (full-fledged) IDS. They could help you reduce the amount of work involved in a "restoration" to manageable amounts, as well as drastically increase the accuracy thereof.

-- Glenn

Expert Comment

ID: 8078454
If you don't want to reinstall, i suggest monitoring your server very closely for the next few days

You could also tighten the firewall for a while (month or so) ... If this is possible ofcourse.

i suggest you install IDS (intrusion detection system) to monitor the server.  
I suggest snort as IDS. http://www.snort.org/about.html
It is a bitch to get installed, but it'll be worth it.

You could also scan your system for rootkits using a program like chkrootkit.  http://www.chkrootkit.org/

It seems the hacker that hit you was a sciptkiddie. He prob scanned from your machine, so he will have found more machines with the hole.  This makes him no real treath for you again, you closed the hole, and he lost his access...

If he doesn't have a rootkit, and you check regularly for intrusions, you should be safe without reinstalling.


Expert Comment

ID: 8086312
Reinstall, unquestionably.

Make sure if you restore anything from the "old" system, that it's non-executable stuff.. e.g., configuration files (after you check them for correctness) and/or app data and so on, as others have suggested.
LVL 20

Expert Comment

ID: 8086737
Funny thing.
Snort turn out to have a vulnerability, so ISS issued an alert last week.
Talk about contraproductive IDS:-).
Fortunately fixed versions are already out (since monday), and workarounds for those who cannot upgrade (just comment out "# preprocessor rpc_decode" in snort.conf).
Oh well...

-- Glenn

Author Comment

ID: 8088909
Thanks everybody for the very helpful information. I am giving the points to Yuzh as he or she answered first and in detail, but I appreciate everybody's help.



Featured Post

Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month11 days, 7 hours left to enroll

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question