Server Hacked: To Reformat or Not?

Posted on 2003-03-05
Medium Priority
Last Modified: 2010-04-22
Hello all,

  My server was broken into yesterday using the sendmail flaw. Around 7:30 in the morning I received emails designed to cause buffer overflow (the address had a long string of <><><><>...). At the same time the traffic chart shows that there was a lot of traffic out of the server (possibly similar emails sent to other servers). Also later in the day I noticed some processes (groff, troff etc.) running which I have not seen before nor do I believe any of my software should cause to run. Conclusion: Highly likely that the server has been broken into. The OS is Redhat 7.1.

  Question is now what? Should I reformat the server or not? Can I detect and disable all rootkits that may have been installed by the hacker? Is it impossible to detect and get rid of any LKM trojans that may have been installed?

  I would appreciate replies detailing the pros or cons of the two strategies. And an informative answer of whether reformatting is the best solution?

  Thanks in advance,

Question by:RuletheNet
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +2
LVL 51

Expert Comment

ID: 8080456
> Can I detect and disable all rootkits ..
if you have clean versions of most programs handy (on floppy, CD, etc.) then there is a chance to detect known (**not all**) rootkits with chkrootkit from:

> Is it impossible to detect and get rid of any LKM trojans ..
In theroy yes, in praxis depends ...
In theory, theory and praxis are identical, in praxis they are not.

The most clean method would be a install from scratch, best with formated disk too.

Accepted Solution

naccad earned 2000 total points
ID: 8080590
I strongly suggest not reformating.. but installing a new
hard disk and sending this one for analysis somewhere.

I had the same situation a few months ago with the OpenSSL
bug, a detailed analysis showed exactly what the intruder
did, how he did it and lots of information that helped me
futher increase the level of security on my network.

The problem with this is that i had to isolate that box
for about a month until i squeezed every piece of information out of it.

The best solution is a fresh install, if you have not done
this before, cleaning a system is a very complicated job
and then there is always that little doubt in the back of
your head.

A better solution is to keep that hard disk isolated on an
isolated system where you can take your time poking around
and learning.

A last note is that according to CERT, there is no known
exploit in the wild that can use the latest sendmail
buffer overflow.. before you do anything, make sure you
really did get hacked, send whatever information you have
to CERT and have them look at it.

Good luck
LVL 14

Expert Comment

ID: 8081065
> sending this one for analysis somewhere.

Or at least to take a sector-level backup (i.e., with dd) so you can reconstruct the data for analysis.

As for whether you really need to start fresh, it depends on whether you feel you can verify that the system is really clean. chkrootkit.org can definitely help.

But you really want something like Tripwire that can tell you for sure what's been changed. Unfortunately, this requires that you started taking snapshots before the breakin.

Another way of doing things is to compare the bits on disk with the install media. But you don't want to boot off the suspect disk to do this, so you'll need a bootable CD with all the tools needed to do the checks.

Of course, reformatting may be easier...

> A last note is that according to CERT, there is no known
> exploit in the wild that can use the latest sendmail
> buffer overflow.

Actually, there is at least one known to be going around now.
The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.


Expert Comment

ID: 8081147
>Actually, there is at least one known to be going around now.

this is indeed news to me, do u have a link to an advisory
or something?

LVL 14

Expert Comment

ID: 8081406
There was a sample-code posting yesterday on Bugtraq by Last Stage of Delerium, and Computer World is reporting that Russian and Polish hacker groups have translated this into a full exploit that is now in the wild.

Expert Comment

ID: 8083674
This question came up in one of the discussions amongst the OClug "should I format or not." I think it was during the Openssl vulnerability. Most of the replies were similar: download patches/fixes, backup all data, and reinstall with the patches/fix or new version. There was a reason for this but I couldn't fine the email.

I use Postfix on my Mandrake box works just as well.

Good luck.

Expert Comment

ID: 8083691
Opps, thats reinstall the OS along with the patches/fix or new version

Author Comment

ID: 8088942
Thanks everybody for your very helpful answers. Nick gets the points, but everybodys help is greatly appreciated.



Featured Post

Give Your Engineering Team a Productivity Boost

Learn why container technology is so powerful and how it can provide your team with productivity gains and other benefits.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

BIND is the most widely used Name Server. A Name Server is the one that translates a site name to it's IP address. There is a new bug in BIND (https://kb.isc.org/article/AA-01272), affecting all versions of BIND 9 from BIND 9.1.0 (inclusive) thro…
Fine Tune your automatic Updates for Ubuntu / Debian
Have you created a query with information for a calendar? ... and then, abra-cadabra, the calendar is done?! I am going to show you how to make that happen. Visualize your data!  ... really see it To use the code to create a calendar from a q…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question