Hackers Guide to Being Hacked: How “Bad Guys” Take Control, and How to Take it Back.

"Oh crap. I think we've been hacked."

That's the last sentence you ever want to hear from your IT guy, and the last sentence you ever want to have to say to your customers and clients. The fact is, hackers are everywhere. They are out there doing bad stuff for profit, pleasure, notoriety, or simply because: "I can".

This is intended to be a quick, down and dirty, guide to recovering from an attack. No matter if you're a noob or a seasoned IT professional, it is really only a matter of time before you will have to deal with an intruder.

An Ounce of Prevention...
It is so much easier to prevent being hacked than it is to recover. The basic way you prevent hackers from compromising your systems is to make it more difficult than average to break your protection. There is no need to build a digital Fort Knox around your network (although I won't disagree with this practice), but there are 5 simple steps you can use to deter hackers from breaking into your computers, servers, and network.

Never Be the Low Hanging Fruit
If you are the low hanging fruit: a.k.a. an easy target, you're going to get nailed. Hackers, for the most part, are looking for an easy target: a weak password, a stupid employee, an open server door, an unlocked office door, or an untrained user.

In reality, you can take some extremely simple, easy steps, to make it orders of magnitude more difficult to break into your network. You don't have to be the safest, you just have to be safe enough.

Just like in the wild, you don't have to be the fastest Gazelle running from the Lions, you just have to be faster than the slowest.

1. Basic Rules.

Do NOT Appear to be an Appetizing Feast.

Do not store credit card numbers, social security numbers, national ID numbers, or any other profitable information on your server or in your databases unless they are encrypted. And even then... it's still a risk.

If you take credit cards or deal with other sensitive information, it is best to let your merchant services gateway deal with the security issues. You don't want to have the liability.

Database security comprises a huge topic and subsequently is beyond the scope of this article; however, there are techniques and methods out there to help protect your data. SQL injection, running queries directly from your web application can be very troublesome. Methodologies to combat them are both numerous and varied such as using stored procedures and encryption. Please feel free to discuss any or all of those issues below or submitting an article on those topics to the EE articles sections, and then posting a link to it here.

Use Strong Passwords.

Stop complaining, and just do it. There is no reason to use "pencil1234" as your password. Hackers can break these in seconds.

Use a random password generator to create your passwords, and make them 11 characters in length.

Of course, this makes it imperative that you use a password manager since every account you have should have a different password. Fortunately, http://keepass.info/ is a free, open source password manager with a built-in random password generator. And, with KeeForm, you don't even have to worry about copy / pasting your usernames and passwords. KeeForm will open your browser, and enter your username and password directly into whatever website you have.

Keep Your System Up-To-Date.

This one is easy. Mac, Linux, Windows. RUN THE UPDATES.

Use a Firewall.

You should have a network firewall and software firewalls on your equipment. The network firewall keeps the bad guys out of your network and puts in in "stealth" mode for all the services you are not using.

I personally recommend m0n0wall as a software firewall as your network firewall.

Other software firewalls are those that are built into most every client operating systems. Windows XP SP2 or later has the windows firewall built in. Linux does. Mac does. Use it.

Educate Yourself and Your Users.

You and your users should all be able to recognize the various virus, spy-ware, and rouge-ware attacks that are out there. Become familiar with the different types of attacks on the Internet and how to stop them.

Train your users to NEVER click on the dancing bear, install those stupid smiley faces, or run a supposed "Windows Update" that claims they have been infected.

The simple fact of the matter is: the only thing that should be telling you that "your infected" is your anti-virus software. Everything else is suspicious.

Lastly, post, publish, email out, and harp on your internal IT security policy. It should be the most well-known document in the company. Period. There should be no one in the entire company who would even think of saying "I didn't know..." when confronted with why they clicked on a dancing bear.

2. Symptoms of an Attack.

Your network upload / download speed is unusually slow.

If you have an invader sending spam, working a bot-net, or otherwise usurping your network connection, you may notice a dramatic decrease of your normal upload or downloading speeds. This is most noticeable if you have VoIP services because the intruder will fill up your Internet pipe so completely, that it will cause the voice calls to sound like they are underwater or they will break up to the point where it is unusable.

You Receive Hate Mail.

If you have a computer or server on your network that has been compromised, you might receive hate mail from someone out there that looks like this:

Subject: attempted felony computer trespass/felony theft of services from your network

This is a log (times in EDT -0400) of attempted computer trespass/attempted theft of computer services, both felonies, from your address space. We report all such attempts to the Boston FBI office to be on record in case the attacks continue. Please investigate and treat these attempts as the felonies that they are. In the (admittedly unlikely) event that the FBI has time to investigate these criminal attempts, please keep all logs and results from the investigation available in case they are subpoenaed. Thanks for your help in investigating these felonies and in preventing future occurrences.

You Receive an Astronomical Phone Bill.

I am often surprised at how commonplace it is to have someone setup a VoIP telephone system such as (Asterisk, Elastix, or Trixbox) on a public IP address, and then not harden the box. In some cases of severe negligence, the person who set it up did not even bother to change the default username and password.

So, hackers find the box, log in, and take over. They will make thousands of dollars of phone calls through your server, and you are stuck with the phone bill.

You Can't Get to your Website.

You may find that one day your website is not viewable. In fact, there are warnings to everyone who goes there about it being an "Attack Page." When hackers break in, they will typically upload code or content to the server for a variety of any malicious purposes including:



The most obvious sign and symptom that you have been a victim of this type of attack is pictured below:

 
   

3. How to Recover from an Attack.

Find the Affected Machine(s).

In some cases, like the website hijack or the "attack page" example, you'll know which machine has been affected. In most cases, however, you're going to have to find it.

The best tool for figuring out which computer has been compromised is netstat. Netstat reveals all the incoming and outgoing connections on a computer, and is available on Linux, Windows, and Mac.

Methodically check all the computers in your network using one of the netstat commands below:

Windows

 netstat -an | find "ESTABLISHED"

Select all Open in new window

Mac / Linux

netstat -an | grep ESTABLISHED

Select all Open in new window

This command will show you all the established connections to the computer. It will be fairly obvious which machine is infected when you find it. For example, a Linux server infested with an SSH scanner (a script that breaks into your network, then tries to break into everyone else's) will have tons of connections to other networks with a destination port of 22.

A spam relay will have tons of connections to everywhere with a destination port of 25.

An FTP brute force attacker will have tons of connections to destination port 21.

...and so on.

When you find a machine that you suspect has been infected, run the same command as above again, only this time leave out the n switch:

Windows

 netstat -a | find "ESTABLISHED"

Select all Open in new window

Mac / Linux

netstat -a | grep ESTABLISHED

Select all Open in new window

This will allow the system to resolve the IP addresses previously shown to their rDNS equivalents. We're looking for a dead giveaway here: connections to foreign countries. When you see 17 connections from Romania, Russian, Taiwan or some other foreign country, and you are not in that country, it is a dead giveaway.

Lastly, run netstat a final time to find the offending processes:

Windows

 netstat -ao | find "ESTABLISHED"

Select all Open in new window

Mac / Linux

netstat -ap | grep ESTABLISHED

Select all Open in new window

Write down the offending processes. We'll need this list later.

Blockade your Network.

By now, you should have already downloaded m0n0wall - as I mentioned, you should have a firewall in the prevention steps. A hardware firewall is your best friend in mitigating the effects of a hack attack. While we originally implement a hardware firewall to keep bad guys out, if you've been compromised, you have a duty to mitigate the damage you are doing by keeping the probes, scans, spam, and other garbage in.

Turn on the outbound firewall, and set it to disallow all outbound traffic except traffic with a destination port of 80 or 443. This allows you to keep surfing the web and access secure sites to get more help and information to recover from the attack, but it keeps all the spam (destination port 25), port scans (many different ports), DoS attacks (ping), BotNets (usually IRC ports), and other nasty stuff from being able to leave your network, which renders them effectively useless to the attacker.

Disconnect or Shutdown Public IP Boxes.

If you have boxes on a public IP address (such as a VoIP server, web server, mail server, etc...) shut them down or unplug them until you are ready to deal with the issue.

Turn on the Software Firewall for Public IP Boxes.

Just like we turned on the network firewall in step 1, you need to turn on the software firewall for the server now. Disallow all inbound connections except those that are necessary for fixing the problem. For example, disallow SSH port 22 from everywhere so that no intruders can get into the box and undo your work while you fix it. Allow RDP (TCP 3389) so that you can get into your Windows servers. Everything else should be blocked.

Flush Queues and Reboot.

If your server has any type of queuing system (like Mail Servers), then you need to dump the queues and reboot.

We dump the queues to free up resources. Microsoft Exchange, for example, is nearly unusable if the queues have 147,000 outbound messages about dating mail-order brides in the queue. Even if you have some important mail in there, kill the queues.

Once you have flushed the queues, reboot the server. You have already setup the firewall so the bad guys can't get back in, and the bad stuff can't get out, so we need to reboot the machines to break any existing connections and kill any established processes. This creates as clean of an environment for recovery as you can get.

Find and Remove the Offending Processes.

In Linux, use the locate command. In Windows, do a search from the root of the drive using:

dir badsoftwarename.exe /s

Select all Open in new window

Delete every occurrence you can find. The bad processes / binaries / scripts / applications cannot start if they do not exist.

Be sure to research these processes and vulnerabilities on the Internet to get as much information on them as you can. You are guaranteed not to be the only one with the problem. Someone else has written about it, blogged about it, complained about it. Find them.

Find and Remove the Offending Content.

In cases of website hijacking, HTML may inserted into your regular web pages using CSS or other techniques so that at first glance they appear normal; however, under certain conditions, they will display malicious content.

The easiest way to remove this content is by restoring from a backup. Otherwise, you have to manually remove the offending code.

Run Anti-Virus Scans.

Windows, Linux, Mac. Use your favorite Anti-virus scanning software to scan the drive for offending items.

I personally like ESET Nod32 for Windows and ClamAV for Linux. However, I don't usually get in that discussion because recommending Anti-virus to another IT professional is about as contentious as a religious debate.

Additionally, in Windows, use something like HijackThis to remove offending registry entries as well.

Retrace Their Steps.

Check the logs (/var/log/* in Linux and the Event Log in Windows) for signs of the break-in. You will usually find that there are a series of attempts to log into the server using a particular service. This happens because during a scan from the outside, your server or computer was fingerprinted and the Operating System type was discovered. Once that is known, they will do penetration testing to various services: especially those with known vulnerabilities. Once they find a weak point, they can get in.

The logs will show you how they got in, usually how long it took, and lastly what service was used to compromise the system.

Once you figure that out, modify the system to prevent future attacks.

Fix all the Problems You Should Have Never Had.

Scroll to the top of this article and implement all the "ounce of prevention" steps above. Most notably: strong passwords and software updates. This INCLUDES upgrading to the newest version of the operating system. Even if that means re-installing.

I once consulted on a security breach at a well-known financial institution who will remain nameless. They were running Fedora Core 4 Linux, which was a full six version behind the times (FC10 was the current version then). No doubt their issues were the result of not doing upgrades.

Changing passwords and removing newly created accounts on the server is another biggie. Change all the passwords for every user, system, and service. Make them strong.

Change ports if possible. For example, if you had SSH listening on port 22, change it to 220 or something else.

Lastly, examine all your code, scripts, and pages. Cross-site scripting, SQL injection attacks, and other web attacks can be easily thwarted by using proper security protocols in your code and scripts. If, during investigation, it is determined hackers exploited these methods, you must fix your poorly written code to prevent future attacks.

Reboot and Observe.

Once you think you have gotten the offending malware off your system, you can return it back to production settings and security. Un-firewall it.

Now watch it like a hawk. If the problems return, you didn't fix the issue, and you have to start all over again.

If you just can't get rid of the issue, they you're going to have to either restore from a backup or reinstall entirely because you've got a rootkit, which is nearly impossible to remove.

4. Tell Others.

Telling others helps them solve the problem when they run into it. By making your experiences known and published on the Internet, you contribute to the collective knowledge of IT Professionals.

Post your experience in the comments section below.  If you have a blog, blog about your experience and post a link to your blog entry here, as well.

...and by all means: share this article. Post it on Facebook, Tweet it. Get the word out. Help as many people as you can.

This article is a reprint of Hackers Guide to Being Hacked: How “Bad Guys” Take Control, and How to Take it Back. Used with Permission.

Get to know the author on Facebook or Experts Exchange.