Debian web server got hacked

I have got a Debian server hacked.  Please help me understand how can I protect it before it can get hacked again within the constraints of my setup?

Below is my setup and the list of processes at the time of the attack.  Some of my php scripts are still using "globals" so 5.2.x is still needed.

Debian O/S: squeeze(6)
Apache: 2.2.16-6+squeeze11
PHP: 5.2.6.dfsg.1-1+lenny16

Attached is the processes at the time of the attack with the commands: "top" and "ps -ef".

Who is Participating?
flowerbloomConnect With a Mentor Author Commented:
Thank you for your input.

Attacks have continued although without a strong effect.  I have added security and now I have no attacks.  If no more inputs in the next week or so, I think I'm going to close this question as no more attacks occurring.  Thank you everyone for your input.

I hope my list below will help someone in the future.

Major things I did:
1. Remove awstat and phpmyadmin.
2. Remove exec cgi (+ExecCGI)
3. Limit the IP's and users that can get into ssh.
4. Create a "watchdog" program to alert, search and destroy these kind of attacks.
5. Install and run ClamAV (limited success in finding malware because that was not these kind of attacks).
6. Search and verify all files that belong to "www-data" and change owner if not necessary to be.
7. The attack has created a cron entry for "www-data", which I have removed.
8. The attack has created files and executables in /tmp and /var/tmp, which I have removed.

What I did not do:
1. Limit access to the server from countries like China and Turkey (see my reason below).
2. Upgrade to the "latest and greatest".  I still have application the require "old" technology.  I'm up to the "latest and greatest" version of the old technology (see title above).

The attacks came from the list of IP's below (Mostly China and Turkey).  These seem to be legitimate IP's that belong to legitimate companies (like real estate, etc.) and my suspicion is that these companies got hacked first and then the attack was launched on my server.
Oh I'd definitely say that looks hacked. The process 'upxCL110NFAJ3U' is doing a lot of work, and I don't recognize that. Plus it's running as www-data, the web server's user. It's probably a perl script. To find it, try:

locate upxCL110NFAJ3U

Open in new window

This may be a job for a security pro, but to mitigate the attack for the moment you have to do two things. First, kill the process(es) that are causing the issue, and secondly, plug the security hole. Actually, first, if possible, is to disconnect the machine from the network.

I would suspect that you have a security flaw in one of the scripts you are hosting. register_globals is a big security risk, and should never be turned on. You'll probably have to rewrite your scripts or secure them. Are these custom scripts or are they open source / reviewed by other developers?

It looks like perl is running something, most likely upxCL110NFAJ3U. (hit "c" when you are in top to verify that -- it will show the command line arguments, and it will look like "perl upxCL110NFAJ3U" or something similar)

The other disturbing part is that sshd is running a heck of a lot. And it, too, is running under www-data. It should never do that. It also means that there is network activity in and/or out of your server that you didn't authorize.

The best course of action, in reality, is a complete, fresh reinstall. It looks like the hacker(s) are pretty deep into the server and there really is no way to tell how far. In the absence of that possibility, kill all of the processes (especially upxCL110NFAJ3U) and restart the server. Then make sure those processes don't come back up on reboot. Then secure your scripts / sites. You'll just run into the same problem again if you don't.

Sorry for the bad news -- being hacked never has a simple solution. I'd strongly recommend hiring a security pro to get into the server and do some forensics. I've only scratched the surface.


Ray PaseurCommented:
still using "globals" so 5.2.x is still needed.
PHP 5.2 is no longer supported at all, not even for security issues.  It's time to refactor so you can upgrade.
Improved Protection from Phishing Attacks

WatchGuard DNSWatch reduces malware infections by detecting and blocking malicious DNS requests, improving your ability to protect employees from phishing attacks. Learn more about our newest service included in Total Security Suite today!

flowerbloomAuthor Commented:
Thanks for the responds.

When the attack happend, I did disconnect the network and try to kill perl and the upxCL110NFAJ3U processes.  It took too long so I rebooted the machine.  I closed all port on the firewall except 22 (ssh), 80 (http), 443 (https), 10000 (webmin).

I have all security measures applied when using "register_globals" and I don't think that this is the case.  Perl is needed for webmin.  I do not have any perl scripts under apache.

I am working on moving the server away from "register_globals" and all new projects do not use globals.  Till then, I need these applications to run.

All your advice is great - but I'm looking to find the security hole before I re-install the server.

Please help.
Are you still seeing a lot of sshd processes after the reboot? Is a randomly-named process still running?

In order to kill a process that won't die you can use ABRT:

kill -ABRT <processid>

That will absolutely force it to die instead of letting it linger around.

flowerbloomAuthor Commented:
No suspected processes are running.

I used "kill -9".

Any ideas how to block the hole?  I have disabled executing CGI (remove "+ExecCGI" from apache configuration).
Sorry for the delay. Weekend was unruly.

Plugging the hole is going to take a bit of detective work. If you are lucky then the logs on the server are still intact. I have reason to believe that they are since it looks like everything is run by www-data instead of root. If you know when the compromise happened then you should check the logs. A good way to determine that time and date is probably to find that random file and check the timestamp on it.

I'm mobile right now so I cant post any really in depth information easily. I just wanted to poke my head in to see how you were progressing.

It's really difficult to plug all of the holes after a compromise. The only really secure way to do it is to reinstall the server in a secure environment and reinstall your site on it . be sure the server is fully up to date on security patches as well.

There are also security tools you can use. I know of one that will scan your server for free once a week for known vulnerabilities. This will help you eliminate any issues introduced by the server software itself, as well as popular packages such as phpmyadmin. When I get back to the desktop I'll post the link I have to one of those services.

The art of securing a server after compromise is a complicated skill that takes quite a bit of experience. Hopefully by leveraging as many tools as you can use you will be able to make headway.

Has the attack continued or did it stop after the reboot?

Just noticed your note about lack of suspicious processes, ignore my question about the attack continuing

Here is one of the services I use for peace of mind, although none of these services is foolproof. It will alert on any really outstanding security issues for sure, though:

flowerbloomAuthor Commented:
Hi Mike,

I'm using ControlScan and RIPS to have quality assurance for my code.  The hackers did not attack the code, but attacking the infrastructure and the way the server was setup.  The company above would do that as my vendor - controlscan.

I have examined the latest and greatest version of each of my components and they all have the same security risk.  I do not know if they close this door by default, but definitely - this door exists.

Thank you.
flowerbloomAuthor Commented:
It works! (so far)

I don't see any more attempt to hack.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.