Debian web server got hacked

Posted on 2013-11-14
Last Modified: 2013-11-27
I have got a Debian server hacked.  Please help me understand how can I protect it before it can get hacked again within the constraints of my setup?

Below is my setup and the list of processes at the time of the attack.  Some of my php scripts are still using "globals" so 5.2.x is still needed.

Debian O/S: squeeze(6)
Apache: 2.2.16-6+squeeze11
PHP: 5.2.6.dfsg.1-1+lenny16

Attached is the processes at the time of the attack with the commands: "top" and "ps -ef".

Question by:flowerbloom
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 5
LVL 12

Expert Comment

ID: 39647865
Oh I'd definitely say that looks hacked. The process 'upxCL110NFAJ3U' is doing a lot of work, and I don't recognize that. Plus it's running as www-data, the web server's user. It's probably a perl script. To find it, try:

locate upxCL110NFAJ3U

Open in new window

This may be a job for a security pro, but to mitigate the attack for the moment you have to do two things. First, kill the process(es) that are causing the issue, and secondly, plug the security hole. Actually, first, if possible, is to disconnect the machine from the network.

I would suspect that you have a security flaw in one of the scripts you are hosting. register_globals is a big security risk, and should never be turned on. You'll probably have to rewrite your scripts or secure them. Are these custom scripts or are they open source / reviewed by other developers?

It looks like perl is running something, most likely upxCL110NFAJ3U. (hit "c" when you are in top to verify that -- it will show the command line arguments, and it will look like "perl upxCL110NFAJ3U" or something similar)

The other disturbing part is that sshd is running a heck of a lot. And it, too, is running under www-data. It should never do that. It also means that there is network activity in and/or out of your server that you didn't authorize.

The best course of action, in reality, is a complete, fresh reinstall. It looks like the hacker(s) are pretty deep into the server and there really is no way to tell how far. In the absence of that possibility, kill all of the processes (especially upxCL110NFAJ3U) and restart the server. Then make sure those processes don't come back up on reboot. Then secure your scripts / sites. You'll just run into the same problem again if you don't.

Sorry for the bad news -- being hacked never has a simple solution. I'd strongly recommend hiring a security pro to get into the server and do some forensics. I've only scratched the surface.


LVL 110

Expert Comment

by:Ray Paseur
ID: 39648015
still using "globals" so 5.2.x is still needed.
PHP 5.2 is no longer supported at all, not even for security issues.  It's time to refactor so you can upgrade.

Author Comment

ID: 39650932
Thanks for the responds.

When the attack happend, I did disconnect the network and try to kill perl and the upxCL110NFAJ3U processes.  It took too long so I rebooted the machine.  I closed all port on the firewall except 22 (ssh), 80 (http), 443 (https), 10000 (webmin).

I have all security measures applied when using "register_globals" and I don't think that this is the case.  Perl is needed for webmin.  I do not have any perl scripts under apache.

I am working on moving the server away from "register_globals" and all new projects do not use globals.  Till then, I need these applications to run.

All your advice is great - but I'm looking to find the security hole before I re-install the server.

Please help.
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

LVL 12

Expert Comment

ID: 39651533
Are you still seeing a lot of sshd processes after the reboot? Is a randomly-named process still running?

In order to kill a process that won't die you can use ABRT:

kill -ABRT <processid>

That will absolutely force it to die instead of letting it linger around.


Author Comment

ID: 39651559
No suspected processes are running.

I used "kill -9".

Any ideas how to block the hole?  I have disabled executing CGI (remove "+ExecCGI" from apache configuration).
LVL 12

Expert Comment

ID: 39658130
Sorry for the delay. Weekend was unruly.

Plugging the hole is going to take a bit of detective work. If you are lucky then the logs on the server are still intact. I have reason to believe that they are since it looks like everything is run by www-data instead of root. If you know when the compromise happened then you should check the logs. A good way to determine that time and date is probably to find that random file and check the timestamp on it.

I'm mobile right now so I cant post any really in depth information easily. I just wanted to poke my head in to see how you were progressing.

It's really difficult to plug all of the holes after a compromise. The only really secure way to do it is to reinstall the server in a secure environment and reinstall your site on it . be sure the server is fully up to date on security patches as well.

There are also security tools you can use. I know of one that will scan your server for free once a week for known vulnerabilities. This will help you eliminate any issues introduced by the server software itself, as well as popular packages such as phpmyadmin. When I get back to the desktop I'll post the link I have to one of those services.

The art of securing a server after compromise is a complicated skill that takes quite a bit of experience. Hopefully by leveraging as many tools as you can use you will be able to make headway.

Has the attack continued or did it stop after the reboot?

LVL 12

Expert Comment

ID: 39658131
Just noticed your note about lack of suspicious processes, ignore my question about the attack continuing


Accepted Solution

flowerbloom earned 0 total points
ID: 39658937
Thank you for your input.

Attacks have continued although without a strong effect.  I have added security and now I have no attacks.  If no more inputs in the next week or so, I think I'm going to close this question as no more attacks occurring.  Thank you everyone for your input.

I hope my list below will help someone in the future.

Major things I did:
1. Remove awstat and phpmyadmin.
2. Remove exec cgi (+ExecCGI)
3. Limit the IP's and users that can get into ssh.
4. Create a "watchdog" program to alert, search and destroy these kind of attacks.
5. Install and run ClamAV (limited success in finding malware because that was not these kind of attacks).
6. Search and verify all files that belong to "www-data" and change owner if not necessary to be.
7. The attack has created a cron entry for "www-data", which I have removed.
8. The attack has created files and executables in /tmp and /var/tmp, which I have removed.

What I did not do:
1. Limit access to the server from countries like China and Turkey (see my reason below).
2. Upgrade to the "latest and greatest".  I still have application the require "old" technology.  I'm up to the "latest and greatest" version of the old technology (see title above).

The attacks came from the list of IP's below (Mostly China and Turkey).  These seem to be legitimate IP's that belong to legitimate companies (like real estate, etc.) and my suspicion is that these companies got hacked first and then the attack was launched on my server.
LVL 12

Expert Comment

ID: 39659831
Here is one of the services I use for peace of mind, although none of these services is foolproof. It will alert on any really outstanding security issues for sure, though:


Author Comment

ID: 39665413
Hi Mike,

I'm using ControlScan and RIPS to have quality assurance for my code.  The hackers did not attack the code, but attacking the infrastructure and the way the server was setup.  The company above would do that as my vendor - controlscan.

I have examined the latest and greatest version of each of my components and they all have the same security risk.  I do not know if they close this door by default, but definitely - this door exists.

Thank you.

Author Closing Comment

ID: 39680224
It works! (so far)

I don't see any more attempt to hack.

Featured Post

Manage your data center from practically anywhere

The KN8164V features HD resolution of 1920 x 1200, FIPS 140-2 with level 1 security standards and virtual media transmissions at twice the speed. Built for reliability, the KN series provides local console and remote over IP access, ensuring 24/7 availability to all servers.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
This article discusses how to implement server side field validation and display customized error messages to the client.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to count occurrences of each item in an array.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question