?
Solved

DoS or DDoS attack?

Posted on 2009-04-27
14
Medium Priority
?
828 Views
Last Modified: 2012-06-27
Hi Experts -
I have been dealing with extremely slow internet connection for the corporate network for some time now.  While troubleshooting, with the help of a local network group, we have discovered that the slowness is caused by one or two internal clients that appear to be targeted by a flood of traffic directed at theier IP address.  We are behind a Cisco Firewall and can see the full circut 'maxed-out' and a list of one or two users in the 'Top 10 Sources' in the ASA who appear to be consuming all bandwidth.  Shutting down or rebooting the PC will 'break' the connection and allow normal bandwidth to resume.  Blocking the source outside address within the firewall fixes temporarily, also.  I have updated the internal 'problem' PC's with all Microsoft Updates, run AntiVirus and Malware scans and nothing is discovered.  If there is something resident, it is hiding very well.  This appears to be some type of attack from a host outside our network.  What can I do to fix this problem?
0
Comment
Question by:davis
  • 7
  • 2
  • 2
  • +1
12 Comments
 
LVL 4

Expert Comment

by:lkraven
ID: 24245720
If they are the Top 10 *Sources*, they are likely to be transmitting data, not receiving data.

They may be zombied computers participating in a botnet.

You say you have an ASA at the edge of your network, and therefore I am making an assumption that the machines in question have internal non-routable addresses.  If you were truly under attack from an outside host on your public internet facing IP address, all your users would seem to suffer from it.  The traffic would go to the ASA directly, and unless you had a DMZ set up, the ASA would have no reason to route them to the two infected machines.

More likely, the machines themselves have initiated contact with an outside host as part of its botnet programming and is now busy relaying spam or is itself being ordered to DDOS some other host.

In general, machines that have been compromised like this are easier to rebuild than fix, but if you are inclined to fix it, step one is to shut down the machines.

You cannot clean a well-designed malware while the malware is running.

Therefore, you must boot from an environment that is not compromised.  Either remove the hard drives from the infected machines and connect them as secondary hard drives or external harddrives via USB adapter to a known-good machine and run the scans on them there.

Alternatively, you can boot from a safe clean boot CD image such as the one you can make from here:

http://www.ultimatebootcd.com/

Then, and only then, can you begin the process of analyzing the machine.  Even so, it may be difficult or impossible to completely clean the machine.

Tools that may be of use to you include Spybot, AdAware and IceSword.  The UltimateBootCD can be configured to download and keep these and other anti-virus and anti-spybot software up to date.

Best bet, in my experience, is to take the machine down, save all the DATA, and rebuild the machine.
0
 
LVL 1

Author Comment

by:davis
ID: 24246668
Yes, the machines have non-routable addreses.  The traffic is traversing port 80.  We have tracked the ip to 97.65.135.154 on the outside and when blocked, traffic returns to normal.  Also, on the ASA,  it's the outside interface which shows the sustained traffic levels which meet the full bandwidth available for the circut.  Inside interface shows very little (normal) traffic during this time.    If PC's on the network were generating traffic, wouldn't the inside interface packet hits be much higher?

I have already rebuilt one of the PC's and am monitoring the situation.  The traffic is not consistently generated by the same machines - there seem to be less than 10 clients who are 'top offenders' and typically randomizes.  The client on Monday who was a problem may lie dormant for a few days and then show up as a problem again on Thursday.  Unfortunately, one of the 'problem' PC's is actually a highly-configured Citrix server which will be a nightmare to rebuild.  I have tried running Malware Bytes, Symantec AV, and Microsoft Malicious SW Removal tool all in Safe Mode.  In your experience, can well-designed malware still run in safemode?  
0
 
LVL 4

Expert Comment

by:lkraven
ID: 24246792
Interestingly, a google search on that IP address points it back to EE here where someone states it is an address for symantec's live update.

Are you running Symantec anti-virus by any chance?

Anyway, regarding your ASA traffic.  It's quite possible that 3megabits of traffic will completely saturate your internet connection, but it represents less than 3% of available bandwidth on a fast ethernet interface such as would be on the ASA.  If you are looking at the ASA graph by itself I believe they scale automatically so the graphs themselves won't tell you the whole picture unless you know how they correlate to available bandwidth.

It seems unlikely that your Citrix server has been compromised if it has been kept patched and up to date, but nothing is impossible.

As for your second question, some of the malware out there, especially the big botnets are very sophisticated.  They are not only capable of running in safe mode, but are capable of running in between the  kernel and the rest of the operating system.  Not only will they run in safe-mode, they will run and be invisible to the task manager and most basic process explorers.  Your so-called root kits essentially prevent your operating system from even detecting that they exist using traditional tools, and this is why IceSword is such a valuable tool.

Unfortunately, that is the current state of affairs, but by and large, vigilant systems admins are usually well protected from 99% of the available malware out there, and are only really susceptible if they're significantly behind on patching systems and firewalls and IPS software is not properly configured.

However, based on the fact that your Citrix server is displaying this, and the odd fact that that IP shows up in relationship to Symantec-- if you ARE running Symantec's antivirus, it may be the cause of the issue and you should hopefully be able to rest a little easier while you figure out what to do about it.
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 1

Author Comment

by:davis
ID: 24246950
Yes, we are running Symantec/Norton Corporate Edition Anti Virus.  All clients are configured to point to the parent server for definition updates. This certainly brings me a big sigh of relief but still doesn't explain the sustained and often increasing packet hits by these otherwise 'idle' PC's.

The 'problem' systems were fairly up-to-date with patches and were behind only 6 security updates when I manually ran Windows Updates after beginning troubleshooting this issue.  Obviously, it would only require one available vulnerability to cause problems.  

In any event, it will be a little easier to rest and can now focus on the possibility Symantec is involved with the increased traffic. I'd like to investigate further and respond to this post with my findings as soon as I have some valid information.  Again, thanks.  This is extremely helpful

0
 
LVL 1

Author Comment

by:davis
ID: 24259636
I spent a considerable amount of time with Symantec support yesterday reviewing the situation and we have decided to uninstall the current NAV CE client and push the latest version of NAV CE 10.1 to the client and monitor the situation.  This will sync the client version - many clients are now at 10.0 with the server version of 10.1.8.8000.  The theory is that with a recent upgrade of the server version, a bug may have surfaced in the older client version that may cause it to continuously poll live update.  The update has been performed on a few of the 'problem' PC's - I will report my findings soon.  Thanks for your help -
0
 
LVL 16

Expert Comment

by:warturtle
ID: 24262428
There's something else that you can also think about as well - are you distributing antivirus updates via a shared folder on a server? or all workstations are making requests to Symantec servers on the internet to get updates?

It might be easier to just set the workstations to download and install updates from a shared location on a server and allow only the server to download updates from the internet-based Symantec servers.

That might make things easier on the bandwidth side of things.

There's also a tool by Trend Micro called RUBotted, its currently in Beta phase, so there are likely to be false-positives from it but might be worth trying:
http://www.trendsecure.com/portal/en-US/tools/security_tools/rubotted

Hope it helps.
0
 
LVL 1

Author Comment

by:davis
ID: 24318297
The console has the clients configured to 'Update virus definitions from parent server'.  also,  'schedule client for automatic updates using LiveUpdate' is unchecked for all clients.  So, if all is working as expected, all updates to clients should be occurring on the LAN, from the parent server.  Although, Symantec suspected there may be a problem with some of the older version 10.0 clients and is recommending upgrading all to the 10.1.8.8000 version.   I have upgraded the AV on those which reappear on the monitoring console as 'top offenders' and do not see them resurface afterwards.  Symantec checked for corrupt LiveUpdae on clients and did not find any problems.  I will keep an eye on the situation but is seems to have improved after upgrading AV clients.  Great link for the RUbotted, thanks - I will check it out as well.
0
 
LVL 1

Author Comment

by:davis
ID: 24375350
Lately, I have had good success with stoping machines from using internet bandwidth simply by upgrading Symantec AV client sw.  Yesterday, after discovering another PC which was sonsuming all 3meg of the bonded T1 internet pipe, simply uninstalling the software without a reboot showed a sharp drop in utilization by the machine.  I then rebooted the pc, pushed the new client AV version out to the system, and monitored for recurring problems. So far, there haven't been any.  I believe the problem may be solved.
0
 
LVL 1

Author Comment

by:davis
ID: 24398522
Thanks for re-opening.  Ater discoveing new information, I wanted this question to have a valid answer to hopefully help someone else (and not lead them down the wrong path)

Of course, after closing the case, I had another user on the network which was consuming the entire available bandwidth but also noticed this was a client that had already been upgraded to the latest version of Symantec AV software.  So, now I realize that the problem is NOT rooted with symantec.  After logging into the system remotely and attempting several 'options' to shutdown the offending service on the problem PC, I find that it appears Adobe Acrobat is the culprit.  Starting 'Adobe Updater' causes the internet to 'peak and hold'.  As soon as hitting 'cancel' on the Updater, the internet utilization by the problem PC returns to normal.  After making this connection, I found the answer in this link (http://forums.adobe.com/thread/391741).  Seems to have nailed it, as we have the same Cisco hardware and all the details presented in the thread mirror my problem exactly.  For some reason 'enable deferred scanning for files larger than 2MB' was unchecked on our ASA.  Enabling it seems to be the real fix.  Nonetheless, I appreciate your support

Here are more links with similar detail:
http://www.experts-exchange.com/Security/Software_Firewalls/Enterprise_Firewalls/Checkpoint_Firewall/Q_23898143.html

http://www.marshal8e6.com/kb/article.aspx?id=12667

http://i-proving.ca/space/Adobe+Automatic+Updates+utilizing+100%25+of+Firewall+Bandwidth

http://www.ghacks.net/2008/10/04/disable-adobe-updater/
0
 
LVL 16

Expert Comment

by:warturtle
ID: 24399570
Thats good piece of information. Thanks, Davis. I am adding it to my knowledgebase to help others who might have a similar problem.
0
 

Accepted Solution

by:
ee_auto earned 0 total points
ID: 24825642
Question PAQ'd, 500 points refunded, and stored in the solution database.
0
 
LVL 1

Author Closing Comment

by:davis
ID: 31575194
Ultimately, solution was to update SAV CE managed clients from 10.1 version to 10.1.8.8000 version.  lkraven discovered the source and assisted wit resolving problem.  thanks
0

Featured Post

 The Evil-ution of Network Security Threats

What are the hacks that forever changed the security industry? To answer that question, we created an exciting new eBook that takes you on a trip through hacking history. It explores the top hacks from the 80s to 2010s, why they mattered, and how the security industry responded.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Did you know that more than 4 billion data records have been recorded as lost or stolen since 2013? It was a staggering number brought to our attention during last week’s ManageEngine webinar, where attendees received a comprehensive look at the ma…
In this article, WatchGuard's Director of Security Strategy and Research Teri Radichel, takes a look at insider threats, the risk they can pose to your organization, and the best ways to defend against them.
Loops Section Overview
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question