DoS or DDoS attack?

If they are the Top 10 *Sources*, they are likely to be transmitting data, not receiving data.

They may be zombied computers participating in a botnet.

You say you have an ASA at the edge of your network, and therefore I am making an assumption that the machines in question have internal non-routable addresses.  If you were truly under attack from an outside host on your public internet facing IP address, all your users would seem to suffer from it.  The traffic would go to the ASA directly, and unless you had a DMZ set up, the ASA would have no reason to route them to the two infected machines.

More likely, the machines themselves have initiated contact with an outside host as part of its botnet programming and is now busy relaying spam or is itself being ordered to DDOS some other host.

In general, machines that have been compromised like this are easier to rebuild than fix, but if you are inclined to fix it, step one is to shut down the machines.

You cannot clean a well-designed malware while the malware is running.

Therefore, you must boot from an environment that is not compromised.  Either remove the hard drives from the infected machines and connect them as secondary hard drives or external harddrives via USB adapter to a known-good machine and run the scans on them there.

Alternatively, you can boot from a safe clean boot CD image such as the one you can make from here:

http://www.ultimatebootcd.com/

Then, and only then, can you begin the process of analyzing the machine.  Even so, it may be difficult or impossible to completely clean the machine.

Tools that may be of use to you include Spybot, AdAware and IceSword.  The UltimateBootCD can be configured to download and keep these and other anti-virus and anti-spybot software up to date.

Best bet, in my experience, is to take the machine down, save all the DATA, and rebuild the machine.

Yes, the machines have non-routable addreses.  The traffic is traversing port 80.  We have tracked the ip to 97.65.135.154 on the outside and when blocked, traffic returns to normal.  Also, on the ASA,  it's the outside interface which shows the sustained traffic levels which meet the full bandwidth available for the circut.  Inside interface shows very little (normal) traffic during this time.    If PC's on the network were generating traffic, wouldn't the inside interface packet hits be much higher?

I have already rebuilt one of the PC's and am monitoring the situation.  The traffic is not consistently generated by the same machines - there seem to be less than 10 clients who are 'top offenders' and typically randomizes.  The client on Monday who was a problem may lie dormant for a few days and then show up as a problem again on Thursday.  Unfortunately, one of the 'problem' PC's is actually a highly-configured Citrix server which will be a nightmare to rebuild.  I have tried running Malware Bytes, Symantec AV, and Microsoft Malicious SW Removal tool all in Safe Mode.  In your experience, can well-designed malware still run in safemode?  

Interestingly, a google search on that IP address points it back to EE here where someone states it is an address for symantec's live update.

Are you running Symantec anti-virus by any chance?

Anyway, regarding your ASA traffic.  It's quite possible that 3megabits of traffic will completely saturate your internet connection, but it represents less than 3% of available bandwidth on a fast ethernet interface such as would be on the ASA.  If you are looking at the ASA graph by itself I believe they scale automatically so the graphs themselves won't tell you the whole picture unless you know how they correlate to available bandwidth.

It seems unlikely that your Citrix server has been compromised if it has been kept patched and up to date, but nothing is impossible.

As for your second question, some of the malware out there, especially the big botnets are very sophisticated.  They are not only capable of running in safe mode, but are capable of running in between the  kernel and the rest of the operating system.  Not only will they run in safe-mode, they will run and be invisible to the task manager and most basic process explorers.  Your so-called root kits essentially prevent your operating system from even detecting that they exist using traditional tools, and this is why IceSword is such a valuable tool.

Unfortunately, that is the current state of affairs, but by and large, vigilant systems admins are usually well protected from 99% of the available malware out there, and are only really susceptible if they're significantly behind on patching systems and firewalls and IPS software is not properly configured.

However, based on the fact that your Citrix server is displaying this, and the odd fact that that IP shows up in relationship to Symantec-- if you ARE running Symantec's antivirus, it may be the cause of the issue and you should hopefully be able to rest a little easier while you figure out what to do about it.

Yes, we are running Symantec/Norton Corporate Edition Anti Virus.  All clients are configured to point to the parent server for definition updates. This certainly brings me a big sigh of relief but still doesn't explain the sustained and often increasing packet hits by these otherwise 'idle' PC's.

The 'problem' systems were fairly up-to-date with patches and were behind only 6 security updates when I manually ran Windows Updates after beginning troubleshooting this issue.  Obviously, it would only require one available vulnerability to cause problems.  

In any event, it will be a little easier to rest and can now focus on the possibility Symantec is involved with the increased traffic. I'd like to investigate further and respond to this post with my findings as soon as I have some valid information.  Again, thanks.  This is extremely helpful

I spent a considerable amount of time with Symantec support yesterday reviewing the situation and we have decided to uninstall the current NAV CE client and push the latest version of NAV CE 10.1 to the client and monitor the situation.  This will sync the client version - many clients are now at 10.0 with the server version of 10.1.8.8000.  The theory is that with a recent upgrade of the server version, a bug may have surfaced in the older client version that may cause it to continuously poll live update.  The update has been performed on a few of the 'problem' PC's - I will report my findings soon.  Thanks for your help -

There's something else that you can also think about as well - are you distributing antivirus updates via a shared folder on a server? or all workstations are making requests to Symantec servers on the internet to get updates?

It might be easier to just set the workstations to download and install updates from a shared location on a server and allow only the server to download updates from the internet-based Symantec servers.

That might make things easier on the bandwidth side of things.

There's also a tool by Trend Micro called RUBotted, its currently in Beta phase, so there are likely to be false-positives from it but might be worth trying:
http://www.trendsecure.com/portal/en-US/tools/security_tools/rubotted

Hope it helps.

The console has the clients configured to 'Update virus definitions from parent server'.  also,  'schedule client for automatic updates using LiveUpdate' is unchecked for all clients.  So, if all is working as expected, all updates to clients should be occurring on the LAN, from the parent server.  Although, Symantec suspected there may be a problem with some of the older version 10.0 clients and is recommending upgrading all to the 10.1.8.8000 version.   I have upgraded the AV on those which reappear on the monitoring console as 'top offenders' and do not see them resurface afterwards.  Symantec checked for corrupt LiveUpdae on clients and did not find any problems.  I will keep an eye on the situation but is seems to have improved after upgrading AV clients.  Great link for the RUbotted, thanks - I will check it out as well.

Lately, I have had good success with stoping machines from using internet bandwidth simply by upgrading Symantec AV client sw.  Yesterday, after discovering another PC which was sonsuming all 3meg of the bonded T1 internet pipe, simply uninstalling the software without a reboot showed a sharp drop in utilization by the machine.  I then rebooted the pc, pushed the new client AV version out to the system, and monitored for recurring problems. So far, there haven't been any.  I believe the problem may be solved.

Thanks for re-opening.  Ater discoveing new information, I wanted this question to have a valid answer to hopefully help someone else (and not lead them down the wrong path)

Of course, after closing the case, I had another user on the network which was consuming the entire available bandwidth but also noticed this was a client that had already been upgraded to the latest version of Symantec AV software.  So, now I realize that the problem is NOT rooted with symantec.  After logging into the system remotely and attempting several 'options' to shutdown the offending service on the problem PC, I find that it appears Adobe Acrobat is the culprit.  Starting 'Adobe Updater' causes the internet to 'peak and hold'.  As soon as hitting 'cancel' on the Updater, the internet utilization by the problem PC returns to normal.  After making this connection, I found the answer in this link (http://forums.adobe.com/thread/391741).  Seems to have nailed it, as we have the same Cisco hardware and all the details presented in the thread mirror my problem exactly.  For some reason 'enable deferred scanning for files larger than 2MB' was unchecked on our ASA.  Enabling it seems to be the real fix.  Nonetheless, I appreciate your support

Here are more links with similar detail:
https://www.experts-exchange.com/questions/23898143/Adobe-updates-randomly-use-all-bandwidth-on-firewall-for-indefinate-period-of-time.html

http://www.marshal8e6.com/kb/article.aspx?id=12667

http://i-proving.ca/space/Adobe+Automatic+Updates+utilizing+100%25+of+Firewall+Bandwidth

http://www.ghacks.net/2008/10/04/disable-adobe-updater/

Thats good piece of information. Thanks, Davis. I am adding it to my knowledgebase to help others who might have a similar problem.

Ultimately, solution was to update SAV CE managed clients from 10.1 version to 10.1.8.8000 version.  lkraven discovered the source and assisted wit resolving problem.  thanks