We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here. Any help on this would be much appreciated. These a the steps we have taken.
Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
class-map inside5_Users-class_Connections
match access-list inside5_Users_mpc
policy-map inside5_Users-policy_Connections
description Limit VLAN 5 connections to 2000
class inside5_Users-class_Connections
set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
service-policy inside5_Users-policy_Connections interface inside5_Users
Turned on " Threat detection" which will shun violators for 60 seconds.
Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).
ran netstat -n on host with 8000 connections and netstat only showed about 50 connections
ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.
found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.
clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found
our baseline for connections on the weekend with very few if any user connections is around 15,000 connections
we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.
i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.
this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
TCP flow count/limit = 2184/unlimited
TCP embryonic count to host = 0
UDP flow count/limit = 12/unlimited
Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.
Thanks in advance
Any ideas would be greatly appreciated.
L. Mace