We help IT Professionals succeed at work.
Get Started

ASA 5510 exceeding connecion limit of 50,000 connections

591 Views
Last Modified: 2015-11-09
We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small  software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here.  Any help on this would be much appreciated. These a the steps we have taken.

Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
      access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
      class-map inside5_Users-class_Connections
        match access-list inside5_Users_mpc
      policy-map inside5_Users-policy_Connections
        description Limit VLAN 5 connections to 2000
        class inside5_Users-class_Connections
          set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
      service-policy inside5_Users-policy_Connections interface inside5_Users

Turned on " Threat detection" which will shun violators for 60 seconds.

Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).

ran netstat -n on host with 8000 connections and netstat only showed about 50 connections

ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.

found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.

clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found

our baseline for connections on the weekend with very few if any user connections is around 15,000 connections

we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.

i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.

this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
    TCP flow count/limit = 2184/unlimited
    TCP embryonic count to host = 0
    UDP flow count/limit = 12/unlimited

Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.

Thanks in advance
Any ideas would be greatly appreciated.

L. Mace
Comment
Watch Question
CERTIFIED EXPERT
Most Valuable Expert 2015
Commented:
This problem has been solved!
Unlock 1 Answer and 26 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE