Link to home
Start Free TrialLog in
Avatar of lmace712
lmace712

asked on

ASA 5510 exceeding connecion limit of 50,000 connections

We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small  software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here.  Any help on this would be much appreciated. These a the steps we have taken.

Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
      access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
      class-map inside5_Users-class_Connections
        match access-list inside5_Users_mpc
      policy-map inside5_Users-policy_Connections
        description Limit VLAN 5 connections to 2000
        class inside5_Users-class_Connections
          set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
      service-policy inside5_Users-policy_Connections interface inside5_Users

Turned on " Threat detection" which will shun violators for 60 seconds.

Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).

ran netstat -n on host with 8000 connections and netstat only showed about 50 connections

ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.

found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.

clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found

our baseline for connections on the weekend with very few if any user connections is around 15,000 connections

we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.

i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.

this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
    TCP flow count/limit = 2184/unlimited
    TCP embryonic count to host = 0
    UDP flow count/limit = 12/unlimited

Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.

Thanks in advance
Any ideas would be greatly appreciated.

L. Mace
Avatar of Jan Bacher
Jan Bacher
Flag of United States of America image

I may be under-caffeinated but I don't see an attachment.
Avatar of lmace712
lmace712

ASKER

Experts Exchange would not allow the upload, not sure why they don't accept this extension, this is my first day on this venue
Send me an EE message with your email and I'll send you mine.  You can email if EE doesn't allow the upload (it may be the size of the file?).
You can change the extension to a generic .txt or something like that then just let us know what the extension is supposed to be.
The extension was .pcapng
her it is
192.168.35.78.txt
Can you configure this IP that has an RDP server to listen only on TCP and restrict the number of connections per IP?
It only happened on this server once and not since. Will it help to set it up this way? Also, not sure if I know how to restrict it to connections per IP and listen only on TCP?
I am not a Windows expert.  

Don may pick up more than I did.  The majority of the data in the capture looked to be UDP port 3389.
That was my connection from 192.168.35.7, I believe.
That was conversation between client 192.168.20.158 and server 192.168.35.78.
that rdp session was from the host I was on at the time.
It would be better represented if you had the ASA logs during the period in question.  I'm not seeing a problem when viewing the packet capture.
Right, that is my dilemma, every host that I check has no information regarding the connections.  I will try to get logs during another "attack"
If you have a linux server, you should syslog the data even if you only keep 4 weeks of it.
I will try to get that setup
i am seeing this message on the log viewer:

Nov 03 2015      12:33:50      733100                              [ DNS 53] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is 10; Current average rate is 49 per second, max configured rate is 5; Cumulative total count is 29515
sh run | i threat

what is your threat detection configuration?
APASA5510-1# show run | i threat
threat-detection basic-threat
threat-detection scanning-threat shun except ip-address 10.3.0.0 255.255.0.0
threat-detection scanning-threat shun except ip-address 192.168.35.7 255.255.255.255
threat-detection scanning-threat shun duration 60
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
It appears that we have a spoofed IP ,the ip 10.6.20.61, is shut down for at least 5 minutes and I see the 10.8.0.99 is still building connections to it, it was up over 10000 connections and I shut the sever down and then cleared the connections, after that 10.8.0.99 started building connections, it is only up to about 12 now, but this, still does not seem right. Both servers are Linux VM's, one running on an esxi server the other on a hyper-v cluster. Any idea how to fix this?


APASA5510-1# show conn address 10.8.0.99
6049 in use, 50015 most used
UDP inside8 10.8.0.99:8400 inside6 10.6.5.20:8400, idle 0:00:00, bytes 3637, flags -
UDP inside8 10.8.0.99:8544 inside6 10.6.5.20:8544, idle 0:00:02, bytes 15017, flags -
TCP inside8 10.8.0.99:38132 inside6 10.6.20.61:443, idle 0:00:01, bytes 0, flags b
TCP inside8 10.8.0.99:38125 inside6 10.6.20.61:443, idle 0:00:13, bytes 0, flags b
TCP inside8 10.8.0.99:38122 inside6 10.6.20.61:443, idle 0:00:44, bytes 0, flags b
TCP inside8 10.8.0.99:38114 inside6 10.6.20.61:443, idle 0:01:14, bytes 0, flags b
TCP inside8 10.8.0.99:38109 inside6 10.6.20.61:443, idle 0:01:44, bytes 0, flags b
What is 10.8.0.99?  

Are these queries (valid vs invalid) coming from an unmanned device?
10.8.0.99 is a Linux server that is building up connections constantly to 10000. It is a QA server that is part of a group of QA servers. it keeps building connections to 10.6.20.61 even when 10.6.20.61 is shut down
What is the app and for what purpose does it build connections?
I am trying to find out now. I don't really have much access to the Linux servers, usually our devops guys do. what would you recommend that they look for or do?

I know that they test software  for locating data from cisco mse, (Cisco Mobility Services), and tag data, but not sure what else is going on. I' looking to find out though. These servers are 3 time zones away from me. so communication with other staff is sometime difficult.
ASKER CERTIFIED SOLUTION
Avatar of Jan Bacher
Jan Bacher
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
OK, thanks, I will let you know.