ASA 5510 exceeding connecion limit of 50,000 connections

lmace712
lmace712 used Ask the Experts™
on
We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small  software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here.  Any help on this would be much appreciated. These a the steps we have taken.

Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
      access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
      class-map inside5_Users-class_Connections
        match access-list inside5_Users_mpc
      policy-map inside5_Users-policy_Connections
        description Limit VLAN 5 connections to 2000
        class inside5_Users-class_Connections
          set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
      service-policy inside5_Users-policy_Connections interface inside5_Users

Turned on " Threat detection" which will shun violators for 60 seconds.

Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).

ran netstat -n on host with 8000 connections and netstat only showed about 50 connections

ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.

found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.

clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found

our baseline for connections on the weekend with very few if any user connections is around 15,000 connections

we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.

i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.

this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
    TCP flow count/limit = 2184/unlimited
    TCP embryonic count to host = 0
    UDP flow count/limit = 12/unlimited

Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.

Thanks in advance
Any ideas would be greatly appreciated.

L. Mace
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2015

Commented:
I may be under-caffeinated but I don't see an attachment.

Author

Commented:
Experts Exchange would not allow the upload, not sure why they don't accept this extension, this is my first day on this venue
Most Valuable Expert 2015

Commented:
Send me an EE message with your email and I'll send you mine.  You can email if EE doesn't allow the upload (it may be the size of the file?).
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Don JohnstonInstructor
Top Expert 2015

Commented:
You can change the extension to a generic .txt or something like that then just let us know what the extension is supposed to be.

Author

Commented:
The extension was .pcapng

Author

Commented:
her it is
192.168.35.78.txt
Most Valuable Expert 2015

Commented:
Can you configure this IP that has an RDP server to listen only on TCP and restrict the number of connections per IP?

Author

Commented:
It only happened on this server once and not since. Will it help to set it up this way? Also, not sure if I know how to restrict it to connections per IP and listen only on TCP?
Most Valuable Expert 2015

Commented:
I am not a Windows expert.  

Don may pick up more than I did.  The majority of the data in the capture looked to be UDP port 3389.

Author

Commented:
That was my connection from 192.168.35.7, I believe.
Most Valuable Expert 2015

Commented:
That was conversation between client 192.168.20.158 and server 192.168.35.78.

Author

Commented:
that rdp session was from the host I was on at the time.
Most Valuable Expert 2015

Commented:
It would be better represented if you had the ASA logs during the period in question.  I'm not seeing a problem when viewing the packet capture.

Author

Commented:
Right, that is my dilemma, every host that I check has no information regarding the connections.  I will try to get logs during another "attack"
Most Valuable Expert 2015

Commented:
If you have a linux server, you should syslog the data even if you only keep 4 weeks of it.

Author

Commented:
I will try to get that setup

Author

Commented:
i am seeing this message on the log viewer:

Nov 03 2015      12:33:50      733100                              [ DNS 53] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is 10; Current average rate is 49 per second, max configured rate is 5; Cumulative total count is 29515
Most Valuable Expert 2015

Commented:
sh run | i threat

what is your threat detection configuration?

Author

Commented:
APASA5510-1# show run | i threat
threat-detection basic-threat
threat-detection scanning-threat shun except ip-address 10.3.0.0 255.255.0.0
threat-detection scanning-threat shun except ip-address 192.168.35.7 255.255.255.255
threat-detection scanning-threat shun duration 60
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200

Author

Commented:
It appears that we have a spoofed IP ,the ip 10.6.20.61, is shut down for at least 5 minutes and I see the 10.8.0.99 is still building connections to it, it was up over 10000 connections and I shut the sever down and then cleared the connections, after that 10.8.0.99 started building connections, it is only up to about 12 now, but this, still does not seem right. Both servers are Linux VM's, one running on an esxi server the other on a hyper-v cluster. Any idea how to fix this?


APASA5510-1# show conn address 10.8.0.99
6049 in use, 50015 most used
UDP inside8 10.8.0.99:8400 inside6 10.6.5.20:8400, idle 0:00:00, bytes 3637, flags -
UDP inside8 10.8.0.99:8544 inside6 10.6.5.20:8544, idle 0:00:02, bytes 15017, flags -
TCP inside8 10.8.0.99:38132 inside6 10.6.20.61:443, idle 0:00:01, bytes 0, flags b
TCP inside8 10.8.0.99:38125 inside6 10.6.20.61:443, idle 0:00:13, bytes 0, flags b
TCP inside8 10.8.0.99:38122 inside6 10.6.20.61:443, idle 0:00:44, bytes 0, flags b
TCP inside8 10.8.0.99:38114 inside6 10.6.20.61:443, idle 0:01:14, bytes 0, flags b
TCP inside8 10.8.0.99:38109 inside6 10.6.20.61:443, idle 0:01:44, bytes 0, flags b
Most Valuable Expert 2015

Commented:
What is 10.8.0.99?  

Are these queries (valid vs invalid) coming from an unmanned device?

Author

Commented:
10.8.0.99 is a Linux server that is building up connections constantly to 10000. It is a QA server that is part of a group of QA servers. it keeps building connections to 10.6.20.61 even when 10.6.20.61 is shut down
Most Valuable Expert 2015

Commented:
What is the app and for what purpose does it build connections?

Author

Commented:
I am trying to find out now. I don't really have much access to the Linux servers, usually our devops guys do. what would you recommend that they look for or do?

I know that they test software  for locating data from cisco mse, (Cisco Mobility Services), and tag data, but not sure what else is going on. I' looking to find out though. These servers are 3 time zones away from me. so communication with other staff is sometime difficult.
Most Valuable Expert 2015
Commented:
netstat -an | grep tcp | grep 443
tshark -i <interface> -n  -V 10.6.20.61 -w /path/to/some/directory/server.pcap
lsof -p SOMEPORT

where SOMEPORT is retrieved from the netstat

is where I'd start.

Author

Commented:
OK, thanks, I will let you know.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial