ASA 5510 exceeding connecion limit of 50,000 connections

We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small  software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here.  Any help on this would be much appreciated. These a the steps we have taken.

Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
      access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
      class-map inside5_Users-class_Connections
        match access-list inside5_Users_mpc
      policy-map inside5_Users-policy_Connections
        description Limit VLAN 5 connections to 2000
        class inside5_Users-class_Connections
          set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
      service-policy inside5_Users-policy_Connections interface inside5_Users

Turned on " Threat detection" which will shun violators for 60 seconds.

Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).

ran netstat -n on host with 8000 connections and netstat only showed about 50 connections

ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.

found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.

clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found

our baseline for connections on the weekend with very few if any user connections is around 15,000 connections

we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.

i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.

this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
    TCP flow count/limit = 2184/unlimited
    TCP embryonic count to host = 0
    UDP flow count/limit = 12/unlimited

Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.

Thanks in advance
Any ideas would be greatly appreciated.

L. Mace
lmace712Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jan SpringerCommented:
I may be under-caffeinated but I don't see an attachment.
0
lmace712Author Commented:
Experts Exchange would not allow the upload, not sure why they don't accept this extension, this is my first day on this venue
0
Jan SpringerCommented:
Send me an EE message with your email and I'll send you mine.  You can email if EE doesn't allow the upload (it may be the size of the file?).
0
The IT Degree for Career Advancement

Earn your B.S. in Network Operations and Security and become a network and IT security expert. This WGU degree program curriculum was designed with tech-savvy, self-motivated students in mind – allowing you to use your technical expertise, to address real-world business problems.

Don JohnstonInstructorCommented:
You can change the extension to a generic .txt or something like that then just let us know what the extension is supposed to be.
0
lmace712Author Commented:
The extension was .pcapng
0
lmace712Author Commented:
her it is
192.168.35.78.txt
0
Jan SpringerCommented:
Can you configure this IP that has an RDP server to listen only on TCP and restrict the number of connections per IP?
0
lmace712Author Commented:
It only happened on this server once and not since. Will it help to set it up this way? Also, not sure if I know how to restrict it to connections per IP and listen only on TCP?
0
Jan SpringerCommented:
I am not a Windows expert.  

Don may pick up more than I did.  The majority of the data in the capture looked to be UDP port 3389.
0
lmace712Author Commented:
That was my connection from 192.168.35.7, I believe.
0
Jan SpringerCommented:
That was conversation between client 192.168.20.158 and server 192.168.35.78.
0
lmace712Author Commented:
that rdp session was from the host I was on at the time.
0
Jan SpringerCommented:
It would be better represented if you had the ASA logs during the period in question.  I'm not seeing a problem when viewing the packet capture.
0
lmace712Author Commented:
Right, that is my dilemma, every host that I check has no information regarding the connections.  I will try to get logs during another "attack"
0
Jan SpringerCommented:
If you have a linux server, you should syslog the data even if you only keep 4 weeks of it.
0
lmace712Author Commented:
I will try to get that setup
0
lmace712Author Commented:
i am seeing this message on the log viewer:

Nov 03 2015      12:33:50      733100                              [ DNS 53] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is 10; Current average rate is 49 per second, max configured rate is 5; Cumulative total count is 29515
0
Jan SpringerCommented:
sh run | i threat

what is your threat detection configuration?
0
lmace712Author Commented:
APASA5510-1# show run | i threat
threat-detection basic-threat
threat-detection scanning-threat shun except ip-address 10.3.0.0 255.255.0.0
threat-detection scanning-threat shun except ip-address 192.168.35.7 255.255.255.255
threat-detection scanning-threat shun duration 60
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
0
lmace712Author Commented:
It appears that we have a spoofed IP ,the ip 10.6.20.61, is shut down for at least 5 minutes and I see the 10.8.0.99 is still building connections to it, it was up over 10000 connections and I shut the sever down and then cleared the connections, after that 10.8.0.99 started building connections, it is only up to about 12 now, but this, still does not seem right. Both servers are Linux VM's, one running on an esxi server the other on a hyper-v cluster. Any idea how to fix this?


APASA5510-1# show conn address 10.8.0.99
6049 in use, 50015 most used
UDP inside8 10.8.0.99:8400 inside6 10.6.5.20:8400, idle 0:00:00, bytes 3637, flags -
UDP inside8 10.8.0.99:8544 inside6 10.6.5.20:8544, idle 0:00:02, bytes 15017, flags -
TCP inside8 10.8.0.99:38132 inside6 10.6.20.61:443, idle 0:00:01, bytes 0, flags b
TCP inside8 10.8.0.99:38125 inside6 10.6.20.61:443, idle 0:00:13, bytes 0, flags b
TCP inside8 10.8.0.99:38122 inside6 10.6.20.61:443, idle 0:00:44, bytes 0, flags b
TCP inside8 10.8.0.99:38114 inside6 10.6.20.61:443, idle 0:01:14, bytes 0, flags b
TCP inside8 10.8.0.99:38109 inside6 10.6.20.61:443, idle 0:01:44, bytes 0, flags b
0
Jan SpringerCommented:
What is 10.8.0.99?  

Are these queries (valid vs invalid) coming from an unmanned device?
0
lmace712Author Commented:
10.8.0.99 is a Linux server that is building up connections constantly to 10000. It is a QA server that is part of a group of QA servers. it keeps building connections to 10.6.20.61 even when 10.6.20.61 is shut down
0
Jan SpringerCommented:
What is the app and for what purpose does it build connections?
0
lmace712Author Commented:
I am trying to find out now. I don't really have much access to the Linux servers, usually our devops guys do. what would you recommend that they look for or do?

I know that they test software  for locating data from cisco mse, (Cisco Mobility Services), and tag data, but not sure what else is going on. I' looking to find out though. These servers are 3 time zones away from me. so communication with other staff is sometime difficult.
0
Jan SpringerCommented:
netstat -an | grep tcp | grep 443
tshark -i <interface> -n  -V 10.6.20.61 -w /path/to/some/directory/server.pcap
lsof -p SOMEPORT

where SOMEPORT is retrieved from the netstat

is where I'd start.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
lmace712Author Commented:
OK, thanks, I will let you know.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Cisco

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.