Avatar of lmace712
lmace712
 asked on

ASA 5510 exceeding connecion limit of 50,000 connections

We have an Cisco ASA 5510, (ver 8.4(4)1), with a connection limit of 50,000 connections. for the last 2 weeks we have random hosts that take as many as 11000 connections for no apparent reason. After about 49000 or above we start getting packet loss which makes the network unusable. We have been trouble shooting for several days and cannot find the root cause. We are a small  software and hardware company so many of our engineers have testing tools that could appear as DoS attacks, however, these connections have started on Accounting and Market hosts as well, so no pattern here.  Any help on this would be much appreciated. These a the steps we have taken.

Setup a service policy to limit users on VLAN5 to 2000 connections. (VLAN5 is where most of our user are)
      access-list inside5_Users_mpc line 1 extended permit ip object VLAN5 any
      class-map inside5_Users-class_Connections
        match access-list inside5_Users_mpc
      policy-map inside5_Users-policy_Connections
        description Limit VLAN 5 connections to 2000
        class inside5_Users-class_Connections
          set connection conn-max 0 embryonic-conn-max 0 per-client-max 2000 per-client-embryonic-max 1000 random-sequence-number enable
      service-policy inside5_Users-policy_Connections interface inside5_Users

Turned on " Threat detection" which will shun violators for 60 seconds.

Ran wireshark for packet capture on host with 8000 connections, did not see anything strange. (I'm not an expert on wireshark so i may have missed something).

ran netstat -n on host with 8000 connections and netstat only showed about 50 connections

ran Sophos AntiVirus on a couple of hosts with thousands of connections, found no malware or viruses.

found a couple of servers that always have 2000 - 3000 connections, but we feel this is "normaL" as one is an "opsview" server, the other is a db server.

clients with runaway connections are both Win7 , Linux, and Windows Server, looking for a pattern here, none found

our baseline for connections on the weekend with very few if any user connections is around 15,000 connections

we have around 100 server running on this network, about 90 of which are VM's both hyper-v and esxi hosts.

i have a trail version of Manage Engine Netflow Analyzer and can see limits being reached on "Scans" and some "DoS" attacks, however, most of it seems to be to our domain controllers, but not sure how to interpret this exactly. I have checked the hosts that seem suspect but cannot find any malware or virus present.

this is an example of a random host with more than "normal"
local host: <10.6.5.229>,
    TCP flow count/limit = 2184/unlimited
    TCP embryonic count to host = 0
    UDP flow count/limit = 12/unlimited

Attached is a wireshark capture on a host with 11000+ connection. Also, after I cleared the connections, it never happened again on this host, that was several days ago.

Thanks in advance
Any ideas would be greatly appreciated.

L. Mace
CiscoHardware FirewallsNetworkingNetwork Analysis

Avatar of undefined
Last Comment
lmace712

8/22/2022 - Mon
Jan Bacher

I may be under-caffeinated but I don't see an attachment.
lmace712

ASKER
Experts Exchange would not allow the upload, not sure why they don't accept this extension, this is my first day on this venue
Jan Bacher

Send me an EE message with your email and I'll send you mine.  You can email if EE doesn't allow the upload (it may be the size of the file?).
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Don Johnston

You can change the extension to a generic .txt or something like that then just let us know what the extension is supposed to be.
lmace712

ASKER
The extension was .pcapng
lmace712

ASKER
her it is
192.168.35.78.txt
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jan Bacher

Can you configure this IP that has an RDP server to listen only on TCP and restrict the number of connections per IP?
lmace712

ASKER
It only happened on this server once and not since. Will it help to set it up this way? Also, not sure if I know how to restrict it to connections per IP and listen only on TCP?
Jan Bacher

I am not a Windows expert.  

Don may pick up more than I did.  The majority of the data in the capture looked to be UDP port 3389.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
lmace712

ASKER
That was my connection from 192.168.35.7, I believe.
Jan Bacher

That was conversation between client 192.168.20.158 and server 192.168.35.78.
lmace712

ASKER
that rdp session was from the host I was on at the time.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Jan Bacher

It would be better represented if you had the ASA logs during the period in question.  I'm not seeing a problem when viewing the packet capture.
lmace712

ASKER
Right, that is my dilemma, every host that I check has no information regarding the connections.  I will try to get logs during another "attack"
Jan Bacher

If you have a linux server, you should syslog the data even if you only keep 4 weeks of it.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
lmace712

ASKER
I will try to get that setup
lmace712

ASKER
i am seeing this message on the log viewer:

Nov 03 2015      12:33:50      733100                              [ DNS 53] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is 10; Current average rate is 49 per second, max configured rate is 5; Cumulative total count is 29515
Jan Bacher

sh run | i threat

what is your threat detection configuration?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
lmace712

ASKER
APASA5510-1# show run | i threat
threat-detection basic-threat
threat-detection scanning-threat shun except ip-address 10.3.0.0 255.255.0.0
threat-detection scanning-threat shun except ip-address 192.168.35.7 255.255.255.255
threat-detection scanning-threat shun duration 60
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
lmace712

ASKER
It appears that we have a spoofed IP ,the ip 10.6.20.61, is shut down for at least 5 minutes and I see the 10.8.0.99 is still building connections to it, it was up over 10000 connections and I shut the sever down and then cleared the connections, after that 10.8.0.99 started building connections, it is only up to about 12 now, but this, still does not seem right. Both servers are Linux VM's, one running on an esxi server the other on a hyper-v cluster. Any idea how to fix this?


APASA5510-1# show conn address 10.8.0.99
6049 in use, 50015 most used
UDP inside8 10.8.0.99:8400 inside6 10.6.5.20:8400, idle 0:00:00, bytes 3637, flags -
UDP inside8 10.8.0.99:8544 inside6 10.6.5.20:8544, idle 0:00:02, bytes 15017, flags -
TCP inside8 10.8.0.99:38132 inside6 10.6.20.61:443, idle 0:00:01, bytes 0, flags b
TCP inside8 10.8.0.99:38125 inside6 10.6.20.61:443, idle 0:00:13, bytes 0, flags b
TCP inside8 10.8.0.99:38122 inside6 10.6.20.61:443, idle 0:00:44, bytes 0, flags b
TCP inside8 10.8.0.99:38114 inside6 10.6.20.61:443, idle 0:01:14, bytes 0, flags b
TCP inside8 10.8.0.99:38109 inside6 10.6.20.61:443, idle 0:01:44, bytes 0, flags b
Jan Bacher

What is 10.8.0.99?  

Are these queries (valid vs invalid) coming from an unmanned device?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
lmace712

ASKER
10.8.0.99 is a Linux server that is building up connections constantly to 10000. It is a QA server that is part of a group of QA servers. it keeps building connections to 10.6.20.61 even when 10.6.20.61 is shut down
Jan Bacher

What is the app and for what purpose does it build connections?
lmace712

ASKER
I am trying to find out now. I don't really have much access to the Linux servers, usually our devops guys do. what would you recommend that they look for or do?

I know that they test software  for locating data from cisco mse, (Cisco Mobility Services), and tag data, but not sure what else is going on. I' looking to find out though. These servers are 3 time zones away from me. so communication with other staff is sometime difficult.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
ASKER CERTIFIED SOLUTION
Jan Bacher

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
lmace712

ASKER
OK, thanks, I will let you know.