Troubleshooting TCP connections on Cisco ASA

Posted on 2011-02-24
Last Modified: 2012-05-11
We need to trace the life of TCP connections on a Cisco ASA to troubleshoot a problem we're having with database connections across our WAN.

To give a bit of history; we run a database app that makes a tcp connection with the database server and then waits for the return data. This can take hours. Consequently we had to create a policy on our ASA to increase the TCP timeout from the default (2?) hours to 8 hours.

Unfortunately we're having some problems and need to rule out the ASA. I've turned on syslogging but I'm not confident that we're seeing the full picture. I'd expect to see for every TCP connection the message stating that it was built and then, later, a message stating that it was torn down (due to timeout, tcp fin, syn timeout, whatever).

However, I'm seeing timeouts for connections where I can find no "built" messages and vice versa.

We're using UDP syslog with the level set to debug.
Question by:tlcsupport
  • 4
  • 4

Accepted Solution

torvir earned 500 total points
ID: 34969288
A capture is a better way to see what's going on.
If you have Telnet or SSH access to the ASA it's rather easy.

First go in to configuration mode to define an access-list describing the traffic you want to see.
I always use access-list 111 for this, so that I know this is for sniffing. We use named access-list for everything else.
You would almost always want to see traffic in both directions, so remember to invert every line of the access-list
access-list 111 permit ip host any
access-list 111 permit ip any host

Now you can leave config mode.

Then you configure capture on the interfaces. You often want to see if the packets coming in really comes out in the other end.
To do this you apply the same access-list on both sides. If the traffic is NAT:ed you have to have different access-lists. Just do a similar acl 112.
You have to have the names of the interfaces to do this. The interface names I used in the example are "Outside" and "Inside"
Name the captures to short names like IN or OUT. I use A or B a lot.
capture A access-list 111 interface Outside
capture B access-list 111 interface Inside

Generate some traffic and have a look what's captured
sh capture A
sh capture B

Maybe you want to start the capture from scratch with empty capture buffers.
clear capture A
clear capture B

When you have found whats wrong you should remove the capture and the access-list.
no capture A
no capture B
conf t
no access-list 111 permit ip host any
no access-list 111 permit ip any host


Author Comment

ID: 34969405
I've used captures before when checking connectivity. Do they actually show connection information beyond the actual data? I'm keen to see what the ASA is doing with the TCP connection - specifically I need to see when it terminates and why.

My problem is that people are pointing their finger at "the network", blaming it for the database session disconnects.

Expert Comment

ID: 34969599
You see every packet with ethernet, IP and TCP headers. So you can definitley see what's going on.
An example is the time-out you are talking about. When you had the shorter time-out, you could see that it was the ASA that disconnected the session, and not the server behind the ASA.
If this is about sessions that lasts for several hours you have to do some clever filtering. The capture buffer is not that big.
One option is to put a sniffer on each side of the firewall on monitor ports in the switches, if possible.
What the ASA does internally is trickier to see. There you only have the logging. Be sure to set it to informational. Or even debugging if you don't see everything.

Author Comment

ID: 34970429
Indeed, the capture session isn't showing or won't show what the ASA does with the TCP connection since there isn't any network traffic associated with, for example, the ASA ending a TCP connection due to timeout - and if there were, it wouldn't be captured by the capture session any way since it wouldn't match the ACL.

We've got logging set to debug on the ASA but as I said in the question, I'm not sure it's accurately logging everything to syslog - I'm seeing TCP connections setup but never torn down and vice versa (i.e. I'm seeing tear downs for TCP connections that, according to the syslog, were never established in the first place. That's assuming I'm interpreting it correctly. I assume when it says Building TCP connection 123456, the number 123456 is the "id" of the TCP connection, internally to the ASA.
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.


Expert Comment

ID: 34970672
You are right regarding the connection ID. And you should find setup and teardown in pairs. It could be that some sessions are setup before the start of the log you are looking at, and you only see teardowns for those. Likewise with sessions that are setup but not yet ended when you look at the log.

Author Comment

ID: 34970753
hmm, perhaps. Although the log spanned 20 hours which is a bloody long time not to see a connection die... especially as the application the connection "belonged" to thought it had died.

I'm worried that I'm losing syslog data. We've got the logging rate limit left as default (unlimited) and the only disabled syslog ids pertain to snmp ipv4 an snmp ipv6.

Any benefit to changing the syslogging from using UDP to TCP?

Expert Comment

ID: 34977619
Just a thought. If you experience that syslog messages are being dropped and that an application is having problems with dropped connections.
Could it be that you have packet drops at some point in your internal network? Between your ASA and syslog server for instance.
Have you checked the interface counters on your inner ASA interface and on the switch port it connects to?

Author Comment

ID: 35116090
Checked the interface counters on the ASA and the connecting switch and seeing zero or no drops. I think I'm going to have to experiment with this in a lab environment

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
site to site tunnel not autostarting 5 59
Cisco Air AP 6 41
How do I grant a service account specific permissions to query Active Directory 3 51
VIRL IP adress 3 58
Cisco Pix/ASA hairpinning The term, hairpinning, comes from the fact that the traffic comes from one source into a router or similar device, makes a U-turn, and goes back the same way it came. Visualize this and you will see something that looks …
This article will cover setting up redundant ISPs for outbound connectivity on an ASA 5510 (although the same should work on the 5520s and up as well).  It’s important to note that this covers outbound connectivity only.  The ASA does not have built…
Here's a very brief overview of the methods PRTG Network Monitor ( offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
Both in life and business – not all partnerships are created equal. Spend 30 short minutes with us to learn:   • Key questions to ask when considering a partnership to accelerate your business into the cloud • Pitfalls and mistakes other partners…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now