Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


Troubleshooting TCP connections on Cisco ASA

Posted on 2011-02-24
Medium Priority
Last Modified: 2012-05-11
We need to trace the life of TCP connections on a Cisco ASA to troubleshoot a problem we're having with database connections across our WAN.

To give a bit of history; we run a database app that makes a tcp connection with the database server and then waits for the return data. This can take hours. Consequently we had to create a policy on our ASA to increase the TCP timeout from the default (2?) hours to 8 hours.

Unfortunately we're having some problems and need to rule out the ASA. I've turned on syslogging but I'm not confident that we're seeing the full picture. I'd expect to see for every TCP connection the message stating that it was built and then, later, a message stating that it was torn down (due to timeout, tcp fin, syn timeout, whatever).

However, I'm seeing timeouts for connections where I can find no "built" messages and vice versa.

We're using UDP syslog with the level set to debug.
Question by:tlcsupport
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4

Accepted Solution

torvir earned 2000 total points
ID: 34969288
A capture is a better way to see what's going on.
If you have Telnet or SSH access to the ASA it's rather easy.

First go in to configuration mode to define an access-list describing the traffic you want to see.
I always use access-list 111 for this, so that I know this is for sniffing. We use named access-list for everything else.
You would almost always want to see traffic in both directions, so remember to invert every line of the access-list
access-list 111 permit ip host any
access-list 111 permit ip any host

Now you can leave config mode.

Then you configure capture on the interfaces. You often want to see if the packets coming in really comes out in the other end.
To do this you apply the same access-list on both sides. If the traffic is NAT:ed you have to have different access-lists. Just do a similar acl 112.
You have to have the names of the interfaces to do this. The interface names I used in the example are "Outside" and "Inside"
Name the captures to short names like IN or OUT. I use A or B a lot.
capture A access-list 111 interface Outside
capture B access-list 111 interface Inside

Generate some traffic and have a look what's captured
sh capture A
sh capture B

Maybe you want to start the capture from scratch with empty capture buffers.
clear capture A
clear capture B

When you have found whats wrong you should remove the capture and the access-list.
no capture A
no capture B
conf t
no access-list 111 permit ip host any
no access-list 111 permit ip any host


Author Comment

ID: 34969405
I've used captures before when checking connectivity. Do they actually show connection information beyond the actual data? I'm keen to see what the ASA is doing with the TCP connection - specifically I need to see when it terminates and why.

My problem is that people are pointing their finger at "the network", blaming it for the database session disconnects.

Expert Comment

ID: 34969599
You see every packet with ethernet, IP and TCP headers. So you can definitley see what's going on.
An example is the time-out you are talking about. When you had the shorter time-out, you could see that it was the ASA that disconnected the session, and not the server behind the ASA.
If this is about sessions that lasts for several hours you have to do some clever filtering. The capture buffer is not that big.
One option is to put a sniffer on each side of the firewall on monitor ports in the switches, if possible.
What the ASA does internally is trickier to see. There you only have the logging. Be sure to set it to informational. Or even debugging if you don't see everything.
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Author Comment

ID: 34970429
Indeed, the capture session isn't showing or won't show what the ASA does with the TCP connection since there isn't any network traffic associated with, for example, the ASA ending a TCP connection due to timeout - and if there were, it wouldn't be captured by the capture session any way since it wouldn't match the ACL.

We've got logging set to debug on the ASA but as I said in the question, I'm not sure it's accurately logging everything to syslog - I'm seeing TCP connections setup but never torn down and vice versa (i.e. I'm seeing tear downs for TCP connections that, according to the syslog, were never established in the first place. That's assuming I'm interpreting it correctly. I assume when it says Building TCP connection 123456, the number 123456 is the "id" of the TCP connection, internally to the ASA.

Expert Comment

ID: 34970672
You are right regarding the connection ID. And you should find setup and teardown in pairs. It could be that some sessions are setup before the start of the log you are looking at, and you only see teardowns for those. Likewise with sessions that are setup but not yet ended when you look at the log.

Author Comment

ID: 34970753
hmm, perhaps. Although the log spanned 20 hours which is a bloody long time not to see a connection die... especially as the application the connection "belonged" to thought it had died.

I'm worried that I'm losing syslog data. We've got the logging rate limit left as default (unlimited) and the only disabled syslog ids pertain to snmp ipv4 an snmp ipv6.

Any benefit to changing the syslogging from using UDP to TCP?

Expert Comment

ID: 34977619
Just a thought. If you experience that syslog messages are being dropped and that an application is having problems with dropped connections.
Could it be that you have packet drops at some point in your internal network? Between your ASA and syslog server for instance.
Have you checked the interface counters on your inner ASA interface and on the switch port it connects to?

Author Comment

ID: 35116090
Checked the interface counters on the ASA and the connecting switch and seeing zero or no drops. I think I'm going to have to experiment with this in a lab environment

Featured Post

[Webinar] Lessons on Recovering from Petya

Skyport is working hard to help customers recover from recent attacks, like the Petya worm. This work has brought to light some important lessons. New malware attacks like this can take down your entire environment. Learn from others mistakes on how to prevent Petya like worms.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

For months I had no idea how to 'discover' the IP address of the other end of a link (without asking someone who knows), and it drove me batty. Think about it. You can't use Cisco Discovery Protocol (CDP) because it's not implemented on the ASAs.…
This article is in regards to the Cisco QSFP-4SFP10G-CU1M cables, which are designed to uplink/downlink 40GB ports to 10GB SFP ports. I recently experienced this and found very little configuration documentation on how these are supposed to be confi…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor ( If you're interested in additional methods for monitoring bandwidt…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question