Solved

Troubleshooting TCP connections on Cisco ASA

Posted on 2011-02-24
8
2,238 Views
Last Modified: 2012-05-11
We need to trace the life of TCP connections on a Cisco ASA to troubleshoot a problem we're having with database connections across our WAN.

To give a bit of history; we run a database app that makes a tcp connection with the database server and then waits for the return data. This can take hours. Consequently we had to create a policy on our ASA to increase the TCP timeout from the default (2?) hours to 8 hours.

Unfortunately we're having some problems and need to rule out the ASA. I've turned on syslogging but I'm not confident that we're seeing the full picture. I'd expect to see for every TCP connection the message stating that it was built and then, later, a message stating that it was torn down (due to timeout, tcp fin, syn timeout, whatever).

However, I'm seeing timeouts for connections where I can find no "built" messages and vice versa.

We're using UDP syslog with the level set to debug.
0
Comment
Question by:tlcsupport
  • 4
  • 4
8 Comments
 
LVL 5

Accepted Solution

by:
torvir earned 500 total points
Comment Utility
A capture is a better way to see what's going on.
If you have Telnet or SSH access to the ASA it's rather easy.

First go in to configuration mode to define an access-list describing the traffic you want to see.
I always use access-list 111 for this, so that I know this is for sniffing. We use named access-list for everything else.
You would almost always want to see traffic in both directions, so remember to invert every line of the access-list
Example:
access-list 111 permit ip host 194.17.178.16 any
access-list 111 permit ip any host 194.17.178.16

Now you can leave config mode.

Then you configure capture on the interfaces. You often want to see if the packets coming in really comes out in the other end.
To do this you apply the same access-list on both sides. If the traffic is NAT:ed you have to have different access-lists. Just do a similar acl 112.
You have to have the names of the interfaces to do this. The interface names I used in the example are "Outside" and "Inside"
Name the captures to short names like IN or OUT. I use A or B a lot.
Example:
capture A access-list 111 interface Outside
capture B access-list 111 interface Inside

Generate some traffic and have a look what's captured
sh capture A
sh capture B

Maybe you want to start the capture from scratch with empty capture buffers.
clear capture A
clear capture B

When you have found whats wrong you should remove the capture and the access-list.
Example:
no capture A
no capture B
conf t
no access-list 111 permit ip host 194.17.178.16 any
no access-list 111 permit ip any host 194.17.178.16




0
 
LVL 1

Author Comment

by:tlcsupport
Comment Utility
I've used captures before when checking connectivity. Do they actually show connection information beyond the actual data? I'm keen to see what the ASA is doing with the TCP connection - specifically I need to see when it terminates and why.

My problem is that people are pointing their finger at "the network", blaming it for the database session disconnects.
0
 
LVL 5

Expert Comment

by:torvir
Comment Utility
You see every packet with ethernet, IP and TCP headers. So you can definitley see what's going on.
An example is the time-out you are talking about. When you had the shorter time-out, you could see that it was the ASA that disconnected the session, and not the server behind the ASA.
If this is about sessions that lasts for several hours you have to do some clever filtering. The capture buffer is not that big.
One option is to put a sniffer on each side of the firewall on monitor ports in the switches, if possible.
What the ASA does internally is trickier to see. There you only have the logging. Be sure to set it to informational. Or even debugging if you don't see everything.
0
 
LVL 1

Author Comment

by:tlcsupport
Comment Utility
Indeed, the capture session isn't showing or won't show what the ASA does with the TCP connection since there isn't any network traffic associated with, for example, the ASA ending a TCP connection due to timeout - and if there were, it wouldn't be captured by the capture session any way since it wouldn't match the ACL.

We've got logging set to debug on the ASA but as I said in the question, I'm not sure it's accurately logging everything to syslog - I'm seeing TCP connections setup but never torn down and vice versa (i.e. I'm seeing tear downs for TCP connections that, according to the syslog, were never established in the first place. That's assuming I'm interpreting it correctly. I assume when it says Building TCP connection 123456, the number 123456 is the "id" of the TCP connection, internally to the ASA.
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 5

Expert Comment

by:torvir
Comment Utility
You are right regarding the connection ID. And you should find setup and teardown in pairs. It could be that some sessions are setup before the start of the log you are looking at, and you only see teardowns for those. Likewise with sessions that are setup but not yet ended when you look at the log.
0
 
LVL 1

Author Comment

by:tlcsupport
Comment Utility
hmm, perhaps. Although the log spanned 20 hours which is a bloody long time not to see a connection die... especially as the application the connection "belonged" to thought it had died.

I'm worried that I'm losing syslog data. We've got the logging rate limit left as default (unlimited) and the only disabled syslog ids pertain to snmp ipv4 an snmp ipv6.

Any benefit to changing the syslogging from using UDP to TCP?
0
 
LVL 5

Expert Comment

by:torvir
Comment Utility
Just a thought. If you experience that syslog messages are being dropped and that an application is having problems with dropped connections.
Could it be that you have packet drops at some point in your internal network? Between your ASA and syslog server for instance.
Have you checked the interface counters on your inner ASA interface and on the switch port it connects to?
0
 
LVL 1

Author Comment

by:tlcsupport
Comment Utility
Checked the interface counters on the ASA and the connecting switch and seeing zero or no drops. I think I'm going to have to experiment with this in a lab environment
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

This article will cover setting up redundant ISPs for outbound connectivity on an ASA 5510 (although the same should work on the 5520s and up as well).  It’s important to note that this covers outbound connectivity only.  The ASA does not have built…
Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now