Solved

Troubleshooting TCP connections on Cisco ASA

Posted on 2011-02-24
8
2,316 Views
Last Modified: 2012-05-11
We need to trace the life of TCP connections on a Cisco ASA to troubleshoot a problem we're having with database connections across our WAN.

To give a bit of history; we run a database app that makes a tcp connection with the database server and then waits for the return data. This can take hours. Consequently we had to create a policy on our ASA to increase the TCP timeout from the default (2?) hours to 8 hours.

Unfortunately we're having some problems and need to rule out the ASA. I've turned on syslogging but I'm not confident that we're seeing the full picture. I'd expect to see for every TCP connection the message stating that it was built and then, later, a message stating that it was torn down (due to timeout, tcp fin, syn timeout, whatever).

However, I'm seeing timeouts for connections where I can find no "built" messages and vice versa.

We're using UDP syslog with the level set to debug.
0
Comment
Question by:tlcsupport
  • 4
  • 4
8 Comments
 
LVL 5

Accepted Solution

by:
torvir earned 500 total points
ID: 34969288
A capture is a better way to see what's going on.
If you have Telnet or SSH access to the ASA it's rather easy.

First go in to configuration mode to define an access-list describing the traffic you want to see.
I always use access-list 111 for this, so that I know this is for sniffing. We use named access-list for everything else.
You would almost always want to see traffic in both directions, so remember to invert every line of the access-list
Example:
access-list 111 permit ip host 194.17.178.16 any
access-list 111 permit ip any host 194.17.178.16

Now you can leave config mode.

Then you configure capture on the interfaces. You often want to see if the packets coming in really comes out in the other end.
To do this you apply the same access-list on both sides. If the traffic is NAT:ed you have to have different access-lists. Just do a similar acl 112.
You have to have the names of the interfaces to do this. The interface names I used in the example are "Outside" and "Inside"
Name the captures to short names like IN or OUT. I use A or B a lot.
Example:
capture A access-list 111 interface Outside
capture B access-list 111 interface Inside

Generate some traffic and have a look what's captured
sh capture A
sh capture B

Maybe you want to start the capture from scratch with empty capture buffers.
clear capture A
clear capture B

When you have found whats wrong you should remove the capture and the access-list.
Example:
no capture A
no capture B
conf t
no access-list 111 permit ip host 194.17.178.16 any
no access-list 111 permit ip any host 194.17.178.16




0
 
LVL 1

Author Comment

by:tlcsupport
ID: 34969405
I've used captures before when checking connectivity. Do they actually show connection information beyond the actual data? I'm keen to see what the ASA is doing with the TCP connection - specifically I need to see when it terminates and why.

My problem is that people are pointing their finger at "the network", blaming it for the database session disconnects.
0
 
LVL 5

Expert Comment

by:torvir
ID: 34969599
You see every packet with ethernet, IP and TCP headers. So you can definitley see what's going on.
An example is the time-out you are talking about. When you had the shorter time-out, you could see that it was the ASA that disconnected the session, and not the server behind the ASA.
If this is about sessions that lasts for several hours you have to do some clever filtering. The capture buffer is not that big.
One option is to put a sniffer on each side of the firewall on monitor ports in the switches, if possible.
What the ASA does internally is trickier to see. There you only have the logging. Be sure to set it to informational. Or even debugging if you don't see everything.
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 
LVL 1

Author Comment

by:tlcsupport
ID: 34970429
Indeed, the capture session isn't showing or won't show what the ASA does with the TCP connection since there isn't any network traffic associated with, for example, the ASA ending a TCP connection due to timeout - and if there were, it wouldn't be captured by the capture session any way since it wouldn't match the ACL.

We've got logging set to debug on the ASA but as I said in the question, I'm not sure it's accurately logging everything to syslog - I'm seeing TCP connections setup but never torn down and vice versa (i.e. I'm seeing tear downs for TCP connections that, according to the syslog, were never established in the first place. That's assuming I'm interpreting it correctly. I assume when it says Building TCP connection 123456, the number 123456 is the "id" of the TCP connection, internally to the ASA.
0
 
LVL 5

Expert Comment

by:torvir
ID: 34970672
You are right regarding the connection ID. And you should find setup and teardown in pairs. It could be that some sessions are setup before the start of the log you are looking at, and you only see teardowns for those. Likewise with sessions that are setup but not yet ended when you look at the log.
0
 
LVL 1

Author Comment

by:tlcsupport
ID: 34970753
hmm, perhaps. Although the log spanned 20 hours which is a bloody long time not to see a connection die... especially as the application the connection "belonged" to thought it had died.

I'm worried that I'm losing syslog data. We've got the logging rate limit left as default (unlimited) and the only disabled syslog ids pertain to snmp ipv4 an snmp ipv6.

Any benefit to changing the syslogging from using UDP to TCP?
0
 
LVL 5

Expert Comment

by:torvir
ID: 34977619
Just a thought. If you experience that syslog messages are being dropped and that an application is having problems with dropped connections.
Could it be that you have packet drops at some point in your internal network? Between your ASA and syslog server for instance.
Have you checked the interface counters on your inner ASA interface and on the switch port it connects to?
0
 
LVL 1

Author Comment

by:tlcsupport
ID: 35116090
Checked the interface counters on the ASA and the connecting switch and seeing zero or no drops. I think I'm going to have to experiment with this in a lab environment
0

Featured Post

Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
Use of TCL script on Cisco devices:  - create file and merge it with running configuration to apply configuration changes
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
Both in life and business – not all partnerships are created equal. Spend 30 short minutes with us to learn:   • Key questions to ask when considering a partnership to accelerate your business into the cloud • Pitfalls and mistakes other partners…

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question