Troubleshooting TCP connections on Cisco ASA

We need to trace the life of TCP connections on a Cisco ASA to troubleshoot a problem we're having with database connections across our WAN.

To give a bit of history; we run a database app that makes a tcp connection with the database server and then waits for the return data. This can take hours. Consequently we had to create a policy on our ASA to increase the TCP timeout from the default (2?) hours to 8 hours.

Unfortunately we're having some problems and need to rule out the ASA. I've turned on syslogging but I'm not confident that we're seeing the full picture. I'd expect to see for every TCP connection the message stating that it was built and then, later, a message stating that it was torn down (due to timeout, tcp fin, syn timeout, whatever).

However, I'm seeing timeouts for connections where I can find no "built" messages and vice versa.

We're using UDP syslog with the level set to debug.
Who is Participating?
torvirConnect With a Mentor Commented:
A capture is a better way to see what's going on.
If you have Telnet or SSH access to the ASA it's rather easy.

First go in to configuration mode to define an access-list describing the traffic you want to see.
I always use access-list 111 for this, so that I know this is for sniffing. We use named access-list for everything else.
You would almost always want to see traffic in both directions, so remember to invert every line of the access-list
access-list 111 permit ip host any
access-list 111 permit ip any host

Now you can leave config mode.

Then you configure capture on the interfaces. You often want to see if the packets coming in really comes out in the other end.
To do this you apply the same access-list on both sides. If the traffic is NAT:ed you have to have different access-lists. Just do a similar acl 112.
You have to have the names of the interfaces to do this. The interface names I used in the example are "Outside" and "Inside"
Name the captures to short names like IN or OUT. I use A or B a lot.
capture A access-list 111 interface Outside
capture B access-list 111 interface Inside

Generate some traffic and have a look what's captured
sh capture A
sh capture B

Maybe you want to start the capture from scratch with empty capture buffers.
clear capture A
clear capture B

When you have found whats wrong you should remove the capture and the access-list.
no capture A
no capture B
conf t
no access-list 111 permit ip host any
no access-list 111 permit ip any host

tlcsupportAuthor Commented:
I've used captures before when checking connectivity. Do they actually show connection information beyond the actual data? I'm keen to see what the ASA is doing with the TCP connection - specifically I need to see when it terminates and why.

My problem is that people are pointing their finger at "the network", blaming it for the database session disconnects.
You see every packet with ethernet, IP and TCP headers. So you can definitley see what's going on.
An example is the time-out you are talking about. When you had the shorter time-out, you could see that it was the ASA that disconnected the session, and not the server behind the ASA.
If this is about sessions that lasts for several hours you have to do some clever filtering. The capture buffer is not that big.
One option is to put a sniffer on each side of the firewall on monitor ports in the switches, if possible.
What the ASA does internally is trickier to see. There you only have the logging. Be sure to set it to informational. Or even debugging if you don't see everything.
Choose an Exciting Career in Cybersecurity

Help prevent cyber-threats and provide solutions to safeguard our global digital economy. Earn your MS in Cybersecurity. WGU’s MSCSIA degree program was designed in collaboration with national intelligence organizations and IT industry leaders.

tlcsupportAuthor Commented:
Indeed, the capture session isn't showing or won't show what the ASA does with the TCP connection since there isn't any network traffic associated with, for example, the ASA ending a TCP connection due to timeout - and if there were, it wouldn't be captured by the capture session any way since it wouldn't match the ACL.

We've got logging set to debug on the ASA but as I said in the question, I'm not sure it's accurately logging everything to syslog - I'm seeing TCP connections setup but never torn down and vice versa (i.e. I'm seeing tear downs for TCP connections that, according to the syslog, were never established in the first place. That's assuming I'm interpreting it correctly. I assume when it says Building TCP connection 123456, the number 123456 is the "id" of the TCP connection, internally to the ASA.
You are right regarding the connection ID. And you should find setup and teardown in pairs. It could be that some sessions are setup before the start of the log you are looking at, and you only see teardowns for those. Likewise with sessions that are setup but not yet ended when you look at the log.
tlcsupportAuthor Commented:
hmm, perhaps. Although the log spanned 20 hours which is a bloody long time not to see a connection die... especially as the application the connection "belonged" to thought it had died.

I'm worried that I'm losing syslog data. We've got the logging rate limit left as default (unlimited) and the only disabled syslog ids pertain to snmp ipv4 an snmp ipv6.

Any benefit to changing the syslogging from using UDP to TCP?
Just a thought. If you experience that syslog messages are being dropped and that an application is having problems with dropped connections.
Could it be that you have packet drops at some point in your internal network? Between your ASA and syslog server for instance.
Have you checked the interface counters on your inner ASA interface and on the switch port it connects to?
tlcsupportAuthor Commented:
Checked the interface counters on the ASA and the connecting switch and seeing zero or no drops. I think I'm going to have to experiment with this in a lab environment
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.